6 ms·

How to spy on a Ruby program

229 points by bartbes 10y ago

nxzero 10y ago
One issue I've run across is spying on JRuby's native Java classes. Anyone have a solution?
- ch4s3 10y ago
  I've only ever used VirtualVM, maybe Charles Nutter or Chris Seaton will appear and enlighten us both.
  - chrisseaton 10y ago
    I'm not sure what you want to do. Is your problem that you do want to see the native Java classes, or you want to hide them and focus on the Ruby code? Or is the key thing you want a Java flame graph?
    - ch4s3 10y ago
      Java classes and a flame graph would be nice sometimes. I really only know how to use pry and Virtual VM with JRuby.
      - chrisseaton 10y ago
        It looks like some people are using flame graphs for the JVM. http://techblog.netflix.com/2015/07/java-in-flames.html http://techblog.netflix.com/2015/07/java-in-flames.html I'm still puzzled that there seems to be no simple gprof-style graph profiler included with the JVM.
        aardvark179 10y ago
        I"m surprised as well. Last time I needed this I knocked something up that could use the internals of Mission Control or Visual VM to turn their profile formats into flame graphs, but I doubt it would keep working across versions, and it really needed more work to be something anybody else could use sensibly. It was however very easy to do (maybe half an hour's work) and produced very useful results. If you're using an invokeDynamic based language implementation however you will want to filter a lot of internal stack frames out of the graph, or you'll have trouble seeing past the LambdaForms to what's really going on.
      - ch4s3 10y ago
        Good find, thanks Chris. Yeah, you'd think the JVM folks would work to add some tooling like that.
    - ch4s3 10y ago
      Interesting, it looks like Sam Saffron's FlameGraph gem https://github.com/SamSaffron/flamegraph https://github.com/SamSaffron/flamegraph has worked with JRuby for about a year and I hadn't noticed.
parennoob 10y ago
From the article: “I'm constantly surprised by how many people don't know you can do this. It's amazing.” I'm probably nitpicking, but sad to see this in the article. One of the things I love about Julia's writing otherwise is that it is free of this sort of 'I'm surprised that you don't know this simple thing.' expressions.
- gkop 10y ago
  http://xkcd.com/1053/ http://xkcd.com/1053/ opened my eyes to this category of behavior and presents an alternative.
- jvns 10y ago
  hey! Thanks for pointing this out. As you say, I do normally try to audit my writing for feigned suprise like this!! I'll fix it when I get home.
  - parennoob 10y ago
    Hey, thanks for taking notice, and more importantly, fixing it. You're awesome!
guptaneil 10y ago
> maybe it doesn't exist because a Linux-only Ruby debugging tool is sort of a weird thing? Linux-only seems perfectly acceptable, especially as more and more development moves to Docker containers. Great write up, looking forward to seeing a completed tool!
- eeZi 10y ago
  Windows Docker is just around the corner!
eeZi 10y ago
For Python, you can just use Pyrasite (https://github.com/lmacken/pyrasite https://github.com/lmacken/pyrasite) and inject a script which prints a stack trace for all threads. I've done this to debug OpenStack in the past and it worked very well. There are many similar projects for Python, I used this one since it's in the RHEL repo. Useful summary on StackOverflow: https://stackoverflow.com/questions/4163964/python-is-it-possible-to-attach-a-console-into-a-running-process https://stackoverflow.com/questions/4163964/python-is-it-pos...
camperman 10y ago
I hadn't heard of perf - very handy indeed. It's in the linux-tools package for distros using apt. You will need to install both the generic package and the one for your kernel version.
- blacksmith_tb 10y ago
  Installing linux-tools-generic seems to have been enough for me (on Trusty).
jon-wood 10y ago
If you're planning ahead you can have your application load rbtrace which then allows connecting to a running process to see what it's doing, with options to limit to slow calls, IO, of specific method calls.
lamontcg 10y ago
When I've needed to dump a stack trace, I've just included an interrupt handler which prints the call stack to a debug log. That has been good enough for the problem of "wtf is this ruby process doing for _minutes_ at a time?" That doesn't get you flame graphs, but you can take a few snapshots and get an idea of what is happening. For more involved perf debugging I've used ruby-prof.
xjlin0 10y ago
Nice write. The FlameGraph link in Julia's article is broken, maybe points to this one? https://github.com/brendangregg/FlameGraph https://github.com/brendangregg/FlameGraph
jzwinck 10y ago
Of course there are equivalent programs for C. On GNU/Linux it's pstack aka gstack. Also gcore if you want an entire core dump you can analyze later with gdb. http://linux.die.net/man/1/pstack http://linux.die.net/man/1/pstack
jsnk 10y ago
"I was going to paste the strace output of what gdb is actually doing, but it is 20 megabytes of system calls." I think this is why you shouldn't run this on production server itself. Each call is very resource intensive on the production server. I believe the right way to analyze memory is to use "gcore", dump the memory, download it to the local machine's VM instance that's running the same OS as the production using scp. Also download the same ruby binary that production is running, and use gdb on the VM to analyze memory dump.
- jeffdavis 10y ago
  That may be "right" given the current options, but that doesn't mean we can't have better options. And that's exactly what this blog post is about.
gabetax 10y ago
If this sort of stuff is relevant to you, http://patshaughnessy.net/ruby-under-a-microscope http://patshaughnessy.net/ruby-under-a-microscope is an essential book to read.
amagumori 10y ago
You know, sometimes i get slightly eye-rolly at the tone of Julia's articles, but fuck, this sort of attitude in programming is so much better than the alternative (i.e. lkml). Bringing more positivity and jolliness into programming is important. Fuck the traditional macho programmer attitude. I'm gonna try to be more like Julia.
- chanux 10y ago
  Ditto, except for the eye-rolly part:D. Also, I have noticed that she takes simple things we don't usually think deep about and drill down to it opening up underlying complexity and pointing a metaphorical magnifying glass at it. (An example I quickly picked from the blog http://jvns.ca/blog/2014/09/28/how-does-sqlite-work-part-1-pages/ http://jvns.ca/blog/2014/09/28/how-does-sqlite-work-part-1-p...)
Freaky 10y ago
Don't forget if you're on a system with DTrace, Ruby supports a nice bunch of probes for various events: http://ruby-doc.org/core-2.3.0/doc/dtrace_probes_rdoc.html http://ruby-doc.org/core-2.3.0/doc/dtrace_probes_rdoc.html

229 points by bartbes 10y ago

nxzero 10y ago

One issue I've run across is spying on JRuby's native Java classes. Anyone have a solution?

ch4s3 10y ago
I've only ever used VirtualVM, maybe Charles Nutter or Chris Seaton will appear and enlighten us both.
- chrisseaton 10y ago
  I'm not sure what you want to do. Is your problem that you do want to see the native Java classes, or you want to hide them and focus on the Ruby code? Or is the key thing you want a Java flame graph?
  - ch4s3 10y ago
    Java classes and a flame graph would be nice sometimes. I really only know how to use pry and Virtual VM with JRuby.
    - chrisseaton 10y ago
      It looks like some people are using flame graphs for the JVM. http://techblog.netflix.com/2015/07/java-in-flames.html http://techblog.netflix.com/2015/07/java-in-flames.html I'm still puzzled that there seems to be no simple gprof-style graph profiler included with the JVM.
      - aardvark179 10y ago
        I"m surprised as well. Last time I needed this I knocked something up that could use the internals of Mission Control or Visual VM to turn their profile formats into flame graphs, but I doubt it would keep working across versions, and it really needed more work to be something anybody else could use sensibly. It was however very easy to do (maybe half an hour's work) and produced very useful results. If you're using an invokeDynamic based language implementation however you will want to filter a lot of internal stack frames out of the graph, or you'll have trouble seeing past the LambdaForms to what's really going on.
    - ch4s3 10y ago
      Good find, thanks Chris. Yeah, you'd think the JVM folks would work to add some tooling like that.
  - ch4s3 10y ago
    Interesting, it looks like Sam Saffron's FlameGraph gem https://github.com/SamSaffron/flamegraph https://github.com/SamSaffron/flamegraph has worked with JRuby for about a year and I hadn't noticed.

parennoob 10y ago

From the article: “I'm constantly surprised by how many people don't know you can do this. It's amazing.” I'm probably nitpicking, but sad to see this in the article. One of the things I love about Julia's writing otherwise is that it is free of this sort of 'I'm surprised that you don't know this simple thing.' expressions.

gkop 10y ago
http://xkcd.com/1053/ http://xkcd.com/1053/ opened my eyes to this category of behavior and presents an alternative.
jvns 10y ago
hey! Thanks for pointing this out. As you say, I do normally try to audit my writing for feigned suprise like this!! I'll fix it when I get home.
- parennoob 10y ago
  Hey, thanks for taking notice, and more importantly, fixing it. You're awesome!

guptaneil 10y ago

> maybe it doesn't exist because a Linux-only Ruby debugging tool is sort of a weird thing? Linux-only seems perfectly acceptable, especially as more and more development moves to Docker containers. Great write up, looking forward to seeing a completed tool!

eeZi 10y ago
Windows Docker is just around the corner!

eeZi 10y ago

For Python, you can just use Pyrasite (https://github.com/lmacken/pyrasite https://github.com/lmacken/pyrasite) and inject a script which prints a stack trace for all threads. I've done this to debug OpenStack in the past and it worked very well. There are many similar projects for Python, I used this one since it's in the RHEL repo. Useful summary on StackOverflow: https://stackoverflow.com/questions/4163964/python-is-it-possible-to-attach-a-console-into-a-running-process https://stackoverflow.com/questions/4163964/python-is-it-pos...

camperman 10y ago

I hadn't heard of perf - very handy indeed. It's in the linux-tools package for distros using apt. You will need to install both the generic package and the one for your kernel version.

blacksmith_tb 10y ago
Installing linux-tools-generic seems to have been enough for me (on Trusty).

jon-wood 10y ago

If you're planning ahead you can have your application load rbtrace which then allows connecting to a running process to see what it's doing, with options to limit to slow calls, IO, of specific method calls.

lamontcg 10y ago

When I've needed to dump a stack trace, I've just included an interrupt handler which prints the call stack to a debug log. That has been good enough for the problem of "wtf is this ruby process doing for _minutes_ at a time?" That doesn't get you flame graphs, but you can take a few snapshots and get an idea of what is happening. For more involved perf debugging I've used ruby-prof.

xjlin0 10y ago

Nice write. The FlameGraph link in Julia's article is broken, maybe points to this one? https://github.com/brendangregg/FlameGraph https://github.com/brendangregg/FlameGraph

jzwinck 10y ago

Of course there are equivalent programs for C. On GNU/Linux it's pstack aka gstack. Also gcore if you want an entire core dump you can analyze later with gdb. http://linux.die.net/man/1/pstack http://linux.die.net/man/1/pstack

jsnk 10y ago

"I was going to paste the strace output of what gdb is actually doing, but it is 20 megabytes of system calls." I think this is why you shouldn't run this on production server itself. Each call is very resource intensive on the production server. I believe the right way to analyze memory is to use "gcore", dump the memory, download it to the local machine's VM instance that's running the same OS as the production using scp. Also download the same ruby binary that production is running, and use gdb on the VM to analyze memory dump.

jeffdavis 10y ago
That may be "right" given the current options, but that doesn't mean we can't have better options. And that's exactly what this blog post is about.

gabetax 10y ago

If this sort of stuff is relevant to you, http://patshaughnessy.net/ruby-under-a-microscope http://patshaughnessy.net/ruby-under-a-microscope is an essential book to read.

amagumori 10y ago

You know, sometimes i get slightly eye-rolly at the tone of Julia's articles, but fuck, this sort of attitude in programming is so much better than the alternative (i.e. lkml). Bringing more positivity and jolliness into programming is important. Fuck the traditional macho programmer attitude. I'm gonna try to be more like Julia.

chanux 10y ago
Ditto, except for the eye-rolly part:D. Also, I have noticed that she takes simple things we don't usually think deep about and drill down to it opening up underlying complexity and pointing a metaphorical magnifying glass at it. (An example I quickly picked from the blog http://jvns.ca/blog/2014/09/28/how-does-sqlite-work-part-1-pages/ http://jvns.ca/blog/2014/09/28/how-does-sqlite-work-part-1-p...)

Freaky 10y ago

Don't forget if you're on a system with DTrace, Ruby supports a nice bunch of probes for various events: http://ruby-doc.org/core-2.3.0/doc/dtrace_probes_rdoc.html http://ruby-doc.org/core-2.3.0/doc/dtrace_probes_rdoc.html