7 ms·

Most Sampling Java Profilers Are Terrible

54 points by dmit 11y ago

imtringued 11y ago
Sampling profilers in general are terrible. Unfortunately I can't get code instrumenting profilers to work on the eclipse process. The worst thing about the JVM aprofiler won't tell you about how much time the JVM spends JITing/class loading. For short running processes you might be surprised to find that the profiler tells you everything is okay but in my case the program still needed one second to start. I was pleasantly suprised that the equivalent python programm merely needs 100ms from start to finish (essentially feeling instant) even though it should be slower overall.
- sevenseconds 11y ago
  jstat can tell you how much time your jvm is spending jiting/classloading ... http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstat.html#class_option http://docs.oracle.com/javase/7/docs/technotes/tools/share/j...
- barrkel 11y ago
  Instrumenting profilers are also terrible because they make tiny methods look way more expensive than they should be, warping the results. Not only does the instrumentation cost, but they can prevent methods from being inlined, interfere with optimizations, cache, and generally give misleading results. For a sampling profiler to give meaningful results, it shouldn't be waiting for safepoints. Ideally it would capture stack traces very cheaply (just walk the stack frames, noting the return addresses) and turn those numbers into symbolic locations on its own time. Yes, inlining is going to confuse things - that's the downside of working with optimized code. Optimization inherently intermingles code that may be separated by some distance, and any one instruction may be a fuse of multiple source lines. If your instrumenting profiler prevents this, you're no longer measuring the actual production code. Java makes it harder, of course, with code on a GC'd heap, dynamic codegen, etc.
- aardvark179 11y ago
  jstat or Java Flight Recorder/Mission Control will give you plenty of info on class loading, JITing, GC, and almost everything else going on in the VM, and there are plugins fort VisualVM which will provide a lot of the same information.
ysleepy 11y ago
Well, a bit click-baity. Yes sampling profilers are inaccurate, and most users know the limitations. It is deliberately a profiler on the one side of the trade-off. The google paper was more interesting.
- arielweisberg 11y ago
  Except you can have a sampling profiler that does better than what most sampling profilers on the JVM are doing? IOW the point of the blog post is not that sampling profilers are bad or that sampling profilers on the JVM are bad, but that some sampling profilers are bad because they use bad methodology. At no additional cost you could be using profilers that use a more accurate mechanism. To me that is very useful information. I'm always looking for opportunities to get something for nothing. I would also not characterize users of sampling profilers as knowing their limitations. Taking a WAG I would guess less than 50% understand the limitations. A much larger percentages know limitations exist, but I don't think they actually know what to do about them.
CJefferson 11y ago
I've only started profiling Java code this week, and I've been shocked by the poor state of affairs. Does anyone have any recommendations, preferably that can tell you (some) information at the line level. I never realised how spoilt I was with C/C++ options (gcov and valgrind aren't great, but they mostly do the job).
- aardvark179 11y ago
  If you want coverage tools then you should be looking for coverage tools, not profiling tools. Jacoco is a good library that you can plug into your build processes, or add as an agent on the command line, and it has an Eclipse plugin. It will tell you about partial and complete coverage of lines, and about the number of branches taken if coverage of a line is not complete.
TYPE_FASTER 11y ago
I've had good results using DynaTrace for profiling .NET code. They have a Java version that I would try if I needed such a thing: http://www.dynatrace.com/en/technology/java.html http://www.dynatrace.com/en/technology/java.html What is really helpful is that you can run DynaTrace in production without impacting performance. DynaTrace agents run on the production web/app servers and send data to DynaTrace collectors. This makes resolving production issues way easier. You sort by CPU time in descending order and work down the list. You already have the stack trace and data associated with the request or SQL query.
- conorgdaly 11y ago
  Could you give rough idea of cost - dev license/prod license? Their own site is 'Contact Us', & Google is not turning up anything of use either.
  - bpchaps 11y ago
    It's expensive. I think when my last company bought it, it was 15k per license or something astronomical like that. They bought 20 or so licenses and never actually used them in prod - mostly because of the effort in building up a sane config for ops to use. The guy who set it up did a pretty good job, but after coming up to him and mentioning the 2hr/day stop the world GCing going on, dynatrace wasn't able to see them, where the jvm logs could. So yeah - it's a good product, but you better hope you have the money for it and the patience to build a good config.
- windowsworkstoo 11y ago
  Sure but the issue with Dynatrace and similar ilk (NewRelic also) is they still attach a profiler and do have some impact. Now, I wouldn't typically care, but you can get at lest 95% of the same information, including yes, stack traces, with ETW for basically free - in cost and in perf impact on Windows at least. Throw the results in a flame graph and boom, done. Windows in particular has so much great inbuilt stuff for troubleshooting perf but its hard to find and not well marketed, so these other folks make a killing selling an inferior product.
zamalek 11y ago
Throw .Net into this category. Spent weeks of my life and our clients' lives wasting time with, not inaccurate, but grossly incorrect results from a paid product. It either confused locks for CPU time (obscuring real hotspots) or simply reported CPU time where there was none. So far, only VTune has resulted in actionable profiler results (there may be others); within hours (not weeks) we had solved many issues by switching to it. A profiler is as indispensable as a compiler, don't skimp on it - you're going to end up spending that money one way or another.
norswap 11y ago
I've never ever gotten any good results with a Java profiler. HPROF runs, but its results are garbage. Yourkit and VisualVM are apparently not able to handle the heat of a CPU intensive application and crash or are generally unusable (and this on two different computers with different OSes). If you must know, I was trying to profile a parser.
- gshx 11y ago
  I have had good results with Yourkit. And so far in my experience, that is the only profiler that has been able to survive profiling busy services running large heaps. ymmv.
- needusername 11y ago
  I've have gotten good results with Oracle Flight Recorder. A big downside is that it requires a license from Oracle for production use.
j_s 11y ago
Was trying to dig up the HN discussion for this article from last summer, but haven't found it yet: http://techblog.netflix.com/2015/07/java-in-flames.html http://techblog.netflix.com/2015/07/java-in-flames.html Java mixed-mode flame graphs provide a complete visualization of CPU usage and [...] can identify all CPU consumers and issues, including those that are hidden from other profilers.
grandinj 11y ago
For me, this is a storm in a teacup. I use Netbeans Profiler regularly, and it produces decent results for me. It's only when the profile starts getting pretty flat that the inaccuracies of this method start showing up, but for run of the mill performance problems where you have nice fat spikes sticking up here and there, it is perfectly adequate.

54 points by dmit 11y ago

Sampling profilers in general are terrible. Unfortunately I can't get code instrumenting profilers to work on the eclipse process. The worst thing about the JVM aprofiler won't tell you about how much time the JVM spends JITing/class loading. For short running processes you might be surprised to find that the profiler tells you everything is okay but in my case the program still needed one second to start. I was pleasantly suprised that the equivalent python programm merely needs 100ms from start to finish (essentially feeling instant) even though it should be slower overall.

sevenseconds 11y ago
jstat can tell you how much time your jvm is spending jiting/classloading ... http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstat.html#class_option http://docs.oracle.com/javase/7/docs/technotes/tools/share/j...
barrkel 11y ago
Instrumenting profilers are also terrible because they make tiny methods look way more expensive than they should be, warping the results. Not only does the instrumentation cost, but they can prevent methods from being inlined, interfere with optimizations, cache, and generally give misleading results. For a sampling profiler to give meaningful results, it shouldn't be waiting for safepoints. Ideally it would capture stack traces very cheaply (just walk the stack frames, noting the return addresses) and turn those numbers into symbolic locations on its own time. Yes, inlining is going to confuse things - that's the downside of working with optimized code. Optimization inherently intermingles code that may be separated by some distance, and any one instruction may be a fuse of multiple source lines. If your instrumenting profiler prevents this, you're no longer measuring the actual production code. Java makes it harder, of course, with code on a GC'd heap, dynamic codegen, etc.
aardvark179 11y ago
jstat or Java Flight Recorder/Mission Control will give you plenty of info on class loading, JITing, GC, and almost everything else going on in the VM, and there are plugins fort VisualVM which will provide a lot of the same information.

ysleepy 11y ago

Well, a bit click-baity. Yes sampling profilers are inaccurate, and most users know the limitations. It is deliberately a profiler on the one side of the trade-off. The google paper was more interesting.

arielweisberg 11y ago
Except you can have a sampling profiler that does better than what most sampling profilers on the JVM are doing? IOW the point of the blog post is not that sampling profilers are bad or that sampling profilers on the JVM are bad, but that some sampling profilers are bad because they use bad methodology. At no additional cost you could be using profilers that use a more accurate mechanism. To me that is very useful information. I'm always looking for opportunities to get something for nothing. I would also not characterize users of sampling profilers as knowing their limitations. Taking a WAG I would guess less than 50% understand the limitations. A much larger percentages know limitations exist, but I don't think they actually know what to do about them.

CJefferson 11y ago

I've only started profiling Java code this week, and I've been shocked by the poor state of affairs. Does anyone have any recommendations, preferably that can tell you (some) information at the line level. I never realised how spoilt I was with C/C++ options (gcov and valgrind aren't great, but they mostly do the job).

aardvark179 11y ago
If you want coverage tools then you should be looking for coverage tools, not profiling tools. Jacoco is a good library that you can plug into your build processes, or add as an agent on the command line, and it has an Eclipse plugin. It will tell you about partial and complete coverage of lines, and about the number of branches taken if coverage of a line is not complete.

TYPE_FASTER 11y ago

I've had good results using DynaTrace for profiling .NET code. They have a Java version that I would try if I needed such a thing: http://www.dynatrace.com/en/technology/java.html http://www.dynatrace.com/en/technology/java.html What is really helpful is that you can run DynaTrace in production without impacting performance. DynaTrace agents run on the production web/app servers and send data to DynaTrace collectors. This makes resolving production issues way easier. You sort by CPU time in descending order and work down the list. You already have the stack trace and data associated with the request or SQL query.

conorgdaly 11y ago
Could you give rough idea of cost - dev license/prod license? Their own site is 'Contact Us', & Google is not turning up anything of use either.
- bpchaps 11y ago
  It's expensive. I think when my last company bought it, it was 15k per license or something astronomical like that. They bought 20 or so licenses and never actually used them in prod - mostly because of the effort in building up a sane config for ops to use. The guy who set it up did a pretty good job, but after coming up to him and mentioning the 2hr/day stop the world GCing going on, dynatrace wasn't able to see them, where the jvm logs could. So yeah - it's a good product, but you better hope you have the money for it and the patience to build a good config.
windowsworkstoo 11y ago
Sure but the issue with Dynatrace and similar ilk (NewRelic also) is they still attach a profiler and do have some impact. Now, I wouldn't typically care, but you can get at lest 95% of the same information, including yes, stack traces, with ETW for basically free - in cost and in perf impact on Windows at least. Throw the results in a flame graph and boom, done. Windows in particular has so much great inbuilt stuff for troubleshooting perf but its hard to find and not well marketed, so these other folks make a killing selling an inferior product.

zamalek 11y ago

Throw .Net into this category. Spent weeks of my life and our clients' lives wasting time with, not inaccurate, but grossly incorrect results from a paid product. It either confused locks for CPU time (obscuring real hotspots) or simply reported CPU time where there was none. So far, only VTune has resulted in actionable profiler results (there may be others); within hours (not weeks) we had solved many issues by switching to it. A profiler is as indispensable as a compiler, don't skimp on it - you're going to end up spending that money one way or another.

norswap 11y ago

I've never ever gotten any good results with a Java profiler. HPROF runs, but its results are garbage. Yourkit and VisualVM are apparently not able to handle the heat of a CPU intensive application and crash or are generally unusable (and this on two different computers with different OSes). If you must know, I was trying to profile a parser.

gshx 11y ago
I have had good results with Yourkit. And so far in my experience, that is the only profiler that has been able to survive profiling busy services running large heaps. ymmv.
needusername 11y ago
I've have gotten good results with Oracle Flight Recorder. A big downside is that it requires a license from Oracle for production use.

j_s 11y ago

Was trying to dig up the HN discussion for this article from last summer, but haven't found it yet: http://techblog.netflix.com/2015/07/java-in-flames.html http://techblog.netflix.com/2015/07/java-in-flames.html Java mixed-mode flame graphs provide a complete visualization of CPU usage and [...] can identify all CPU consumers and issues, including those that are hidden from other profilers.

grandinj 11y ago

For me, this is a storm in a teacup. I use Netbeans Profiler regularly, and it produces decent results for me. It's only when the profile starts getting pretty flat that the inaccuracies of this method start showing up, but for run of the mill performance problems where you have nice fat spikes sticking up here and there, it is perfectly adequate.