3 ms·

Try looking at it with async-profiler. This can be done in production. I discovered performance problems in unexpected (and some expected) places with it in the

by morsch 4y ago

Try looking at it with async-profiler. This can be done in production. I discovered performance problems in unexpected (and some expected) places with it in the past. It may be more helpful if it's your application code that's to blame, though, less so if it's the JVM itself. https://github.com/jvm-profiling-tools/async-profiler https://github.com/jvm-profiling-tools/async-profiler

kelseyfrog 4y ago
I'm surprised to find this comment at the bottom of the comment section. My advice, like yours, is to profile performance issues before getting into the hypothesis-change-measure loop. Like you said, it quickly bifurcates the problem space into application code and the JVM and eliminates entire classes of performance problems. It's important to point out too that while most people assume cpu profiling, the JVM and hence most profiling tools also have memory profiling which can be helpful in diagnosing problems (especially ones related to GC). I hope the querent ends up profiling and shares the results. It would be both fun and productive to investigate the results collaboratively.
- morsch 4y ago
  The cool thing is that async-profiler is lightweight enough to run it in production, even against a server that's already under heavy load (in my experience, ymmv). Oh and it's free. We have a cron job running it three times a day (alloc and cpu, both). Of course the report is rudimentary compared to heavyweight profilers.
Matthias247 4y ago
+1. Continuous profiling should help figuring out the root cause of such issues. I have only used an in-house tool for Java for this so far, so I don't have a recommendation for a specific one. But the linked one looks reasonable, and there might be other tools too. It might not even require continuous profiling. One alternative approach is to capture some minutes of profiling data after startup (when CPU usage looks good), and some minutes after the CPU spikes occur. Then compare those two and check for major differences.

by morsch 4y ago

kelseyfrog 4y ago

I'm surprised to find this comment at the bottom of the comment section. My advice, like yours, is to profile performance issues before getting into the hypothesis-change-measure loop. Like you said, it quickly bifurcates the problem space into application code and the JVM and eliminates entire classes of performance problems. It's important to point out too that while most people assume cpu profiling, the JVM and hence most profiling tools also have memory profiling which can be helpful in diagnosing problems (especially ones related to GC). I hope the querent ends up profiling and shares the results. It would be both fun and productive to investigate the results collaboratively.

morsch 4y ago
The cool thing is that async-profiler is lightweight enough to run it in production, even against a server that's already under heavy load (in my experience, ymmv). Oh and it's free. We have a cron job running it three times a day (alloc and cpu, both). Of course the report is rudimentary compared to heavyweight profilers.

Matthias247 4y ago

+1. Continuous profiling should help figuring out the root cause of such issues. I have only used an in-house tool for Java for this so far, so I don't have a recommendation for a specific one. But the linked one looks reasonable, and there might be other tools too. It might not even require continuous profiling. One alternative approach is to capture some minutes of profiling data after startup (when CPU usage looks good), and some minutes after the CPU spikes occur. Then compare those two and check for major differences.