4 ms·

Tracking Java native memory with JDK flight recorder

136 points by mfiguiere 3y ago

pron 3y ago
For summary views: $ jfr view native-memory-reserved rec.jfr and $ jfr view native-memory-committed rec.jfr https://x.com/ErikGahlin/status/1736530559231201484 https://x.com/ErikGahlin/status/1736530559231201484
twic 3y ago
> the one thing which NMT does not report, despite what the name might suggest, is any memory allocated by native libraries, for instance invoked via JNI If you're using glibc, then malloc does have information about that, and provides ways to read it, so it's a shame this isn't exposed. It would be quite helpful in the face of suspected native library memory leaks.
- strangemonad 3y ago
  jemalloc + memleak and perf work pretty well in that case. I think you could do something similar with tcmalloc
  - planede 3y ago
    If we are talking replacing the libc allocator, then something like heaptrack is worth mentioning. https://github.com/KDE/heaptrack https://github.com/KDE/heaptrack
DarkmSparks 3y ago
been a big fan of flight recorder since it was part of jrocket & mission control iirc. great to see it making something of a come back.
the-smug-one 3y ago
Here's some more NMT usage without flight recorder: https://blog.arkey.fr/2020/11/30/off-heap-reconnaissance/ https://blog.arkey.fr/2020/11/30/off-heap-reconnaissance/
vanillax 3y ago
whats a real world use case for this? Been doing big enterprise java for a decade and never ran into a scenario where I would need this. Don't get me wrong, I think theres most certainly value for this, but usually doing plain ole Spring Boot or the dreaded Adobe Experience manager, things just "work". So just honestly curious what problem this would help me solve? Im guessing if you develop a platform you need performance and need to find weak points in your platform? Or is this for building out tools like Dynatrace / New Relic.
- bzzzt 3y ago
  I’ve encountered native memory issues in those “enterprise” apps. Lots a frameworks set an upper bound and if you load one library too much you can run out of JIT compiler cache leading to constant compilation which tanks performance. More ways to get insights can’t hurt.
- brabel 3y ago
  Using offheap memory allocation in Java is generally a bad idea. Java is a GC'd language. If you need to manage memory manually, use a language designed for that. The lengths people go to NOT learn another language (while having to learn stuff like this and do primitive memory management in your application code - which itself may use the GC if you're not careful) amazes me.
  - bzzzt 3y ago
    While I fully agree as a rule you shouldn't have to do manual memory management in Java, the functionality is there and useful in exceptional situations. I've seen integrations where there's no '100% pure Java' implementation available, so you have to choose between picking a battle-tested native library or reimplementing from scratch in Java (which sometimes isn't even possible with closed source systems)
  - pjmlp 3y ago
    It is called taking your cake and eating it too, in terms of productivity, IDE tooling and library ecosystem, for the same reasons ML frameworks use Python with bindings to C++ libraries, instead of being 100% written in C++.
kkcorps 3y ago
debugging native calls in itself is also painful. I have switched to using async-profiler (https://github.com/async-profiler/async-profiler https://github.com/async-profiler/async-profiler) instead of JFR for most of my usecases. A. it tracks native calls by default B. it can track wall time as well C. you can have neat interactive flamegraphs
- Sarkie 3y ago
  I can't believe we need to use async profiler tbh. It should all be in jfc/jmc
  - aardvark179 3y ago
    I hear there are plans to fix that. In the meantime you can get async profiler to output in JFR format and combine it with a separate JFR recording.
    - Sarkie 3y ago
      Got any links by chance?
gunnarmorling 3y ago
Author of the post here, so nice to see it being discussed here. For folks interested to learn more about JFR, here are a few other post on that topic: * https://www.morling.dev/blog/finding-java-thread-leaks-with-jdk-flight-recorder-and-bit-of-sql/ https://www.morling.dev/blog/finding-java-thread-leaks-with-...: Discusses how to find thread likes with JFR and JFR Analytics, a project I've created for querying recordings with SQL * https://www.morling.dev/blog/towards-continuous-performance-regression-testing/ https://www.morling.dev/blog/towards-continuous-performance-...: Discusses how to use JFR for continuous performance testing, by means of asserting "proxy metrics" such as allocation rates and IO * https://www.morling.dev/blog/rest-api-monitoring-with-custom-jdk-flight-recorder-events/ https://www.morling.dev/blog/rest-api-monitoring-with-custom...: Discusses how to create your own application-specific JFR events
vinay_ys 3y ago
In the past with Java 8 JFR wasn't free for use in commercial production environments. So we built our own profilers. It's nice that with NFTC, Oracle allows you to use JFR in production from Java 11 onwards. But they can revoke NFTC anytime. So, it is still wise to develop and use open-source or private alternatives to JFR. Besides JFR still doesn't profile everything well and you still need to build infrastructure for integrated distributed profiling and tracing.

136 points by mfiguiere 3y ago

pron 3y ago

For summary views: $ jfr view native-memory-reserved rec.jfr and $ jfr view native-memory-committed rec.jfr https://x.com/ErikGahlin/status/1736530559231201484 https://x.com/ErikGahlin/status/1736530559231201484

twic 3y ago

> the one thing which NMT does not report, despite what the name might suggest, is any memory allocated by native libraries, for instance invoked via JNI If you're using glibc, then malloc does have information about that, and provides ways to read it, so it's a shame this isn't exposed. It would be quite helpful in the face of suspected native library memory leaks.

strangemonad 3y ago
jemalloc + memleak and perf work pretty well in that case. I think you could do something similar with tcmalloc
- planede 3y ago
  If we are talking replacing the libc allocator, then something like heaptrack is worth mentioning. https://github.com/KDE/heaptrack https://github.com/KDE/heaptrack

DarkmSparks 3y ago

been a big fan of flight recorder since it was part of jrocket & mission control iirc. great to see it making something of a come back.

the-smug-one 3y ago

Here's some more NMT usage without flight recorder: https://blog.arkey.fr/2020/11/30/off-heap-reconnaissance/ https://blog.arkey.fr/2020/11/30/off-heap-reconnaissance/

vanillax 3y ago

whats a real world use case for this? Been doing big enterprise java for a decade and never ran into a scenario where I would need this. Don't get me wrong, I think theres most certainly value for this, but usually doing plain ole Spring Boot or the dreaded Adobe Experience manager, things just "work". So just honestly curious what problem this would help me solve? Im guessing if you develop a platform you need performance and need to find weak points in your platform? Or is this for building out tools like Dynatrace / New Relic.

bzzzt 3y ago
I’ve encountered native memory issues in those “enterprise” apps. Lots a frameworks set an upper bound and if you load one library too much you can run out of JIT compiler cache leading to constant compilation which tanks performance. More ways to get insights can’t hurt.
brabel 3y ago
Using offheap memory allocation in Java is generally a bad idea. Java is a GC'd language. If you need to manage memory manually, use a language designed for that. The lengths people go to NOT learn another language (while having to learn stuff like this and do primitive memory management in your application code - which itself may use the GC if you're not careful) amazes me.
- bzzzt 3y ago
  While I fully agree as a rule you shouldn't have to do manual memory management in Java, the functionality is there and useful in exceptional situations. I've seen integrations where there's no '100% pure Java' implementation available, so you have to choose between picking a battle-tested native library or reimplementing from scratch in Java (which sometimes isn't even possible with closed source systems)
- pjmlp 3y ago
  It is called taking your cake and eating it too, in terms of productivity, IDE tooling and library ecosystem, for the same reasons ML frameworks use Python with bindings to C++ libraries, instead of being 100% written in C++.

kkcorps 3y ago

debugging native calls in itself is also painful. I have switched to using async-profiler (https://github.com/async-profiler/async-profiler https://github.com/async-profiler/async-profiler) instead of JFR for most of my usecases. A. it tracks native calls by default B. it can track wall time as well C. you can have neat interactive flamegraphs

Sarkie 3y ago
I can't believe we need to use async profiler tbh. It should all be in jfc/jmc
- aardvark179 3y ago
  I hear there are plans to fix that. In the meantime you can get async profiler to output in JFR format and combine it with a separate JFR recording.
  - Sarkie 3y ago
    Got any links by chance?

gunnarmorling 3y ago

Author of the post here, so nice to see it being discussed here. For folks interested to learn more about JFR, here are a few other post on that topic: * https://www.morling.dev/blog/finding-java-thread-leaks-with-jdk-flight-recorder-and-bit-of-sql/ https://www.morling.dev/blog/finding-java-thread-leaks-with-...: Discusses how to find thread likes with JFR and JFR Analytics, a project I've created for querying recordings with SQL * https://www.morling.dev/blog/towards-continuous-performance-regression-testing/ https://www.morling.dev/blog/towards-continuous-performance-...: Discusses how to use JFR for continuous performance testing, by means of asserting "proxy metrics" such as allocation rates and IO * https://www.morling.dev/blog/rest-api-monitoring-with-custom-jdk-flight-recorder-events/ https://www.morling.dev/blog/rest-api-monitoring-with-custom...: Discusses how to create your own application-specific JFR events

vinay_ys 3y ago

In the past with Java 8 JFR wasn't free for use in commercial production environments. So we built our own profilers. It's nice that with NFTC, Oracle allows you to use JFR in production from Java 11 onwards. But they can revoke NFTC anytime. So, it is still wise to develop and use open-source or private alternatives to JFR. Besides JFR still doesn't profile everything well and you still need to build infrastructure for integrated distributed profiling and tracing.