4 ms·

I am not really that familiar with Java, but I don't understand how to get a JVM heap dump if there is an exception that brings down the JVM or even a thread. I

by js8 8y ago

I am not really that familiar with Java, but I don't understand how to get a JVM heap dump if there is an exception that brings down the JVM or even a thread. Is it even possible? I work on mainframe (z/OS), and it is completely normal there that when an application fails with exception (they're called ABENDs - from ABnormal ENDing), you can get a dump of memory of the application. From that, you can see all the values of the offending variables and all the relevant system areas and so on.

jillesvangurp 8y ago
You can turn this on but the dumps are kind of huge. Also, you can attach to a running JVM and get a memory dump that way. NPEs don't bring down the JVM though unless that happens in your main method of course, in which case the process exits normally.
- js8 8y ago
  What I mean, to produce the dump in the case of exceptions that percolate to top level and exit the process or thread (such as NPE). I am aware that I can attach, but that kind of defeats the ability to get the information in case of unexpected error (in production). Dumps might be huge, but is that really a problem on modern hardware? On z/OS this is doable (and common practice).
  - coldtea 8y ago
    >What I mean, to produce the dump in the case of exceptions that percolate to top level and exit the process or thread (such as NPE). Why not have a top level try catch then?
    - js8 8y ago
      If you have a top level catch, then you lose the local context of the error (local variables, parameters, stack trace). I mean seriously, this is a solved problem on z/OS (the ABEND condition bypasses normal stack, so to speak), it has been solved for over 50 years. I don't see why JVM couldn't do the same thing - is it a technical problem? It seems like a purely cultural problem to me.
      - deepsun 8y ago
        Maybe because in Java it's usually considered a very bad practice to halt your program. All the web containers that run your code will catch any error, to prevent bringing down the whole web service if only a small piece of it fails (for example, cannot serve one URL, but can others). So instead, in Java, it's your responsibility to catch and correctly handle all the exceptions. Also, variables in the last function (where exception was raised from) are unlikely to give me understanding of the bug, because it's usually thrown somewhere deep in others code, and also I'm more interested in _why_ it happened, not just _what_ happened. We could dump all the variables in the stack trace, but I'm not sure I would like to read the whole dump of every variable in stack trace, that's just too much. I'd better write my code with proper logging, so I would be able to read the logs in order to understand not only _what_ happened, but _why_ it happened. With good logs you rarely even need to debug anything, I'd say 95% of the time it's pretty clear where to fix.
        js8 8y ago
        Again, on z/OS, this is a solved problem. If the application runs inside an "application server", for instance CICS, you can still get the dump and the CICS can continue or even restart your app (or it can be restarted through other things). You can get the dump of the process even if just one thread will fail. I know from experience on z/OS how useful full dump can be, and I sorely miss it in Java. You wouldn't have to read the full dump yourself - a good postmortem debugger can do that work for you - displaying stack trace, locals, etc. Good logs can certainly save you.. if you have them, and you happen to log exactly what you need. But it's much easier to look at the memory and see what went wrong. (Personally, I think logs are great on the boundary, to describe inputs and outputs.) I also think that the exceptions are kind of wrong approach (compared to z/OS ABEND conditions), and it's not entirely true that failing loudly is a bad practice, but that's for another discussion.
      - jillesvangurp 8y ago
        Indeed a cultural thing in the sense that a JVM crashing is rare and exceptions are both common and non-fatal. In short, that's what logging is for and there are some awesome options for that in Java. We log all exceptions with stacktrace + full MDC (mapped diagnostic context) to a central logging cluster. Also, we start alerting if that happens. Most application servers will have a top level catch and a way of logging/handling exceptions (e.g. sending a 500 response for unhandled exceptions). The JVM crashing usually means you are hitting a jvm bug; it's not supposed to do that. It's been a while since I had that going on. Even out of memory exceptions tend to happen in a somewhat orderly fashion. It will print some message probably. Setting -XX:+HeapDumpOnOutOfMemoryError does what you want here: dumping the memory. Of course the use of that on an ephemeral disk in e.g. Docker is kind of limited. These heap dumps are super detailed though and you can dig through them using special tools. A common issue with docker is not setting limits correctly and then docker killing your process instead of it exiting the normal way. You can dump a stack trace of all threads to std out by sending kill -QUIT to the process (does not kill the process). Great for debugging deadlocks.
rockinghigh 8y ago
Yes, you can enable dumping the heap via command line arguments when starting the JVM: https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/clopts001.html https://docs.oracle.com/javase/8/docs/technotes/guides/troub...
- js8 8y ago
  Well, it seems to me that XX:HeapDumpOnOutOfMemoryError is only useful if you run out of memory, not for other errors such as NPE. The other one XX:OnError I am not really sure how to use with the JVM.. I mean can probably get the dump of the process on certain systems but I am not clear how to then proceed to get the actual dump of Java heap out of that. (And also, it's not clear what kind of cleanup happens inside JVM between the error and the execution of the command.)
  - wtetzner 8y ago
    I believe you can do something like this: -XX:OnError="jcmd %p GC.heap_dump /heap-dumps/%p.dump" Then it will put a heap dump into `/heap-dumps/<pid>.dump`. https://www.baeldung.com/java-heap-dump-capture https://www.baeldung.com/java-heap-dump-capture

by js8 8y ago

jillesvangurp 8y ago

You can turn this on but the dumps are kind of huge. Also, you can attach to a running JVM and get a memory dump that way. NPEs don't bring down the JVM though unless that happens in your main method of course, in which case the process exits normally.

js8 8y ago
What I mean, to produce the dump in the case of exceptions that percolate to top level and exit the process or thread (such as NPE). I am aware that I can attach, but that kind of defeats the ability to get the information in case of unexpected error (in production). Dumps might be huge, but is that really a problem on modern hardware? On z/OS this is doable (and common practice).
- coldtea 8y ago
  >What I mean, to produce the dump in the case of exceptions that percolate to top level and exit the process or thread (such as NPE). Why not have a top level try catch then?
  - js8 8y ago
    If you have a top level catch, then you lose the local context of the error (local variables, parameters, stack trace). I mean seriously, this is a solved problem on z/OS (the ABEND condition bypasses normal stack, so to speak), it has been solved for over 50 years. I don't see why JVM couldn't do the same thing - is it a technical problem? It seems like a purely cultural problem to me.
    - deepsun 8y ago
      Maybe because in Java it's usually considered a very bad practice to halt your program. All the web containers that run your code will catch any error, to prevent bringing down the whole web service if only a small piece of it fails (for example, cannot serve one URL, but can others). So instead, in Java, it's your responsibility to catch and correctly handle all the exceptions. Also, variables in the last function (where exception was raised from) are unlikely to give me understanding of the bug, because it's usually thrown somewhere deep in others code, and also I'm more interested in _why_ it happened, not just _what_ happened. We could dump all the variables in the stack trace, but I'm not sure I would like to read the whole dump of every variable in stack trace, that's just too much. I'd better write my code with proper logging, so I would be able to read the logs in order to understand not only _what_ happened, but _why_ it happened. With good logs you rarely even need to debug anything, I'd say 95% of the time it's pretty clear where to fix.
      - js8 8y ago
        Again, on z/OS, this is a solved problem. If the application runs inside an "application server", for instance CICS, you can still get the dump and the CICS can continue or even restart your app (or it can be restarted through other things). You can get the dump of the process even if just one thread will fail. I know from experience on z/OS how useful full dump can be, and I sorely miss it in Java. You wouldn't have to read the full dump yourself - a good postmortem debugger can do that work for you - displaying stack trace, locals, etc. Good logs can certainly save you.. if you have them, and you happen to log exactly what you need. But it's much easier to look at the memory and see what went wrong. (Personally, I think logs are great on the boundary, to describe inputs and outputs.) I also think that the exceptions are kind of wrong approach (compared to z/OS ABEND conditions), and it's not entirely true that failing loudly is a bad practice, but that's for another discussion.
    - jillesvangurp 8y ago
      Indeed a cultural thing in the sense that a JVM crashing is rare and exceptions are both common and non-fatal. In short, that's what logging is for and there are some awesome options for that in Java. We log all exceptions with stacktrace + full MDC (mapped diagnostic context) to a central logging cluster. Also, we start alerting if that happens. Most application servers will have a top level catch and a way of logging/handling exceptions (e.g. sending a 500 response for unhandled exceptions). The JVM crashing usually means you are hitting a jvm bug; it's not supposed to do that. It's been a while since I had that going on. Even out of memory exceptions tend to happen in a somewhat orderly fashion. It will print some message probably. Setting -XX:+HeapDumpOnOutOfMemoryError does what you want here: dumping the memory. Of course the use of that on an ephemeral disk in e.g. Docker is kind of limited. These heap dumps are super detailed though and you can dig through them using special tools. A common issue with docker is not setting limits correctly and then docker killing your process instead of it exiting the normal way. You can dump a stack trace of all threads to std out by sending kill -QUIT to the process (does not kill the process). Great for debugging deadlocks.

rockinghigh 8y ago

Yes, you can enable dumping the heap via command line arguments when starting the JVM: https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/clopts001.html https://docs.oracle.com/javase/8/docs/technotes/guides/troub...

js8 8y ago
Well, it seems to me that XX:HeapDumpOnOutOfMemoryError is only useful if you run out of memory, not for other errors such as NPE. The other one XX:OnError I am not really sure how to use with the JVM.. I mean can probably get the dump of the process on certain systems but I am not clear how to then proceed to get the actual dump of Java heap out of that. (And also, it's not clear what kind of cleanup happens inside JVM between the error and the execution of the command.)
- wtetzner 8y ago
  I believe you can do something like this: -XX:OnError="jcmd %p GC.heap_dump /heap-dumps/%p.dump" Then it will put a heap dump into `/heap-dumps/<pid>.dump`. https://www.baeldung.com/java-heap-dump-capture https://www.baeldung.com/java-heap-dump-capture