3 ms·

Greplin's code independence day: open source for Nagios, OOM debugging, and more

62 points by rwalker 15y ago

bsergean 15y ago
The first two links gave me a 404 on github...
koenigdavidmj 15y ago
Getting a 404 on greplin-nagios-utils.
- danicgross 15y ago
  give it a try now! apologies.
justinsb 15y ago
For the Java OOM analyzer, did you look at creating a heapdump automatically (using -XX:+HeapDumpOnOutOfMemoryError), and then using the Memory Analyzer Toolkit (http://www.eclipse.org/mat/ http://www.eclipse.org/mat/)? I can see that your approach is (presumably) much faster because it avoids the need to dump all the memory, but I've found heapdump + MAT to be incredibly powerful.
- rwalker 15y ago
  We did, but for basically anything over 1GB we found heap dumps to be prohibitively slow. Then, instead of failing for one request we'd basically end up killing the whole server.
  - justinsb 15y ago
    Definitely hear you on that one! It only really works if you can put up with a service going down for a while (e.g. not customer facing, or one back-end in a web cluster with a load balancer in front). I can see how that wouldn't work for Greplin. Looks very useful now that I better understand the context.
  - SpikeGronim 15y ago
    I've had good experience with 2GB heap dumps. Actually dumping it was, in my experience, pretty quick. It was essentially as fast as the machine could do that much sequential IO. You get a lot more detail, like what was in those Strings that took up the whole heap. That is usually invaluable for diagnosing what's really wrong. Actually analyzing a multi-GB heap dump takes 2-3x as much heap for the analyzer. The jhat tool that ships with the JVM is perfectly capable of analyzing a 2 GB heap if it has 6 GB of its own heap to play with.
nwmcsween 15y ago
I recommend https://www.icinga.org/ https://www.icinga.org/ over nagios, it's simply a fork with quite a few updates so greplin-nagios-utils should work on it as well.
e1ven 15y ago
Thank you! The Nagios utils look very cool. I write all my Nagios checks in Python locally, and the startup/stop through NRPE is getting to be a problem. I'd been thinking "Hrmm.. I bet I could use Tornado to fire these off inline.." and Bam, here it is ;) Seriously awesome.

62 points by rwalker 15y ago

bsergean 15y ago

The first two links gave me a 404 on github...

koenigdavidmj 15y ago

Getting a 404 on greplin-nagios-utils.

danicgross 15y ago
give it a try now! apologies.

justinsb 15y ago

For the Java OOM analyzer, did you look at creating a heapdump automatically (using -XX:+HeapDumpOnOutOfMemoryError), and then using the Memory Analyzer Toolkit (http://www.eclipse.org/mat/ http://www.eclipse.org/mat/)? I can see that your approach is (presumably) much faster because it avoids the need to dump all the memory, but I've found heapdump + MAT to be incredibly powerful.

rwalker 15y ago
We did, but for basically anything over 1GB we found heap dumps to be prohibitively slow. Then, instead of failing for one request we'd basically end up killing the whole server.
- justinsb 15y ago
  Definitely hear you on that one! It only really works if you can put up with a service going down for a while (e.g. not customer facing, or one back-end in a web cluster with a load balancer in front). I can see how that wouldn't work for Greplin. Looks very useful now that I better understand the context.
- SpikeGronim 15y ago
  I've had good experience with 2GB heap dumps. Actually dumping it was, in my experience, pretty quick. It was essentially as fast as the machine could do that much sequential IO. You get a lot more detail, like what was in those Strings that took up the whole heap. That is usually invaluable for diagnosing what's really wrong. Actually analyzing a multi-GB heap dump takes 2-3x as much heap for the analyzer. The jhat tool that ships with the JVM is perfectly capable of analyzing a 2 GB heap if it has 6 GB of its own heap to play with.

nwmcsween 15y ago

I recommend https://www.icinga.org/ https://www.icinga.org/ over nagios, it's simply a fork with quite a few updates so greplin-nagios-utils should work on it as well.

e1ven 15y ago

Thank you! The Nagios utils look very cool. I write all my Nagios checks in Python locally, and the startup/stop through NRPE is getting to be a problem. I'd been thinking "Hrmm.. I bet I could use Tornado to fire these off inline.." and Bam, here it is ;) Seriously awesome.