6 ms·

I have the opposite problem, I can't get java to use "enough" memory. Every time we double it in production that leads to another few weeks of re-jiggering jvm

by cagenut 14y ago

I have the opposite problem, I can't get java to use "enough" memory. Every time we double it in production that leads to another few weeks of re-jiggering jvm flags and ratios and grepping&graphing the garbage collection log or staring at visualgc for days. We're thinking of giving up on scaling above 16gb/jvm and going to something crazy like 16x2gb jvms. Treat them like the worlds biggest mongrel and let the loadbalancer sort it out.

ivom2gi 14y ago
That is indeed our experience as well. Going beyond 8-12GB of heap starts to produce all kind of weird side effects. So our best recommendation is to scale vertically on the heap size ...
theatrus2 14y ago
Note that there is a cliff at about 20GB with +CompressedOoops, as the larger heap makes object reference compression impossible, so references return to a full machine word (8 bytes), which depending on your workload and code, can be significant.
dcminter 14y ago
It does irritate me that the Sun JVMs do not offer any flag to specify the heap size along the lines of "limited only by the environment" rather than obliging an upper limit to be specified.
pdog 14y ago
Honestly, if you really need your JVM to scale above 16 GBs of heap memory, you should go for something like the Zing JVM[1] from Azul Systems. However, the standard JVMs should suffice for 99% of workloads. [1]: http://www.azulsystems.com/products/zing/virtual-machine http://www.azulsystems.com/products/zing/virtual-machine
- aut0mat0n1c 14y ago
  Huh.. thanks for this. Never heard of zing, looks worth a test drive.
joshhart 14y ago
Hi cagenut, I'm a dev @ LinkedIn and I have regularly pushed services close to 30gb of heap, which serve close to 10k qps. After that it does get wonky, but I'm surprised you're having trouble getting it that far. Mind emailing me your config? You should be using a CMS collector. -XX:+UseConcMarkSweepGC. Also try reducing CMSInitiatingOccupancyFraction until things stabilize. I've never had to drop it below 70%. Finally, ALWAYS make sure Xmx and Xms have the same value so the JVM doesn't try to resize itself.
- cagenut 14y ago
  Hi Josh, Thanks for offering to help. Your colleague Jacob Kessler wrote the closest thing I have to a bible on this: http://engineering.linkedin.com/26/tuning-java-garbage-collection-web-services http://engineering.linkedin.com/26/tuning-java-garbage-colle... That (and a truckload of trial & error) led me to this config: http://pastie.org/pastes/7130367/text?key=s8jajyejyhgz7x303shbmq http://pastie.org/pastes/7130367/text?key=s8jajyejyhgz7x303s... You're probably right, I probably could just keep dialing up xms/xmx/xmn, but even then what exactly to watch for in the gc log and how to do it tools-wise is a dark art. Exactly what field in what line to grep for and be mindful of when is a constant source of debate/confusion, and once you get up to a few hundred req/s visualgc just can't keep up so the graphs stop being useful. As you can see from my config link there I've tried throwing money at the problem but that didn't do it either. I'm at the point where I can hire an engineer to truly master this, or go bootleg parallel.
  - joshhart 14y ago
    Jacob Kessler knows quite a bit about garbage collection :) Understanding the gc logs can be difficult. gcviewer is pretty good though at figuring it out. Always remember that garbage collection is slowest whenever the JVM has to copy objects from one generation to another or between survivor spaces. The logs print the sizes of the young gen, old gen, and survivor spaces. Sometimes after a collection you can see the sizes of these change dramatically - that means a big object has moved around. You can either change your code to use big objects less often or hold onto them for a shorter time or try to increase your survivor space size or new gen size. A major, major, major thing you are missing is -XX:+UseParNewGC. With the CMS collector, your new gen collection can be done in parallel, and the parallelization scales very well with the number of cores. Without ParNewGC, the service I work on would take over 1 second to garbage collect. With ParNewGC, it only takes 50ms.

by cagenut 14y ago

ivom2gi 14y ago

That is indeed our experience as well. Going beyond 8-12GB of heap starts to produce all kind of weird side effects. So our best recommendation is to scale vertically on the heap size ...

theatrus2 14y ago

Note that there is a cliff at about 20GB with +CompressedOoops, as the larger heap makes object reference compression impossible, so references return to a full machine word (8 bytes), which depending on your workload and code, can be significant.

dcminter 14y ago

It does irritate me that the Sun JVMs do not offer any flag to specify the heap size along the lines of "limited only by the environment" rather than obliging an upper limit to be specified.

pdog 14y ago

Honestly, if you really need your JVM to scale above 16 GBs of heap memory, you should go for something like the Zing JVM[1] from Azul Systems. However, the standard JVMs should suffice for 99% of workloads. [1]: http://www.azulsystems.com/products/zing/virtual-machine http://www.azulsystems.com/products/zing/virtual-machine

aut0mat0n1c 14y ago
Huh.. thanks for this. Never heard of zing, looks worth a test drive.

joshhart 14y ago

Hi cagenut, I'm a dev @ LinkedIn and I have regularly pushed services close to 30gb of heap, which serve close to 10k qps. After that it does get wonky, but I'm surprised you're having trouble getting it that far. Mind emailing me your config? You should be using a CMS collector. -XX:+UseConcMarkSweepGC. Also try reducing CMSInitiatingOccupancyFraction until things stabilize. I've never had to drop it below 70%. Finally, ALWAYS make sure Xmx and Xms have the same value so the JVM doesn't try to resize itself.

cagenut 14y ago
Hi Josh, Thanks for offering to help. Your colleague Jacob Kessler wrote the closest thing I have to a bible on this: http://engineering.linkedin.com/26/tuning-java-garbage-collection-web-services http://engineering.linkedin.com/26/tuning-java-garbage-colle... That (and a truckload of trial & error) led me to this config: http://pastie.org/pastes/7130367/text?key=s8jajyejyhgz7x303shbmq http://pastie.org/pastes/7130367/text?key=s8jajyejyhgz7x303s... You're probably right, I probably could just keep dialing up xms/xmx/xmn, but even then what exactly to watch for in the gc log and how to do it tools-wise is a dark art. Exactly what field in what line to grep for and be mindful of when is a constant source of debate/confusion, and once you get up to a few hundred req/s visualgc just can't keep up so the graphs stop being useful. As you can see from my config link there I've tried throwing money at the problem but that didn't do it either. I'm at the point where I can hire an engineer to truly master this, or go bootleg parallel.
- joshhart 14y ago
  Jacob Kessler knows quite a bit about garbage collection :) Understanding the gc logs can be difficult. gcviewer is pretty good though at figuring it out. Always remember that garbage collection is slowest whenever the JVM has to copy objects from one generation to another or between survivor spaces. The logs print the sizes of the young gen, old gen, and survivor spaces. Sometimes after a collection you can see the sizes of these change dramatically - that means a big object has moved around. You can either change your code to use big objects less often or hold onto them for a shorter time or try to increase your survivor space size or new gen size. A major, major, major thing you are missing is -XX:+UseParNewGC. With the CMS collector, your new gen collection can be done in parallel, and the parallelization scales very well with the number of cores. Without ParNewGC, the service I work on would take over 1 second to garbage collect. With ParNewGC, it only takes 50ms.