4 ms·

Hi cagenut, I'm a dev @ LinkedIn and I have regularly pushed services close to 30gb of heap, which serve close to 10k qps. After that it does get wonky, but I'm

Hi cagenut, I'm a dev @ LinkedIn and I have regularly pushed services close to 30gb of heap, which serve close to 10k qps. After that it does get wonky, but I'm surprised you're having trouble getting it that far. Mind emailing me your config? You should be using a CMS collector. -XX:+UseConcMarkSweepGC. Also try reducing CMSInitiatingOccupancyFraction until things stabilize. I've never had to drop it below 70%. Finally, ALWAYS make sure Xmx and Xms have the same value so the JVM doesn't try to resize itself.

cagenut 14y ago
Hi Josh, Thanks for offering to help. Your colleague Jacob Kessler wrote the closest thing I have to a bible on this: http://engineering.linkedin.com/26/tuning-java-garbage-collection-web-services http://engineering.linkedin.com/26/tuning-java-garbage-colle... That (and a truckload of trial & error) led me to this config: http://pastie.org/pastes/7130367/text?key=s8jajyejyhgz7x303shbmq http://pastie.org/pastes/7130367/text?key=s8jajyejyhgz7x303s... You're probably right, I probably could just keep dialing up xms/xmx/xmn, but even then what exactly to watch for in the gc log and how to do it tools-wise is a dark art. Exactly what field in what line to grep for and be mindful of when is a constant source of debate/confusion, and once you get up to a few hundred req/s visualgc just can't keep up so the graphs stop being useful. As you can see from my config link there I've tried throwing money at the problem but that didn't do it either. I'm at the point where I can hire an engineer to truly master this, or go bootleg parallel.
- joshhart 14y ago
  Jacob Kessler knows quite a bit about garbage collection :) Understanding the gc logs can be difficult. gcviewer is pretty good though at figuring it out. Always remember that garbage collection is slowest whenever the JVM has to copy objects from one generation to another or between survivor spaces. The logs print the sizes of the young gen, old gen, and survivor spaces. Sometimes after a collection you can see the sizes of these change dramatically - that means a big object has moved around. You can either change your code to use big objects less often or hold onto them for a shorter time or try to increase your survivor space size or new gen size. A major, major, major thing you are missing is -XX:+UseParNewGC. With the CMS collector, your new gen collection can be done in parallel, and the parallelization scales very well with the number of cores. Without ParNewGC, the service I work on would take over 1 second to garbage collect. With ParNewGC, it only takes 50ms.
  - cagenut 14y ago
    According to my notes from when I first put that together -XX:+UseParNewGC is automatically implied by +UseConcMarkSweepGC, for which my source was this very old web page about a very old jvm version (again, dark art): http://cybergav.in/2010/01/11/tuning-hotspot-jvm-1-4-2-with-cms-low-pause-garbage-collector/ http://cybergav.in/2010/01/11/tuning-hotspot-jvm-1-4-2-with-... Also, more concretely, this is the most frequent type of entry in my gc log: http://pastie.org/pastes/7142036/text?key=dkwvlvaaysk63rcajbp2na http://pastie.org/pastes/7142036/text?key=dkwvlvaaysk63rcajb... I (perhaps wrongly?) interpret the ParNew there to imply that's the young gen collector in use.

cagenut 14y ago

Hi Josh, Thanks for offering to help. Your colleague Jacob Kessler wrote the closest thing I have to a bible on this: http://engineering.linkedin.com/26/tuning-java-garbage-collection-web-services http://engineering.linkedin.com/26/tuning-java-garbage-colle... That (and a truckload of trial & error) led me to this config: http://pastie.org/pastes/7130367/text?key=s8jajyejyhgz7x303shbmq http://pastie.org/pastes/7130367/text?key=s8jajyejyhgz7x303s... You're probably right, I probably could just keep dialing up xms/xmx/xmn, but even then what exactly to watch for in the gc log and how to do it tools-wise is a dark art. Exactly what field in what line to grep for and be mindful of when is a constant source of debate/confusion, and once you get up to a few hundred req/s visualgc just can't keep up so the graphs stop being useful. As you can see from my config link there I've tried throwing money at the problem but that didn't do it either. I'm at the point where I can hire an engineer to truly master this, or go bootleg parallel.

joshhart 14y ago
Jacob Kessler knows quite a bit about garbage collection :) Understanding the gc logs can be difficult. gcviewer is pretty good though at figuring it out. Always remember that garbage collection is slowest whenever the JVM has to copy objects from one generation to another or between survivor spaces. The logs print the sizes of the young gen, old gen, and survivor spaces. Sometimes after a collection you can see the sizes of these change dramatically - that means a big object has moved around. You can either change your code to use big objects less often or hold onto them for a shorter time or try to increase your survivor space size or new gen size. A major, major, major thing you are missing is -XX:+UseParNewGC. With the CMS collector, your new gen collection can be done in parallel, and the parallelization scales very well with the number of cores. Without ParNewGC, the service I work on would take over 1 second to garbage collect. With ParNewGC, it only takes 50ms.
- cagenut 14y ago
  According to my notes from when I first put that together -XX:+UseParNewGC is automatically implied by +UseConcMarkSweepGC, for which my source was this very old web page about a very old jvm version (again, dark art): http://cybergav.in/2010/01/11/tuning-hotspot-jvm-1-4-2-with-cms-low-pause-garbage-collector/ http://cybergav.in/2010/01/11/tuning-hotspot-jvm-1-4-2-with-... Also, more concretely, this is the most frequent type of entry in my gc log: http://pastie.org/pastes/7142036/text?key=dkwvlvaaysk63rcajbp2na http://pastie.org/pastes/7142036/text?key=dkwvlvaaysk63rcajb... I (perhaps wrongly?) interpret the ParNew there to imply that's the young gen collector in use.