8 ms·

Everything I Ever Learned About JVM Performance Tuning At Twitter

146 points by koevet 15y ago

sehugg 15y ago
A couple other things: -XX:+UseCompressedOops converts many 64-bit references into 32-bit, trading a bit of CPU for a lot of memory If you use a lot of threads, lower your stack size (-Xss) (no real method here, just drive some trucks over the bridge until it breaks, and that's your weight limit :P ) You don't want to swap! A good rule of thumb is to make your max heap (-Xmx) about 60% of total RAM if you only have 1 JVM running Adaptive GC tuning algorithms can become unstable in long-running processes. I like to use -XX:GCTimeRatio for the throughput collector which effectively turns off the adaptive stuff. CMSInitiatingOccupancyFraction/UseCMSInitiatingOccupancyOnlydoes the same thing for the concurrent mark-sweep colletor. Keep the ice pick handy :P
- VladRussian 15y ago
  >You don't want to swap! A good rule of thumb is to make your max heap (-Xmx) about 60% of total RAM if you only have 1 JVM running i'd prepend the advise with "use large pages, Luke". Set Xmx to almost 100% of large pages, while the total of large pages set to 60% of the total RAM in your case or whatever your application specific is. For non-filesystem-IO application (middleware between DB and client applications) in my case the 14G on 16G machine works fine.
  - wolf550e 15y ago
    Have you tried transparent hugepages in the newer kernel?
    - VladRussian 15y ago
      nope. Enterprise software, not on the bleeding edge.
rubashov 15y ago
If you're fighting with the JVM this much, why not just use C++? The problems were garbage collection, lack of control over memory layout, and bloated types. Tool for the job, no?
- gorset 15y ago
  The JVM is very good if you are able to handle or avoid the GC pauses. Unless you are running Azul's Zing JVM, some tuning is usually required for non-trivial systems. We have tuned our system to be very fast with low latency most of the time, and handle timeouts by using a load balancer in front of a few servers. The chance that all servers are having a GC pause at the same time can be made sufficiently small.
  - maximusprime 15y ago
    -Xincgc avoids pauses no?
    - mbell 15y ago
      In Java 6 -Xincgc simply translates to: -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalMode causes more frequent, smaller pauses. It was designed for low CPU core count, small heap size applications where your willing to pay a CPU premium to gain better response times. It is specifically not recommended for 2+ CPUs and/or large heap sizes. With large heap sizes you'll be paying a large CPU penalty and if your are using / freeing a lot of objects it may not be able to "keep up" with its short pauses resulting in the heap filling up quickly.
    - gorset 15y ago
      There's usually a tradeoff between many small pauses and few big ones (but even with CMS there will be a point where the JVM needs to do a full GC). The usual approach for low latency systems are to spend the GC pauses incrementally over time. But for us, even though we require consistent low latency, big pauses are perfectly fine because we can run several replicas of the system behind a load balancer. Not all systems can be designed to do this, and it usually implies a idempotent architecture where you can run the same transaction multiple times without any problems. But when it's possible, you can start guaranteeing low latency with a very high probability. Worst case: simply run the transaction on all instances and use the result from the first one that answers.
moonboots 15y ago
Good presentation - I hate slideshare though, randomly skips slides, requires flash, needs facebook login to download, etc.
- koevet 15y ago
  I have watched the presentation with my iPad so, no, Slideshare does not require Flash. I do agree on the suckiness of the FB marriage.
cshesse 15y ago
My understanding is that some people just restart the JVM periodically to avoid GC pauses. How does that compare for web applications?
- wolf550e 15y ago
  While the JVM is down because you have restarted it, presumably your load balancer is routing requests to another JVM/machine. Then the new JVM starts up and warms up until it's ready to receive requsts. A restart transformed a JVM with a fragmented old gen into a JVM with fresh old gen using X1 wall-clock seconds and Y1 cpu time/watts. A full GC transforms a JVM with a fragmented old gen into a JVM with fresh old gen using X2 wall-clock seconds and Y2 cpu time/watts. Have you ever compared X1,Y1 vs. X2,Y2?
  - jbri 15y ago
    Does the JVM finish up outstanding requests and ask the load balancer to please-don't-send-me-anything-for-a-bit before it does a full GC?
    - nl 15y ago
      No. I suppose in theory you could use JMX instrumentation and tie it to your load balancer, but I've never heard of it being done. (It's actually quite a good idea! hmm...)
    - wolf550e 15y ago
      If you trigger a full GC "from outside", the way you would trigger a restart, you'd perform whatever you do for a restart (stop accepting requests, finish accepted requests, be unavailable, start accepting requests). My point was that a full GC does not need to re-profile the code, re-JIT the code and warm data caches, like a newly restarted JVM does. Sending a user request to a JVM to be run in an interpreter with cold caches is not good for user satisfaction, so a newly restarted JVM needs to be given mock requests to warm it up (like a script for PGO static compilation). A JVM doing a full GC does not need all that to become fully ready, and if it has enough free RAM to defragment quickly, the process should be much more efficient.
bfrog 15y ago
Funnily enough erlang does not have these horrible GC issues people are seeing in .NET and the HotSpot JVM.
- derleth 15y ago
  Yeah, but Erlang is a fake language, and fake languages are known for doing impossible things. ;)
- wglb 15y ago
  You have my curiosity piqued. Why is that?
  - derleth 15y ago
    Apparently, the secret of Erlang garbage collection is that it's done per-thread, and the whole Erlang model is based on spawning many threads. As a result, each thread only has a few K of RAM to collect, as opposed to a bigger pool per thread for Java. http://prog21.dadgum.com/16.html http://prog21.dadgum.com/16.html
    - simcop2387 15y ago
      I imagine that some of it is also the almost entirely strictly functional design of the language. Some of the ways it makes you write things should make certain tasks like that easier for the compiler. It should be almost always known at compile time when it'll be able to reap things.
      - derleth 15y ago
        Right. For one thing, nothing in Erlang is mutable: To change something, you copy it and change the copy. This sounds like it should make generational garbage collection easier (to a first approximation, nothing in the older generation is being used) but I don't know.
jlarocco 15y ago
"A research project had to load the full follower graph in memory" Really? It had to? There was no possible alternative? As often as I get the "fail whale" page on Twitter, I'm always skeptical seeing them present stuff like this and release code. Maybe I'm just not grasping how large and popular Twitter is, but of the popular web services I use, Twitter fails more than all the others combined. Though it was an interesting read, it seems suspect that they're having so many more problems than everybody else.
- jQueryIsAwesome 15y ago
  > Though it was an interesting read, it seems suspect that they're having so many more problems than everybody else. So true; i don't even know what a "facebook overload" looks like because i have never had one.
xentronium 15y ago
Law of leaky abstractions at its best (or worst): you decide to use a vm and gc to abstract from manual memory management only to deal with gory details of memory padding and internals of jvm later. Ouch.
- gchpaco 15y ago
  The stock JVM is exceptionally bizarre about this because of the insane decision to pad all fields to memory boundaries (which means on a 32 bit machine a class with 10 individual bit fields consumes 40 bytes of memory instead of something sane, like, say, 2). It also has an enormous amount of header overhead; a smallint on a competent Lisp or Smalltalk image takes up, say, one 4 byte word and can use 30 bits of that for data. Java's Integer class which is similar in intent is (in stock JVM) somewhere between 12 and 32 depending on VM settings. I've never seen another VM quite like it. In a previous job we got burned pretty hard by the amount of sheer overhead in this and I had to spend about six months converting everything over to use raw int arrays and copy-on-write objects so we could fit a reasonable but not really huge dataset into memory. Despite this, Hotspot is very good and the stock JVM GC is very useful and it is reasonably straightforward to get visibility into the state of the heap and what is going on with the GC. I've never worked in a professional capacity that has anything comparable to VisualVM, although I gather Smalltalk and commercial Lisps have comparable offerings, and you only have to program in Ruby for a while to learn to appreciate the GC quality.
  - jules 15y ago
    The bits can't be in the same memory word for concurrency reasons. Setting a bit is not an atomic operation on most processors.
ananthrk 15y ago
Does anyone have a PDF copy of this? The slideshare download is broken for this presentation.
- js2 15y ago
  dropbox: http://dl.dropbox.com/u/2138120/twitter-jvmperformancetuning.pdf http://dl.dropbox.com/u/2138120/twitter-jvmperformancetuning... google docs: http://goo.gl/g7y02 http://goo.gl/g7y02

146 points by koevet 15y ago

A couple other things: -XX:+UseCompressedOops converts many 64-bit references into 32-bit, trading a bit of CPU for a lot of memory If you use a lot of threads, lower your stack size (-Xss) (no real method here, just drive some trucks over the bridge until it breaks, and that's your weight limit :P ) You don't want to swap! A good rule of thumb is to make your max heap (-Xmx) about 60% of total RAM if you only have 1 JVM running Adaptive GC tuning algorithms can become unstable in long-running processes. I like to use -XX:GCTimeRatio for the throughput collector which effectively turns off the adaptive stuff. CMSInitiatingOccupancyFraction/UseCMSInitiatingOccupancyOnlydoes the same thing for the concurrent mark-sweep colletor. Keep the ice pick handy :P

VladRussian 15y ago
>You don't want to swap! A good rule of thumb is to make your max heap (-Xmx) about 60% of total RAM if you only have 1 JVM running i'd prepend the advise with "use large pages, Luke". Set Xmx to almost 100% of large pages, while the total of large pages set to 60% of the total RAM in your case or whatever your application specific is. For non-filesystem-IO application (middleware between DB and client applications) in my case the 14G on 16G machine works fine.
- wolf550e 15y ago
  Have you tried transparent hugepages in the newer kernel?
  - VladRussian 15y ago
    nope. Enterprise software, not on the bleeding edge.

rubashov 15y ago

If you're fighting with the JVM this much, why not just use C++? The problems were garbage collection, lack of control over memory layout, and bloated types. Tool for the job, no?

gorset 15y ago
The JVM is very good if you are able to handle or avoid the GC pauses. Unless you are running Azul's Zing JVM, some tuning is usually required for non-trivial systems. We have tuned our system to be very fast with low latency most of the time, and handle timeouts by using a load balancer in front of a few servers. The chance that all servers are having a GC pause at the same time can be made sufficiently small.
- maximusprime 15y ago
  -Xincgc avoids pauses no?
  - mbell 15y ago
    In Java 6 -Xincgc simply translates to: -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalMode causes more frequent, smaller pauses. It was designed for low CPU core count, small heap size applications where your willing to pay a CPU premium to gain better response times. It is specifically not recommended for 2+ CPUs and/or large heap sizes. With large heap sizes you'll be paying a large CPU penalty and if your are using / freeing a lot of objects it may not be able to "keep up" with its short pauses resulting in the heap filling up quickly.
  - gorset 15y ago
    There's usually a tradeoff between many small pauses and few big ones (but even with CMS there will be a point where the JVM needs to do a full GC). The usual approach for low latency systems are to spend the GC pauses incrementally over time. But for us, even though we require consistent low latency, big pauses are perfectly fine because we can run several replicas of the system behind a load balancer. Not all systems can be designed to do this, and it usually implies a idempotent architecture where you can run the same transaction multiple times without any problems. But when it's possible, you can start guaranteeing low latency with a very high probability. Worst case: simply run the transaction on all instances and use the result from the first one that answers.

moonboots 15y ago

Good presentation - I hate slideshare though, randomly skips slides, requires flash, needs facebook login to download, etc.

koevet 15y ago
I have watched the presentation with my iPad so, no, Slideshare does not require Flash. I do agree on the suckiness of the FB marriage.

cshesse 15y ago

My understanding is that some people just restart the JVM periodically to avoid GC pauses. How does that compare for web applications?

wolf550e 15y ago
While the JVM is down because you have restarted it, presumably your load balancer is routing requests to another JVM/machine. Then the new JVM starts up and warms up until it's ready to receive requsts. A restart transformed a JVM with a fragmented old gen into a JVM with fresh old gen using X1 wall-clock seconds and Y1 cpu time/watts. A full GC transforms a JVM with a fragmented old gen into a JVM with fresh old gen using X2 wall-clock seconds and Y2 cpu time/watts. Have you ever compared X1,Y1 vs. X2,Y2?
- jbri 15y ago
  Does the JVM finish up outstanding requests and ask the load balancer to please-don't-send-me-anything-for-a-bit before it does a full GC?
  - nl 15y ago
    No. I suppose in theory you could use JMX instrumentation and tie it to your load balancer, but I've never heard of it being done. (It's actually quite a good idea! hmm...)
  - wolf550e 15y ago
    If you trigger a full GC "from outside", the way you would trigger a restart, you'd perform whatever you do for a restart (stop accepting requests, finish accepted requests, be unavailable, start accepting requests). My point was that a full GC does not need to re-profile the code, re-JIT the code and warm data caches, like a newly restarted JVM does. Sending a user request to a JVM to be run in an interpreter with cold caches is not good for user satisfaction, so a newly restarted JVM needs to be given mock requests to warm it up (like a script for PGO static compilation). A JVM doing a full GC does not need all that to become fully ready, and if it has enough free RAM to defragment quickly, the process should be much more efficient.

bfrog 15y ago

Funnily enough erlang does not have these horrible GC issues people are seeing in .NET and the HotSpot JVM.

derleth 15y ago
Yeah, but Erlang is a fake language, and fake languages are known for doing impossible things. ;)
wglb 15y ago
You have my curiosity piqued. Why is that?
- derleth 15y ago
  Apparently, the secret of Erlang garbage collection is that it's done per-thread, and the whole Erlang model is based on spawning many threads. As a result, each thread only has a few K of RAM to collect, as opposed to a bigger pool per thread for Java. http://prog21.dadgum.com/16.html http://prog21.dadgum.com/16.html
  - simcop2387 15y ago
    I imagine that some of it is also the almost entirely strictly functional design of the language. Some of the ways it makes you write things should make certain tasks like that easier for the compiler. It should be almost always known at compile time when it'll be able to reap things.
    - derleth 15y ago
      Right. For one thing, nothing in Erlang is mutable: To change something, you copy it and change the copy. This sounds like it should make generational garbage collection easier (to a first approximation, nothing in the older generation is being used) but I don't know.

jlarocco 15y ago

"A research project had to load the full follower graph in memory" Really? It had to? There was no possible alternative? As often as I get the "fail whale" page on Twitter, I'm always skeptical seeing them present stuff like this and release code. Maybe I'm just not grasping how large and popular Twitter is, but of the popular web services I use, Twitter fails more than all the others combined. Though it was an interesting read, it seems suspect that they're having so many more problems than everybody else.

jQueryIsAwesome 15y ago
> Though it was an interesting read, it seems suspect that they're having so many more problems than everybody else. So true; i don't even know what a "facebook overload" looks like because i have never had one.

xentronium 15y ago

Law of leaky abstractions at its best (or worst): you decide to use a vm and gc to abstract from manual memory management only to deal with gory details of memory padding and internals of jvm later. Ouch.

gchpaco 15y ago
The stock JVM is exceptionally bizarre about this because of the insane decision to pad all fields to memory boundaries (which means on a 32 bit machine a class with 10 individual bit fields consumes 40 bytes of memory instead of something sane, like, say, 2). It also has an enormous amount of header overhead; a smallint on a competent Lisp or Smalltalk image takes up, say, one 4 byte word and can use 30 bits of that for data. Java's Integer class which is similar in intent is (in stock JVM) somewhere between 12 and 32 depending on VM settings. I've never seen another VM quite like it. In a previous job we got burned pretty hard by the amount of sheer overhead in this and I had to spend about six months converting everything over to use raw int arrays and copy-on-write objects so we could fit a reasonable but not really huge dataset into memory. Despite this, Hotspot is very good and the stock JVM GC is very useful and it is reasonably straightforward to get visibility into the state of the heap and what is going on with the GC. I've never worked in a professional capacity that has anything comparable to VisualVM, although I gather Smalltalk and commercial Lisps have comparable offerings, and you only have to program in Ruby for a while to learn to appreciate the GC quality.
- jules 15y ago
  The bits can't be in the same memory word for concurrency reasons. Setting a bit is not an atomic operation on most processors.

ananthrk 15y ago

Does anyone have a PDF copy of this? The slideshare download is broken for this presentation.

js2 15y ago
dropbox: http://dl.dropbox.com/u/2138120/twitter-jvmperformancetuning.pdf http://dl.dropbox.com/u/2138120/twitter-jvmperformancetuning... google docs: http://goo.gl/g7y02 http://goo.gl/g7y02