5 ms·

Here's an example of someone from Linked-In describing a "low latency" setup with a 6GiB heap. https://engineering.linkedin.com/garbage-collection/garbage-coll

by yminsky 11y ago

Here's an example of someone from Linked-In describing a "low latency" setup with a 6GiB heap. https://engineering.linkedin.com/garbage-collection/garbage-collection-optimization-high-throughput-and-low-latency-java-applications https://engineering.linkedin.com/garbage-collection/garbage-... I've heard similar things from folks at Twitter, IIRC. But I do find the whole thing kind of mysterious, I have to admit. I'd love to learn that I was wrong.

bjourne 11y ago
I don't think that person knows what he is talking about. Either you pick high-throughput or low-latency. You don't get both. They got the young generation collection pause down to 60 ms which is completely crap. :) The gc for Factor which I've been hacking on has a young generation pause of 2-3 ms (depending on object topology, generation sizes and so on). But the young generation is only 2mb so the number of gc pauses is significantly higher.
- chipsy 11y ago
  Although it's probably not at all what that author meant, there is a way in which throughput and latency have a kind of fractally-layered relationship. For example, if you have a distributed computation that requires all parts to be finished before any result is returned, your throughput is dependent on the average latency for each part being consistently low. If one of them gets stuck it becomes a huge bottleneck, and speeding up the rest won't matter. And at the lowest levels of optimization a similar concern arises when CPU throughput is ultimately concerned with avoiding unnecessary latency such as cache misses. For modern systems, latency seems to be increasingly important, as the fast paths have all grown various asynchronous aspects. For some systems 60ms pause might be fine, for others 2ms might be way too much. It's definitely a "what scale are you thinking about" kind of thing.
  - bjourne 11y ago
    What the author (and I) meant by "low-latency" is a gc were the work is very fairly distributed among the mutators allocation requests. The gc with highest throughput just allocates memory and never frees it. But then you sacrifice other properties such as extremely high memory usage..
- istvan__ 11y ago
  In fact low latency and high-throughput are usually best friends. You cannot maintain high throughput if your operations are taking longer. Also have a look to this picture. Using G1, doing few hundred MB/s. http://postimg.org/image/gms25ibnl/ http://postimg.org/image/gms25ibnl/
  - the8472 11y ago
    Throughput is generally measured in the fraction of CPU cycles spent on GCing. The Parallel Old Gen collector is more efficient in that regard (i.e. provides more compute-throughput) than CMS, Zing or G1, but it is not concurrent and thus you have longer STW pauses compare to the latter collectors. The concurrent collectors trade some throughput for latency by occupying additional threads for concurrent work. Due to the additional synchronization actions (more atomic instructions, read/write barriers, additional cleanups during shorter STW pauses) they are less efficient in overall CPU cycles spent. So it certainly is a tradeoff. Of course a collector that can burn more memory bandwidth and CPU cycles will generally lead to lower pause times, so in that sense increasing throughput of the collector is good for latency. But increasing the collector's throughput leaves less resources for your actual application, decreasing effective throughput.
- tmd83 11y ago
  I think what he meant is an application that is serving a lot of requests at low latency which would be the ideal scenario for a api/cache server so not much of crap. And I'm not sure why you are comparing a 6G new gen to a 2MB new gen. Do you mean to say that a 70ms GC for a 6G heap is too low? It is fairly possible to hit those or even lower range for a heap of that size depending on the data. I have even heard of people hitting even lower GC pause though I myself haven't been able to do that personally.
istvan__ 11y ago
I don't think that you are wrong, most of the JVM users are not forced to look into how GC works unless they are exposed to extreme high scale and load like LinkedIN, Twitter, etc. It is not uncommon to roll with 6GB+ heap. GC gets in your way if this high scale meets with low latency requirements for p99 (and above) latency, again which these guys care about a lot. I think the best approach to GC based memory management is what Erlang does, extremely limited scope, no global locking and tiny GC time. I am not entirely familiar how the OCaml VM works, just started to play with the environment. Also, my understanding is that OCaml is not for highly concurrent systems. Anyways it is kind of offtopic here. The summarize: - JVM GC details are extremely important for high throughput low latency systems at scale, as far as I know the G1 GC is used for predictable low latency compactions, and I can verify that with my experiments, having 10ms GC pauses - I think the Erlang approach is superior to garbage collected systems, but it requires no shared memory across your threads (or in the Erlang case processes), so the GC scope is tiny (and few other niceties in BEAM) More on Java G1: http://www.infoq.com/articles/G1-One-Garbage-Collector-To-Rule-Them-All http://www.infoq.com/articles/G1-One-Garbage-Collector-To-Ru... More about Erlang's GC here: http://prog21.dadgum.com/16.html http://prog21.dadgum.com/16.html

by yminsky 11y ago

bjourne 11y ago

I don't think that person knows what he is talking about. Either you pick high-throughput or low-latency. You don't get both. They got the young generation collection pause down to 60 ms which is completely crap. :) The gc for Factor which I've been hacking on has a young generation pause of 2-3 ms (depending on object topology, generation sizes and so on). But the young generation is only 2mb so the number of gc pauses is significantly higher.

chipsy 11y ago
Although it's probably not at all what that author meant, there is a way in which throughput and latency have a kind of fractally-layered relationship. For example, if you have a distributed computation that requires all parts to be finished before any result is returned, your throughput is dependent on the average latency for each part being consistently low. If one of them gets stuck it becomes a huge bottleneck, and speeding up the rest won't matter. And at the lowest levels of optimization a similar concern arises when CPU throughput is ultimately concerned with avoiding unnecessary latency such as cache misses. For modern systems, latency seems to be increasingly important, as the fast paths have all grown various asynchronous aspects. For some systems 60ms pause might be fine, for others 2ms might be way too much. It's definitely a "what scale are you thinking about" kind of thing.
- bjourne 11y ago
  What the author (and I) meant by "low-latency" is a gc were the work is very fairly distributed among the mutators allocation requests. The gc with highest throughput just allocates memory and never frees it. But then you sacrifice other properties such as extremely high memory usage..
istvan__ 11y ago
In fact low latency and high-throughput are usually best friends. You cannot maintain high throughput if your operations are taking longer. Also have a look to this picture. Using G1, doing few hundred MB/s. http://postimg.org/image/gms25ibnl/ http://postimg.org/image/gms25ibnl/
- the8472 11y ago
  Throughput is generally measured in the fraction of CPU cycles spent on GCing. The Parallel Old Gen collector is more efficient in that regard (i.e. provides more compute-throughput) than CMS, Zing or G1, but it is not concurrent and thus you have longer STW pauses compare to the latter collectors. The concurrent collectors trade some throughput for latency by occupying additional threads for concurrent work. Due to the additional synchronization actions (more atomic instructions, read/write barriers, additional cleanups during shorter STW pauses) they are less efficient in overall CPU cycles spent. So it certainly is a tradeoff. Of course a collector that can burn more memory bandwidth and CPU cycles will generally lead to lower pause times, so in that sense increasing throughput of the collector is good for latency. But increasing the collector's throughput leaves less resources for your actual application, decreasing effective throughput.
tmd83 11y ago
I think what he meant is an application that is serving a lot of requests at low latency which would be the ideal scenario for a api/cache server so not much of crap. And I'm not sure why you are comparing a 6G new gen to a 2MB new gen. Do you mean to say that a 70ms GC for a 6G heap is too low? It is fairly possible to hit those or even lower range for a heap of that size depending on the data. I have even heard of people hitting even lower GC pause though I myself haven't been able to do that personally.

istvan__ 11y ago

I don't think that you are wrong, most of the JVM users are not forced to look into how GC works unless they are exposed to extreme high scale and load like LinkedIN, Twitter, etc. It is not uncommon to roll with 6GB+ heap. GC gets in your way if this high scale meets with low latency requirements for p99 (and above) latency, again which these guys care about a lot. I think the best approach to GC based memory management is what Erlang does, extremely limited scope, no global locking and tiny GC time. I am not entirely familiar how the OCaml VM works, just started to play with the environment. Also, my understanding is that OCaml is not for highly concurrent systems. Anyways it is kind of offtopic here. The summarize: - JVM GC details are extremely important for high throughput low latency systems at scale, as far as I know the G1 GC is used for predictable low latency compactions, and I can verify that with my experiments, having 10ms GC pauses - I think the Erlang approach is superior to garbage collected systems, but it requires no shared memory across your threads (or in the Erlang case processes), so the GC scope is tiny (and few other niceties in BEAM) More on Java G1: http://www.infoq.com/articles/G1-One-Garbage-Collector-To-Rule-Them-All http://www.infoq.com/articles/G1-One-Garbage-Collector-To-Ru... More about Erlang's GC here: http://prog21.dadgum.com/16.html http://prog21.dadgum.com/16.html