4 ms·

Has anyone experienced a significant performance improvement just by switching GCs? We have an Elasticsearch cluster with G1GC and I’ve been wondering if it’s w

by vosper 7y ago

Has anyone experienced a significant performance improvement just by switching GCs? We have an Elasticsearch cluster with G1GC and I’ve been wondering if it’s worth the time to replace a small number of nodes with different GCs. We could swap out 10% of the G1GC nodes for ZGC nodes, for example.

Kognito 7y ago
Yes, we moved to G1 (before it was the default) and then on to Shenandoah in a stream processor doing a lot of in-memory processing, lots of small allocations. As we reached a certain threshold of in-memory objects, Parallel GC would choke up and we’d see GC pauses which could reach 1s every few seconds, totally unacceptable. No doubt we could have spent time optimising this but moving to G1 brought this down to rarely exceeding 100ms, Shenandoah brought this down further to typically <10ms.
breischl 7y ago
Yes, we had an app that would hold a large amount of data in memory as a cache, about 16GB out of a 24GB heap. Before G1GC we'd see brutal stop-the-world collection pauses. Switching to G1GC we'd get pauses usually under 100ms, occasionally around 200ms. Although if anything ever caused a full GC it would cause a 30+ second pause to process the whole heap. Never got around to trying any of the newer GCs before it was rearchitected.
mdasen 7y ago
I know that the Shenandoah people have tested with Elasticsearch and seen great improvements. https://d3i71xaburhd42.cloudfront.net/605efbc0fed86899dcbf8733884a9256493acb25/8-Table3-1.png https://d3i71xaburhd42.cloudfront.net/605efbc0fed86899dcbf87... Christine Flood is pretty great and she has some wonderful talks on YouTube if you're interested in how Shenandoah works. The image above comes from one of her papers on GC (https://www.semanticscholar.org/paper/Shenandoah%3A-An-open-source-concurrent-compacting-Flood-Kennke/605efbc0fed86899dcbf8733884a9256493acb250 https://www.semanticscholar.org/paper/Shenandoah%3A-An-open-...) and shows order-of-magnitude improvements in pause times with average pause times 1/10th that of G1 and total pause times 1/27th that of G1. Shenandoah and ZGC are also made for larger heaps and often datastore have larger heaps.
- vosper 7y ago
  Thanks for that, I’ll definitely check out the paper! I don’t know much about the JVM, or GCs, but it’s always interesting to learn.
the8472 7y ago
Performance is not precise enough. Throughput, latency, peak memory consumption. Those characteristics can also vary by workload, as TFA demonstrates. As far as latency goes the new collectors do perform better in my experience, but you should try and measure.
- vosper 7y ago
  By "latency" do you mean "how long an Elasticsearch query takes to return results"? Or do you mean "how long the GC pauses the application for"? I suppose they are related, to some extent, but it's probably the ES search query time we'd be interested in improving.
  - the8472 7y ago
    Latency is not a single number, it's a distribution. Your 99.9%tile latency will suffer when your GC has a 2s pause every 20 seconds, even when the mean latency is low. But yes, they also affect mean latency.
  - paulddraper 7y ago
    GC latency, which is the commonly used term for the duration of GC interruption. > The period of time during which the garbage collector is active [and threads are stopped] is referred to as its latency. [1] This does of course affects the distribution of response latency. High GC latency causes more varying response times (though not necessarily higher average response time). [1] https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/latency https://docs.microsoft.com/en-us/dotnet/standard/garbage-col...
daxfohl 7y ago
On CLR we moved our server from server GC to client GC. It reduced efficiency, but fixed the problem we were having with long GC pauses. So yes, and it depends what you want to do.

by vosper 7y ago

Kognito 7y ago

Yes, we moved to G1 (before it was the default) and then on to Shenandoah in a stream processor doing a lot of in-memory processing, lots of small allocations. As we reached a certain threshold of in-memory objects, Parallel GC would choke up and we’d see GC pauses which could reach 1s every few seconds, totally unacceptable. No doubt we could have spent time optimising this but moving to G1 brought this down to rarely exceeding 100ms, Shenandoah brought this down further to typically <10ms.

breischl 7y ago

Yes, we had an app that would hold a large amount of data in memory as a cache, about 16GB out of a 24GB heap. Before G1GC we'd see brutal stop-the-world collection pauses. Switching to G1GC we'd get pauses usually under 100ms, occasionally around 200ms. Although if anything ever caused a full GC it would cause a 30+ second pause to process the whole heap. Never got around to trying any of the newer GCs before it was rearchitected.

mdasen 7y ago

I know that the Shenandoah people have tested with Elasticsearch and seen great improvements. https://d3i71xaburhd42.cloudfront.net/605efbc0fed86899dcbf8733884a9256493acb25/8-Table3-1.png https://d3i71xaburhd42.cloudfront.net/605efbc0fed86899dcbf87... Christine Flood is pretty great and she has some wonderful talks on YouTube if you're interested in how Shenandoah works. The image above comes from one of her papers on GC (https://www.semanticscholar.org/paper/Shenandoah%3A-An-open-source-concurrent-compacting-Flood-Kennke/605efbc0fed86899dcbf8733884a9256493acb250 https://www.semanticscholar.org/paper/Shenandoah%3A-An-open-...) and shows order-of-magnitude improvements in pause times with average pause times 1/10th that of G1 and total pause times 1/27th that of G1. Shenandoah and ZGC are also made for larger heaps and often datastore have larger heaps.

vosper 7y ago
Thanks for that, I’ll definitely check out the paper! I don’t know much about the JVM, or GCs, but it’s always interesting to learn.

the8472 7y ago

Performance is not precise enough. Throughput, latency, peak memory consumption. Those characteristics can also vary by workload, as TFA demonstrates. As far as latency goes the new collectors do perform better in my experience, but you should try and measure.

vosper 7y ago
By "latency" do you mean "how long an Elasticsearch query takes to return results"? Or do you mean "how long the GC pauses the application for"? I suppose they are related, to some extent, but it's probably the ES search query time we'd be interested in improving.
- the8472 7y ago
  Latency is not a single number, it's a distribution. Your 99.9%tile latency will suffer when your GC has a 2s pause every 20 seconds, even when the mean latency is low. But yes, they also affect mean latency.
- paulddraper 7y ago
  GC latency, which is the commonly used term for the duration of GC interruption. > The period of time during which the garbage collector is active [and threads are stopped] is referred to as its latency. [1] This does of course affects the distribution of response latency. High GC latency causes more varying response times (though not necessarily higher average response time). [1] https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/latency https://docs.microsoft.com/en-us/dotnet/standard/garbage-col...

daxfohl 7y ago

On CLR we moved our server from server GC to client GC. It reduced efficiency, but fixed the problem we were having with long GC pauses. So yes, and it depends what you want to do.