4 ms·

i'm curious if allocating a memory ballast would have helped discord in this case? a one line patch in main() with something like ballast := make([]byte, 10<<

by kbar13 6y ago

i'm curious if allocating a memory ballast would have helped discord in this case? a one line patch in main() with something like ballast := make([]byte, 10<<30) might help their GC spikes, since garbage collection would only trigger when the heap size reaches 20GB, at which point the heap will be reduced back down to 10GB or so. https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...

aw1621107 6y ago
The GC cycles were triggered not due to heap size but due to the forced 2 minute max time between GC cycles [0], so a ballast wouldn't have helped here. [0]: https://github.com/golang/go/blob/0f66fb7b856b02497cf801ce72d80f375f53358b/src/runtime/proc.go#L5015 https://github.com/golang/go/blob/0f66fb7b856b02497cf801ce72...
- ttymck 6y ago
  This is really interesting. Are there not alternative runtimes, or configurable gc periods? My understanding of the JVM GC is limited, but it seems highly configurable: does it (or other languages/VMs) have an analogous construct, and is it (usually) configurable?
  - masklinn 6y ago
    The Go core team intensely dislikes tunables, so Go has very few of these (really the opposite of the JVM). There is a central issue for GC tunables, but it's mostly where the various concern GC have gone to die so far: https://github.com/golang/go/issues/42430 https://github.com/golang/go/issues/42430
masklinn 6y ago
> i'm curious if allocating a memory ballast would have helped discord in this case? No. Twitch's ballast is because > At steady state, our application was triggering ~8–10 garbage collection (GC) cycles a second (400–600 per minute). Go's GC triggers a collection every time the heap size doubles (until that has no return on investment and it just grows the heap). By creating a very large heap up-front they'd give themselves breathing room to avoid being just on the edge of (but remaining below) a resize, leading to continuous collections keeping them below. But Discord had the opposite issue: a large cache means a large heap size, in a steady state that would be extremely stable (since the heap size is a function of cache size, once the cache is full the only changes are replacing old keys for news). As a result, the only GC would be the mandatory periodic run, which would have to go through the entire (very large) heap, causing STW latency spikes every 2mn.

by kbar13 6y ago

aw1621107 6y ago

The GC cycles were triggered not due to heap size but due to the forced 2 minute max time between GC cycles [0], so a ballast wouldn't have helped here. [0]: https://github.com/golang/go/blob/0f66fb7b856b02497cf801ce72d80f375f53358b/src/runtime/proc.go#L5015 https://github.com/golang/go/blob/0f66fb7b856b02497cf801ce72...

ttymck 6y ago
This is really interesting. Are there not alternative runtimes, or configurable gc periods? My understanding of the JVM GC is limited, but it seems highly configurable: does it (or other languages/VMs) have an analogous construct, and is it (usually) configurable?
- masklinn 6y ago
  The Go core team intensely dislikes tunables, so Go has very few of these (really the opposite of the JVM). There is a central issue for GC tunables, but it's mostly where the various concern GC have gone to die so far: https://github.com/golang/go/issues/42430 https://github.com/golang/go/issues/42430

masklinn 6y ago

> i'm curious if allocating a memory ballast would have helped discord in this case? No. Twitch's ballast is because > At steady state, our application was triggering ~8–10 garbage collection (GC) cycles a second (400–600 per minute). Go's GC triggers a collection every time the heap size doubles (until that has no return on investment and it just grows the heap). By creating a very large heap up-front they'd give themselves breathing room to avoid being just on the edge of (but remaining below) a resize, leading to continuous collections keeping them below. But Discord had the opposite issue: a large cache means a large heap size, in a steady state that would be extremely stable (since the heap size is a function of cache size, once the cache is full the only changes are replacing old keys for news). As a result, the only GC would be the mandatory periodic run, which would have to go through the entire (very large) heap, causing STW latency spikes every 2mn.