4 ms·

It does sound like Discord's case was fairly extraordinary in terms of the degree of the spike: > We kept digging and learned the spikes were huge not because

by _bxg1 7y ago

It does sound like Discord's case was fairly extraordinary in terms of the degree of the spike: > We kept digging and learned the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references. So maybe this is one of those things that just doesn't come up in most cases? Maybe most services also generate enough garbage that that 2-minute maximum doesn't really come into play?

spullara 7y ago
Heap caches that keep things longer than a GC cycle are terrible under GC unless you have a collector in the new style like ZGC, Azul or Shenandoah.
- deleted 7y ago
  [deleted]
- sitkack 7y ago
  Systems with poor GC and the need to keep data for lifetimes greater than a request should have an easy to use off heap mechanism to prevent these problems. Often something like Redis is used as a shared cache that is invisible to the garbage collector, there is a natural key with a weak reference (by name) into a KV store. One could embed a KV store into an application that the GC can't scan into.
  - spullara 7y ago
    100%. In Java, you would often use OpenHFT's ChronicleMap for now and hopefully inline classes/records in Java 16 or so.
    - cgh 7y ago
      Ehcache has an efficient off-heap store: https://github.com/Terracotta-OSS/offheap-store/ https://github.com/Terracotta-OSS/offheap-store/ Doesn't Go have something like this available? It's an obvious thing for garbage-collected languages to have.
      - mappu 7y ago
        You can usually resort to `import C` and using `C.malloc` to get an unsafe.Pointer.
- mappu 7y ago
  What feature of "the new style" makes them more suitable in this case?
  - spullara 7y ago
    They have very short pause times even for very large heaps with lots of objects in them as they don't have to crawl the entire live tree when collecting.
wongarsu 7y ago
Games written in the Unity engine are (predominately) written in C#, a garbage collected language. Keeping large amounts of data around isn't that unusual since reading from disk is often prohibitively slow, and it's normal to minimize memory allocation/garbage generation (using object pools, caches etc), and manually trigger the GC in loading screens and in other opportune places (as easy as calling System.GC.Collect()). At 60 fps each frame is about 16ms. You do a lot in those 16ms, adding a 4ms garbage collection easily leads to dropping a frame. Of course whether that matters depends on the game, but Unity and C# seem to handle it well for the games that need tiny or no GC pauses. But (virtually) nobody is writing games in Go, so it's entirely possible that it's an unusual case in the Go ecosystem. Being an unsupported usecase is a great reason to switch language.
- _bxg1 7y ago
  Right; Go is purpose-built for writing web services, and web services tend to be pretty tolerant of (small) latency spikes because virtually anyone who's calling one is already expecting at least some latency
  - latch 7y ago
    > Go is purpose-built for writing web services Is this true? Go was built specifically for C++ developers, which, even when Go was first release, was a pretty unpopular language for writing web services (though maybe not at Google?). That a non-trivial number of Ruby/Python/Node developers switched was unexpected. (1) https://commandcenter.blogspot.com/2012/06/less-is-exponentially-more.html https://commandcenter.blogspot.com/2012/06/less-is-exponenti...
    - blondin 7y ago
      your quote is just corroborating what the reply is saying. go was written for web services which were written in c++ at google.
      - latch 7y ago
loeg 7y ago
A GC scan of a large LRU (or any large object graph) is expensive in CPU terms because the many of the pointers traversed will not be in any CPU cache. Memory access latency is extremely high relative to how fast CPUs can process cached data. You could maybe hack around the GC performance without destroying the aims of LRU eviction by batching additions to your LRU data structure to reduce the number of pointers by a factor of N. It's also possible that a Go BTree indexed by timestamp, with embedded data, would provide acceptable LRU performance and would be much friendlier on the cache. But it might also not have acceptable performance. And Go's lack of generic datastructures makes this trickier to implement vs Rust's BtreeMap provided out of the box.
jerf 7y ago
Yes, this is a maximally pessimal case for most forms of garbage collection. They don't say, but I would imagine these are very RAM-heavy systems. You can get up to 768GB right now on EC2. Fill that entire thing up with little tiny objects the size of usernames or IDs for users, or even merely 128GB systems or something, and the phase where you crawl the RAM to check references by necessity is going to be slow. This is something important to know before choosing a GC-based language for a task like this. I don't think "generating more garbage" would help, the problem is the scan is slow. If Discord was forced to do this in pure Go, there is a solution, which is basically to allocate a []byte or a set of []bytes, and then treat it as expanse of memory yourself, managing hashing, etc., basically, doing manual arena allocation yourself. GC would drop to basically zero in that case because the GC would only see the []byte slices, not all the contents as individual objects. You'll see this technique used in GC'd languages, including Java. But it's tricky code. At that point you've shucked off all the conveniences and features of modern languages and in terms of memory safety within the context of the byte expanses, you're writing in assembler. (You can't escape those arrays, which is still nice, but hardly the only possible issue.) Which is, of course, where Rust comes in. The tricky code you'd be writing in Go/Java/other GC'd language with tons of tricky bugs, you end up writing with compiler support and built-in static checking in Rust. I would imagine the Discord team evaluated the option of just grabbing some byte arrays and going to town, but it's fairly scary code to write. There are just too many ways to even describe for such code to end up having a 0.00001% bug that will result in something like the entire data structure getting intermittently trashed every six days on average or something, virtually impossible to pick up in testing and possibly even escaping canary deploys. Probably some other languages have libraries that could support this use case. I know Go doesn't ship with one and at first guess, I wouldn't expect to find one for Go, or one I would expect to stand up at this scale. Besides, honestly, at feature-set maturity limit for such a library, you just end up with "a non-GC'd inner platform" for your GC'd language, and may well be better off getting a real non-GC'd platform that isn't an inner platform [1]. I've learned to really hate inner platforms. By contrast... I'd bet this is fairly "boring" Rust code, and way, way less scary to deploy. [1]: https://en.wikipedia.org/wiki/Inner-platform_effect https://en.wikipedia.org/wiki/Inner-platform_effect
- _bxg1 7y ago

by _bxg1 7y ago

spullara 7y ago

Heap caches that keep things longer than a GC cycle are terrible under GC unless you have a collector in the new style like ZGC, Azul or Shenandoah.

deleted 7y ago
[deleted]
sitkack 7y ago
Systems with poor GC and the need to keep data for lifetimes greater than a request should have an easy to use off heap mechanism to prevent these problems. Often something like Redis is used as a shared cache that is invisible to the garbage collector, there is a natural key with a weak reference (by name) into a KV store. One could embed a KV store into an application that the GC can't scan into.
- spullara 7y ago
  100%. In Java, you would often use OpenHFT's ChronicleMap for now and hopefully inline classes/records in Java 16 or so.
  - cgh 7y ago
    Ehcache has an efficient off-heap store: https://github.com/Terracotta-OSS/offheap-store/ https://github.com/Terracotta-OSS/offheap-store/ Doesn't Go have something like this available? It's an obvious thing for garbage-collected languages to have.
    - mappu 7y ago
      You can usually resort to `import C` and using `C.malloc` to get an unsafe.Pointer.
mappu 7y ago
What feature of "the new style" makes them more suitable in this case?
- spullara 7y ago
  They have very short pause times even for very large heaps with lots of objects in them as they don't have to crawl the entire live tree when collecting.

wongarsu 7y ago

Games written in the Unity engine are (predominately) written in C#, a garbage collected language. Keeping large amounts of data around isn't that unusual since reading from disk is often prohibitively slow, and it's normal to minimize memory allocation/garbage generation (using object pools, caches etc), and manually trigger the GC in loading screens and in other opportune places (as easy as calling System.GC.Collect()). At 60 fps each frame is about 16ms. You do a lot in those 16ms, adding a 4ms garbage collection easily leads to dropping a frame. Of course whether that matters depends on the game, but Unity and C# seem to handle it well for the games that need tiny or no GC pauses. But (virtually) nobody is writing games in Go, so it's entirely possible that it's an unusual case in the Go ecosystem. Being an unsupported usecase is a great reason to switch language.

_bxg1 7y ago
Right; Go is purpose-built for writing web services, and web services tend to be pretty tolerant of (small) latency spikes because virtually anyone who's calling one is already expecting at least some latency
- latch 7y ago
  > Go is purpose-built for writing web services Is this true? Go was built specifically for C++ developers, which, even when Go was first release, was a pretty unpopular language for writing web services (though maybe not at Google?). That a non-trivial number of Ruby/Python/Node developers switched was unexpected. (1) https://commandcenter.blogspot.com/2012/06/less-is-exponentially-more.html https://commandcenter.blogspot.com/2012/06/less-is-exponenti...
  - blondin 7y ago
    your quote is just corroborating what the reply is saying. go was written for web services which were written in c++ at google.
    - latch 7y ago

loeg 7y ago

A GC scan of a large LRU (or any large object graph) is expensive in CPU terms because the many of the pointers traversed will not be in any CPU cache. Memory access latency is extremely high relative to how fast CPUs can process cached data. You could maybe hack around the GC performance without destroying the aims of LRU eviction by batching additions to your LRU data structure to reduce the number of pointers by a factor of N. It's also possible that a Go BTree indexed by timestamp, with embedded data, would provide acceptable LRU performance and would be much friendlier on the cache. But it might also not have acceptable performance. And Go's lack of generic datastructures makes this trickier to implement vs Rust's BtreeMap provided out of the box.

jerf 7y ago

Yes, this is a maximally pessimal case for most forms of garbage collection. They don't say, but I would imagine these are very RAM-heavy systems. You can get up to 768GB right now on EC2. Fill that entire thing up with little tiny objects the size of usernames or IDs for users, or even merely 128GB systems or something, and the phase where you crawl the RAM to check references by necessity is going to be slow. This is something important to know before choosing a GC-based language for a task like this. I don't think "generating more garbage" would help, the problem is the scan is slow. If Discord was forced to do this in pure Go, there is a solution, which is basically to allocate a []byte or a set of []bytes, and then treat it as expanse of memory yourself, managing hashing, etc., basically, doing manual arena allocation yourself. GC would drop to basically zero in that case because the GC would only see the []byte slices, not all the contents as individual objects. You'll see this technique used in GC'd languages, including Java. But it's tricky code. At that point you've shucked off all the conveniences and features of modern languages and in terms of memory safety within the context of the byte expanses, you're writing in assembler. (You can't escape those arrays, which is still nice, but hardly the only possible issue.) Which is, of course, where Rust comes in. The tricky code you'd be writing in Go/Java/other GC'd language with tons of tricky bugs, you end up writing with compiler support and built-in static checking in Rust. I would imagine the Discord team evaluated the option of just grabbing some byte arrays and going to town, but it's fairly scary code to write. There are just too many ways to even describe for such code to end up having a 0.00001% bug that will result in something like the entire data structure getting intermittently trashed every six days on average or something, virtually impossible to pick up in testing and possibly even escaping canary deploys. Probably some other languages have libraries that could support this use case. I know Go doesn't ship with one and at first guess, I wouldn't expect to find one for Go, or one I would expect to stand up at this scale. Besides, honestly, at feature-set maturity limit for such a library, you just end up with "a non-GC'd inner platform" for your GC'd language, and may well be better off getting a real non-GC'd platform that isn't an inner platform [1]. I've learned to really hate inner platforms. By contrast... I'd bet this is fairly "boring" Rust code, and way, way less scary to deploy. [1]: https://en.wikipedia.org/wiki/Inner-platform_effect https://en.wikipedia.org/wiki/Inner-platform_effect

_bxg1 7y ago