7 ms·

Seems like you were hitting: runtime: Large maps cause significant GC pauses #9477 [0] Looks like this issue was resolved for maps that don't contain pointers

by rvcdbn 7y ago

Seems like you were hitting: runtime: Large maps cause significant GC pauses #9477 [0] Looks like this issue was resolved for maps that don't contain pointers by [1]. From the article, sounds like the map keys were strings (which do contain pointers, so the map would need to be scanned by the GC). If pointers in the map keys and values could be avoided, it would have (if my understanding is correct) removed the need for the GC to scan the map. You could do this for example by replacing string keys with fixed size byte arrays. Curious if you experimented this approach? [0] https://github.com/golang/go/issues/9477 https://github.com/golang/go/issues/9477 [1] https://go-review.googlesource.com/c/go/+/3288 https://go-review.googlesource.com/c/go/+/3288

ryuukk_ 7y ago
exactly, they use 3 years old version with the improvements made to runtime and that issue fixed, then is safe to say that GO is much faster than rust, based on their graphs
jasondclinton 7y ago
Finding out if that does resolve the author's issue would be interesting but I'm not sure that that would be particularly supportive data in favor of Go. If anything it would reinforce the downsides of Go's GC implementation: prone sudden pitfalls only avoidable with obtuse, error-prone fiddling that makes the code more complex. After spending weeks fighting with Java's GC tuning for a similar production service tail latency problem, I wouldn't want to be caught having to do that again.
- masklinn 7y ago
  The good news are that Go's GC has basically no tunables, so you wouldn't have spent weeks on that. The bad news is that it has basically no tunables so if it's a tuning issue you're either fucked or have to put "tuning" hacks right into the code if you find any that works (e.g. twitch's "memory ballast" to avoid overly aggressive GC runs: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...)
  - PeterCorless 7y ago
    There are tradeoffs with all languages. C++ avoids the GC, but you then have to make sure you know how to avoid the common pitfalls of that language. We use C++ at Scylla (saw that we got a shout-out in the blog! Woot!) but it's not like there isn't a whole industry about writing blogs avoiding C++ pitfalls. C++ pitfalls through the years... • https://www.horstmann.com/cpp/pitfalls.html https://www.horstmann.com/cpp/pitfalls.html (1997) • https://stackoverflow.com/questions/30373/what-c-pitfalls-should-i-avoid https://stackoverflow.com/questions/30373/what-c-pitfalls-sh... (2008) • http://blog.davidecoppola.com/2013/09/cpp-pitfalls/ http://blog.davidecoppola.com/2013/09/cpp-pitfalls/ (2013) • https://www.typemock.com/pitfalls-c/ https://www.typemock.com/pitfalls-c/ (2018) I am not saying any of these (Go, Rust, C++, or even Java) are "right" or "wrong" per se, because that determination is situational. Are you trying to optimize for performance, for code safety, for taking advantage of specific OS hooks, or oppositely, to be generically deployable across OSes, or for ease of development? For the devs at Scylla, the core DB code is C++. Some of our drivers and utilities are Golang (like our shard aware driver). There's also a Cassandra Rust driver — it'd be sweet if someone wants to make it shard-aware for Scylla!
    - xb95 7y ago
      (Discord infra person here.) Actually we didn't update the reference to Cassandra in the article -- the read states workload is now on Scylla too, as of last week. ;) We'll be writing up a blog post on our migration with Scylla at some point in the next few months, but we've been super happy with it. I replaced our TokuMX cluster with it and it's faster, more reliable, _and_ cheaper (including the support contract). Pretty great for us.
      - PeterCorless 7y ago
        Woot! Go you! (Or Rust you! Whichever you prefer!)
      - mathw 7y ago
        What a glorious combination of things! What a shame faster, more reliable and cheaper don't usually go together, but that's the challenge all developers face...
- hinkley 7y ago
  The common factor in most of my decisions to look for a new job has been realizing that I feel like a very highly compensated janitor instead of a developer. Once I spend even the plurality of my time cleaning up messes instead of doing something new (and there are ways to do both), then all the life is sucked out of me and I just have to escape. Telling me that I have to keep using a tool with known issues that we have to process or patches to fix would be super frustrating. And the more times we stumble over that problem the worse my confirmation bias will be. Even if the new solution has a bunch of other problems, the set that is making someone unhappy is the one that will cause them to switch teams or quit. This is one area where management is in a tough spot with respect to rewrites. Rewrites don't often fix many things, but if you suspect they're the only thing between you and massive employee turnover, you're between a rock and a hard place. The product is going to change dramatically, regardless of what decision you make.
  - outworlder 7y ago
    While I completely agree with the "janitor" sentiment... and for Newton's sake I feel like Wall-E daily... > Telling me that I have to keep using a tool with known issues that we have to process or patches to fix would be super frustrating. All tools have known issues. It's just that some have way more issues than others. And some may hurt more than others. Go has reached an interesting compromise. It has some elegant constructs and interesting design choices (like static compilation which also happens to be fast). The language is simple, so much so that you can learn the basics and start writing useful stuff in a weekend. But it is even more limiting than Java. A Lisp, this thing is not. You can't get very creative – which is an outstanding property for 'enterprises'. Boring, verbose code that makes you want to pull your teeth out is the name of the game. And I'm saying this as someone who dragged a team kicking and screaming from Python to Go. That's on them – no-one has written a single line of unit tests in years, so now they at least get a whiny compiler which will do basic sanity checks before things blow up in prod. Things still 'panic', but less frequently.
    - pstuart 7y ago
      I'll take boring over WTF code any day :-)
- rvcdbn 7y ago
  For any tracing GC, costs are going to be proportional to the number of pointers that need to be traced. So I would not call reducing the use of pointers to ameliorate a GC issue "obtuse, error-prone fiddling". On the contrary, it seems like one of the first approaches to look at when faced with the problem of too much GC work. Really all languages with tracing GC are at a disadvantage when you have a huge number of long-lived objects in the heap. The situation is improved with generational GC (which Go doesn't have) but the widespread use of off-heap data structures to solve the problem even in languages like Java with generational GC suggests this alone isn't a good enough solution. In Go's defense, I don't know another GC'ed language in which this optimization is present in the native map data structure.
  - pjmlp 7y ago
    Except that plenty of languages with tracing GC have also off GC memory allocation. Since you mention not knowing such languages, have a look at Eiffel, D, Modula-3, Active Oberon, Nim, C#/F# (specially after the latest improvements). Also Java will eventually follow the same idea as Eiffel (where inline classes are similar to expanded classes in Eiffel), and ByteBuffers can be off-GC heap.
gwbas1c 7y ago
Everything I've read indicates that RAM caches work poorly in a GC environment. The problem is that garbage collectors are optimized for applications that mostly have short-lived objects, and a small amount of long-lived objects. Things like large in-RAM LRU are basically the slowest thing for a garbage collector to do, because the mark-and-sweep phase always has to go through the entire cache, and because you're constantly generating garbage that needs to be cleaned.
- pkolaczk 7y ago
  A high number of short lived allocations is also a bad thing in a compacting GC environment, because every allocation gets you a reference to a memory region touched very long time ago and it is likely a cache miss. You would like to do an object pool to avoid this but then you run into a pitfall with long living objects, so there is really no good way out.
  - mcguire 7y ago
    ??? The allocation is going to be close to the last allocation, which was touched recently, no? The first allocation after a compaction wii be far from recent allocations, but close to the compacted objects?
    - pkolaczk 7y ago
      Close to the last allocation doesn't matter. What matters is the memory returned to the application - and this is memory that has been touched long ago and unlikely in cache. If your new generation size is larger than L3 cache it will have to be fetched from main memory for sure every time you start the next 64 bytes. I believe a smart cpu will notice the pattern and will prefetch to reduce cache miss latency. But a high allocation rate will use a lot of memory bandwidth and would thrash the caches. An extreme case of that problem happens when using GC in an app that gets swapped out. Performance drops to virtually zero then.
- sorokod 7y ago
  In this[1] video at about 32 min, mark there is a discussion on GC and apps that do caching. [1] https://www.youtube.com/watch?v=VCeHkcwfF9Q https://www.youtube.com/watch?v=VCeHkcwfF9Q
asimpletune 7y ago
Ok but in rust those pointers can just be borrowed obviating the need for gc at all.
- masklinn 7y ago
  Given it's a cache the entries would not have an existing natural owner… except for the cache itself. There would be no need for a GC to traverse the entire map, but that's because rust doesn't use a GC.
  - falcolas 7y ago
    While Rust does not have a discrete runtime GC process, it does utilize reference counting for dynamic memory cleanup. So you could argue that they are still going to suffer some of the downsides of a GC'ed memory allocation. Some potential issues include non-deterministic object lifespan, and ensuring that any unsafe code they write which interacts with the cache does the "right thing" with the reference counts (potentially including de-allocation; I'm not sure what unsafe code needs to do when referencing reference counted boxes).
    - masklinn 7y ago
      > While Rust does not have a discrete runtime GC process, it does utilize reference counting for dynamic memory cleanup. That's so misleading as to essentially be a lie. Rust uses reference counting if and only if you opt into it via reference-counted pointers. Using Rc or Arc is not the normal or default course of action, and I'm not aware of any situation where it is ubiquitous. > So you could argue [nonsense] No, you really could not.
      - mcguire 7y ago
        On the other hand, Rust's RAII management model behaves similarly to a reference counting system where the counts are limited to 0 and 1 (well, for a loose approximation of the 0 state), right?
bearcherian 7y ago
The article also mentions the service was on go 1.9.2, which was released 10/2017. I'd be curious to see if the same issues exist on a build based on a more recent version of Go.
typical182 7y ago
Maybe that is what they hit... but it seems there is a pretty healthy chance they could have resolved this by upgrading to a more modern runtime. Go 1.9 is fairly old (1.14 is about to pop out), and there have been large improvements on tail latency for the Go GC over that period. One of the Go 1. 12 improvements in particular seems to at least symptomatically line up with what they described, at least at the level of detail covered in the blog post: https://golang.org/doc/go1.12#runtime https://golang.org/doc/go1.12#runtime “Go 1.12 significantly improves the performance of sweeping when a large fraction of the heap remains live.“
kristianp 7y ago
I was thinking that if their cache is just one large hash table, essentially an array of structs, the GC wouldn't need to scan it. What you say about strings contained in the map would explain their problems, however I don't see the reason for it. Wouldn't you make sure every identifier uses a fixed-length GUID or similar, which would be contained in such a struct used in the array-of-structs?

by rvcdbn 7y ago

ryuukk_ 7y ago

exactly, they use 3 years old version with the improvements made to runtime and that issue fixed, then is safe to say that GO is much faster than rust, based on their graphs

jasondclinton 7y ago

Finding out if that does resolve the author's issue would be interesting but I'm not sure that that would be particularly supportive data in favor of Go. If anything it would reinforce the downsides of Go's GC implementation: prone sudden pitfalls only avoidable with obtuse, error-prone fiddling that makes the code more complex. After spending weeks fighting with Java's GC tuning for a similar production service tail latency problem, I wouldn't want to be caught having to do that again.

masklinn 7y ago
The good news are that Go's GC has basically no tunables, so you wouldn't have spent weeks on that. The bad news is that it has basically no tunables so if it's a tuning issue you're either fucked or have to put "tuning" hacks right into the code if you find any that works (e.g. twitch's "memory ballast" to avoid overly aggressive GC runs: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...)
- PeterCorless 7y ago
  There are tradeoffs with all languages. C++ avoids the GC, but you then have to make sure you know how to avoid the common pitfalls of that language. We use C++ at Scylla (saw that we got a shout-out in the blog! Woot!) but it's not like there isn't a whole industry about writing blogs avoiding C++ pitfalls. C++ pitfalls through the years... • https://www.horstmann.com/cpp/pitfalls.html https://www.horstmann.com/cpp/pitfalls.html (1997) • https://stackoverflow.com/questions/30373/what-c-pitfalls-should-i-avoid https://stackoverflow.com/questions/30373/what-c-pitfalls-sh... (2008) • http://blog.davidecoppola.com/2013/09/cpp-pitfalls/ http://blog.davidecoppola.com/2013/09/cpp-pitfalls/ (2013) • https://www.typemock.com/pitfalls-c/ https://www.typemock.com/pitfalls-c/ (2018) I am not saying any of these (Go, Rust, C++, or even Java) are "right" or "wrong" per se, because that determination is situational. Are you trying to optimize for performance, for code safety, for taking advantage of specific OS hooks, or oppositely, to be generically deployable across OSes, or for ease of development? For the devs at Scylla, the core DB code is C++. Some of our drivers and utilities are Golang (like our shard aware driver). There's also a Cassandra Rust driver — it'd be sweet if someone wants to make it shard-aware for Scylla!
  - xb95 7y ago
    (Discord infra person here.) Actually we didn't update the reference to Cassandra in the article -- the read states workload is now on Scylla too, as of last week. ;) We'll be writing up a blog post on our migration with Scylla at some point in the next few months, but we've been super happy with it. I replaced our TokuMX cluster with it and it's faster, more reliable, _and_ cheaper (including the support contract). Pretty great for us.
    - PeterCorless 7y ago
      Woot! Go you! (Or Rust you! Whichever you prefer!)
    - mathw 7y ago
      What a glorious combination of things! What a shame faster, more reliable and cheaper don't usually go together, but that's the challenge all developers face...
hinkley 7y ago
The common factor in most of my decisions to look for a new job has been realizing that I feel like a very highly compensated janitor instead of a developer. Once I spend even the plurality of my time cleaning up messes instead of doing something new (and there are ways to do both), then all the life is sucked out of me and I just have to escape. Telling me that I have to keep using a tool with known issues that we have to process or patches to fix would be super frustrating. And the more times we stumble over that problem the worse my confirmation bias will be. Even if the new solution has a bunch of other problems, the set that is making someone unhappy is the one that will cause them to switch teams or quit. This is one area where management is in a tough spot with respect to rewrites. Rewrites don't often fix many things, but if you suspect they're the only thing between you and massive employee turnover, you're between a rock and a hard place. The product is going to change dramatically, regardless of what decision you make.
- outworlder 7y ago
  While I completely agree with the "janitor" sentiment... and for Newton's sake I feel like Wall-E daily... > Telling me that I have to keep using a tool with known issues that we have to process or patches to fix would be super frustrating. All tools have known issues. It's just that some have way more issues than others. And some may hurt more than others. Go has reached an interesting compromise. It has some elegant constructs and interesting design choices (like static compilation which also happens to be fast). The language is simple, so much so that you can learn the basics and start writing useful stuff in a weekend. But it is even more limiting than Java. A Lisp, this thing is not. You can't get very creative – which is an outstanding property for 'enterprises'. Boring, verbose code that makes you want to pull your teeth out is the name of the game. And I'm saying this as someone who dragged a team kicking and screaming from Python to Go. That's on them – no-one has written a single line of unit tests in years, so now they at least get a whiny compiler which will do basic sanity checks before things blow up in prod. Things still 'panic', but less frequently.
  - pstuart 7y ago
    I'll take boring over WTF code any day :-)
rvcdbn 7y ago
For any tracing GC, costs are going to be proportional to the number of pointers that need to be traced. So I would not call reducing the use of pointers to ameliorate a GC issue "obtuse, error-prone fiddling". On the contrary, it seems like one of the first approaches to look at when faced with the problem of too much GC work. Really all languages with tracing GC are at a disadvantage when you have a huge number of long-lived objects in the heap. The situation is improved with generational GC (which Go doesn't have) but the widespread use of off-heap data structures to solve the problem even in languages like Java with generational GC suggests this alone isn't a good enough solution. In Go's defense, I don't know another GC'ed language in which this optimization is present in the native map data structure.
- pjmlp 7y ago
  Except that plenty of languages with tracing GC have also off GC memory allocation. Since you mention not knowing such languages, have a look at Eiffel, D, Modula-3, Active Oberon, Nim, C#/F# (specially after the latest improvements). Also Java will eventually follow the same idea as Eiffel (where inline classes are similar to expanded classes in Eiffel), and ByteBuffers can be off-GC heap.

gwbas1c 7y ago

Everything I've read indicates that RAM caches work poorly in a GC environment. The problem is that garbage collectors are optimized for applications that mostly have short-lived objects, and a small amount of long-lived objects. Things like large in-RAM LRU are basically the slowest thing for a garbage collector to do, because the mark-and-sweep phase always has to go through the entire cache, and because you're constantly generating garbage that needs to be cleaned.

pkolaczk 7y ago
A high number of short lived allocations is also a bad thing in a compacting GC environment, because every allocation gets you a reference to a memory region touched very long time ago and it is likely a cache miss. You would like to do an object pool to avoid this but then you run into a pitfall with long living objects, so there is really no good way out.
- mcguire 7y ago
  ??? The allocation is going to be close to the last allocation, which was touched recently, no? The first allocation after a compaction wii be far from recent allocations, but close to the compacted objects?
  - pkolaczk 7y ago
    Close to the last allocation doesn't matter. What matters is the memory returned to the application - and this is memory that has been touched long ago and unlikely in cache. If your new generation size is larger than L3 cache it will have to be fetched from main memory for sure every time you start the next 64 bytes. I believe a smart cpu will notice the pattern and will prefetch to reduce cache miss latency. But a high allocation rate will use a lot of memory bandwidth and would thrash the caches. An extreme case of that problem happens when using GC in an app that gets swapped out. Performance drops to virtually zero then.
sorokod 7y ago
In this[1] video at about 32 min, mark there is a discussion on GC and apps that do caching. [1] https://www.youtube.com/watch?v=VCeHkcwfF9Q https://www.youtube.com/watch?v=VCeHkcwfF9Q

asimpletune 7y ago

Ok but in rust those pointers can just be borrowed obviating the need for gc at all.

masklinn 7y ago
Given it's a cache the entries would not have an existing natural owner… except for the cache itself. There would be no need for a GC to traverse the entire map, but that's because rust doesn't use a GC.
- falcolas 7y ago
  While Rust does not have a discrete runtime GC process, it does utilize reference counting for dynamic memory cleanup. So you could argue that they are still going to suffer some of the downsides of a GC'ed memory allocation. Some potential issues include non-deterministic object lifespan, and ensuring that any unsafe code they write which interacts with the cache does the "right thing" with the reference counts (potentially including de-allocation; I'm not sure what unsafe code needs to do when referencing reference counted boxes).
  - masklinn 7y ago
    > While Rust does not have a discrete runtime GC process, it does utilize reference counting for dynamic memory cleanup. That's so misleading as to essentially be a lie. Rust uses reference counting if and only if you opt into it via reference-counted pointers. Using Rc or Arc is not the normal or default course of action, and I'm not aware of any situation where it is ubiquitous. > So you could argue [nonsense] No, you really could not.
    - mcguire 7y ago
      On the other hand, Rust's RAII management model behaves similarly to a reference counting system where the counts are limited to 0 and 1 (well, for a loose approximation of the 0 state), right?

bearcherian 7y ago

The article also mentions the service was on go 1.9.2, which was released 10/2017. I'd be curious to see if the same issues exist on a build based on a more recent version of Go.

typical182 7y ago

Maybe that is what they hit... but it seems there is a pretty healthy chance they could have resolved this by upgrading to a more modern runtime. Go 1.9 is fairly old (1.14 is about to pop out), and there have been large improvements on tail latency for the Go GC over that period. One of the Go 1. 12 improvements in particular seems to at least symptomatically line up with what they described, at least at the level of detail covered in the blog post: https://golang.org/doc/go1.12#runtime https://golang.org/doc/go1.12#runtime “Go 1.12 significantly improves the performance of sweeping when a large fraction of the heap remains live.“

kristianp 7y ago

I was thinking that if their cache is just one large hash table, essentially an array of structs, the GC wouldn't need to scan it. What you say about strings contained in the map would explain their problems, however I don't see the reason for it. Wouldn't you make sure every identifier uses a fixed-length GUID or similar, which would be contained in such a struct used in the array-of-structs?