4 ms·

It does look like with some GC tuning (e.g. manually triggering GC's at a smaller interval than the Go automatic GC threshold) they might've mitigated the spike

It does look like with some GC tuning (e.g. manually triggering GC's at a smaller interval than the Go automatic GC threshold) they might've mitigated the spikes, although I don't think they would have gotten the level of perf improvement they did. Golang assembly code IME is not very optimized compared to Rust/C++. edit: reading comprehension skills are lacking, please see comment below for why I'm wrong

jholman 4y ago
I don't understand.... isn't this idea (triggering GC more often) explicitly discussed in the article?
- ceeplusplus 4y ago
  My understanding is they tried to tune the GC percent to make the automatic heuristic do GC sooner, but they didn't allocate enough to have that make a difference. However, Go has a way to manually trigger a GC which they could've set on a timer on a goroutine. If they weren't actually generating that much garbage then pause times should theoretically be pretty short if you're doing a GC every 5 seconds or something like that. That being said it's not something that 100% is guaranteed to fix the issue so maybe they did test this and just didn't mention it in the blog.
  - jholman 4y ago
    Okay, I see what you're saying about the timer. That isn't in TFA, agreed. But I still don't understand, because.... (NB: I'm not a GC expert, just a curious amateur, so my apologies if there are errors in the following, and the opportunity to be corrected in these errors is part of why I'm posting this.) Regarding the "not much garbage => theoretically times would be shorter", my understanding is that this is actually not how GC works. The GC time is a function of the size of the GC pool, because GC works by walking ("tracing") the tree of live references. So the only way to make GC faster is to have not less garbage, but less stuff allocated at all. Multi-generational GC works by dividing the whole pool into smaller pools, so that most GC passes only visit the high-churn nursery, but even then some GC passes need to read the TFA mentions this, where they say "the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire [thing we were keeping track of]". That is, they had virtually no garbage to collect, and that wasn't speeding up the GC. Which is consistent with how all tracing GC works, as far as I know. Comments/corrections/clarifications are requested!!
- mohanmcgeek 4y ago
  I remember when this article came out, everybody was pointing out the fact that they used a go version that was several releases older. Perhaps if the intent wasn't to convince their managers to let them write it in Rust, they would have tried using the latest Go version at the time?
  - tbillington 4y ago
    https://news.ycombinator.com/item?id=31021719 https://news.ycombinator.com/item?id=31021719
    - mohanmcgeek 4y ago
      Then why publish it at all? Not to mention, the article made no effort to establish that it's describing the world 2 years prior to this being written
mountainriver 4y ago
They actually would have mitigated it by upgrading their Go version, because the latest release at the time had a fix in the runtime that would have basically solved this. Turns out no one on the team actually looked into issues in the Go repo to see if it was being addressed. Looks like they just wanted to write Rust, which is fine Rust is cool, but let’s not deceive ourselves.
- steveklabnik 4y ago
  That is not what happened, they did not publish the blog post immediately after the transition, and those changes to the GC did not happen until after the port happened. Some people made assumptions about timeline that were incorrect, and then repeated. The discussion at the time on Reddit [1] mentions this. The general discussion as well talked about if the improvements, which were big in many cases, would have even improved this particular case. We’ll never truly know. That said it is important to recognize that Go’s GC has received significant upgrades over the years, and remember that what’s true in the past may not be true today. 1: https://www.reddit.com/r/programming/comments/eyuebc/why_discord_is_switching_from_go_to_rust/fgjsjxd/?context=3 https://www.reddit.com/r/programming/comments/eyuebc/why_dis...
  - mountainriver 4y ago
    The point is that they could have found the issue and seen that it was about to be released. That would be good engineering, bad engineering is when you don't find the root cause of your problem and see if its being worked on
    - steveklabnik 4y ago
      Good engineering is when you solve the problems you have. Sometimes there are multiple ways to solve a problem. Just because they did not choose the solution (which again, we're only speculating would actually solve the issue here, we don't have proof of that) you prefer does not make it poor engineering.

jholman 4y ago

I don't understand.... isn't this idea (triggering GC more often) explicitly discussed in the article?

ceeplusplus 4y ago
My understanding is they tried to tune the GC percent to make the automatic heuristic do GC sooner, but they didn't allocate enough to have that make a difference. However, Go has a way to manually trigger a GC which they could've set on a timer on a goroutine. If they weren't actually generating that much garbage then pause times should theoretically be pretty short if you're doing a GC every 5 seconds or something like that. That being said it's not something that 100% is guaranteed to fix the issue so maybe they did test this and just didn't mention it in the blog.
- jholman 4y ago
  Okay, I see what you're saying about the timer. That isn't in TFA, agreed. But I still don't understand, because.... (NB: I'm not a GC expert, just a curious amateur, so my apologies if there are errors in the following, and the opportunity to be corrected in these errors is part of why I'm posting this.) Regarding the "not much garbage => theoretically times would be shorter", my understanding is that this is actually not how GC works. The GC time is a function of the size of the GC pool, because GC works by walking ("tracing") the tree of live references. So the only way to make GC faster is to have not less garbage, but less stuff allocated at all. Multi-generational GC works by dividing the whole pool into smaller pools, so that most GC passes only visit the high-churn nursery, but even then some GC passes need to read the TFA mentions this, where they say "the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire [thing we were keeping track of]". That is, they had virtually no garbage to collect, and that wasn't speeding up the GC. Which is consistent with how all tracing GC works, as far as I know. Comments/corrections/clarifications are requested!!
mohanmcgeek 4y ago
I remember when this article came out, everybody was pointing out the fact that they used a go version that was several releases older. Perhaps if the intent wasn't to convince their managers to let them write it in Rust, they would have tried using the latest Go version at the time?
- tbillington 4y ago
  https://news.ycombinator.com/item?id=31021719 https://news.ycombinator.com/item?id=31021719
  - mohanmcgeek 4y ago
    Then why publish it at all? Not to mention, the article made no effort to establish that it's describing the world 2 years prior to this being written

mountainriver 4y ago

They actually would have mitigated it by upgrading their Go version, because the latest release at the time had a fix in the runtime that would have basically solved this. Turns out no one on the team actually looked into issues in the Go repo to see if it was being addressed. Looks like they just wanted to write Rust, which is fine Rust is cool, but let’s not deceive ourselves.

steveklabnik 4y ago
That is not what happened, they did not publish the blog post immediately after the transition, and those changes to the GC did not happen until after the port happened. Some people made assumptions about timeline that were incorrect, and then repeated. The discussion at the time on Reddit [1] mentions this. The general discussion as well talked about if the improvements, which were big in many cases, would have even improved this particular case. We’ll never truly know. That said it is important to recognize that Go’s GC has received significant upgrades over the years, and remember that what’s true in the past may not be true today. 1: https://www.reddit.com/r/programming/comments/eyuebc/why_discord_is_switching_from_go_to_rust/fgjsjxd/?context=3 https://www.reddit.com/r/programming/comments/eyuebc/why_dis...
- mountainriver 4y ago
  The point is that they could have found the issue and seen that it was about to be released. That would be good engineering, bad engineering is when you don't find the root cause of your problem and see if its being worked on
  - steveklabnik 4y ago
    Good engineering is when you solve the problems you have. Sometimes there are multiple ways to solve a problem. Just because they did not choose the solution (which again, we're only speculating would actually solve the issue here, we don't have proof of that) you prefer does not make it poor engineering.