5 ms·

> Second, even if the CPU profiler tells you the GC is consuming a lot of time, that doesn't mean it's taking it away from your app I have experienced the same

by kccqzy 4y ago

> Second, even if the CPU profiler tells you the GC is consuming a lot of time, that doesn't mean it's taking it away from your app I have experienced the same issue here. Our load balancer used CPU usage as a proxy for deciding how much traffic should be assigned when performing load balancing. When the app was written in Go, we consistently found that the GC is consuming a lot of CPU time even though all other metrics like request latency were very good, even in the microseconds range. This was the case even when the app was massively parallel with lots of goroutines. But the load balancer kept sloshing traffic around unnecessarily based on its observation that GC is consuming a lot of CPU time.

tdudzik 4y ago
> When the app was written in Go Did you rewrite it to something else?
- kccqzy 4y ago
  Yes we did. But the rewrite was not because of this issue.
  - silisili 4y ago
    Out of curiosity...which language did you choose and why? How did it turn out?
    - oorza 4y ago
      They wrote it in PHP4 and it currently routes 2/3 of the internet.
    - kccqzy 4y ago
      We chose C++. For our particular domain, most libraries were written in C++. Previously we used a lot of cgo and SWIG.
cube2222 4y ago
That does actually sound like it could be scenario one too. If you have a lot of small requests, with only few requests active at the same time, but many requests per second overall, with each making a few allocations, you will have a small live heap size, while quickly reaching the threshold for another GC. This way you get a lot of GC runs. Latency isn't affected too much because Go is quite good at keeping the stop-the-world's short. You might have interleaving application/stop-the-world in a 50/50 ratio of computation time (that's something you can diagnose very easily with go tool trace btw). Having a higher GOGC threshold might help a lot there, since it will make stop-the-world's less frequent, while keeping their duration mostly unchanged (as that scales proportionally to live heap size). That's obviously just a guess based on the limited info I have though.
- morelisp 4y ago
  > it will make stop-the-world's less frequent, while keeping their duration mostly unchanged (as that scales proportionally to live heap size). Go's STW phases are mostly not proportional to live heap size; iirc one never is and the other is proportional to something variable but only weakly correlated with heap size (cleaning up cached spans). It's hard to figure out what GP is describing exactly but I don't think GOGC alone would necessarily address that. If latency was still good it was probably not over-triggering STW nor dragging other threads into mark assist. I think they may have been hitting the pathological case described where stack roots were not being counted so a little allocation with a lot of stack-heavy goroutines could get confused for a lot of allocation, trigger a CPU-intensive mark phase, but neither clean much real memory in absolute terms nor effectively count a larger live set after. Prior to 1.18, GOGC might have to be set dangerously large avoid that.

by kccqzy 4y ago

tdudzik 4y ago

> When the app was written in Go Did you rewrite it to something else?

kccqzy 4y ago
Yes we did. But the rewrite was not because of this issue.
- silisili 4y ago
  Out of curiosity...which language did you choose and why? How did it turn out?
  - oorza 4y ago
    They wrote it in PHP4 and it currently routes 2/3 of the internet.
  - kccqzy 4y ago
    We chose C++. For our particular domain, most libraries were written in C++. Previously we used a lot of cgo and SWIG.

cube2222 4y ago

That does actually sound like it could be scenario one too. If you have a lot of small requests, with only few requests active at the same time, but many requests per second overall, with each making a few allocations, you will have a small live heap size, while quickly reaching the threshold for another GC. This way you get a lot of GC runs. Latency isn't affected too much because Go is quite good at keeping the stop-the-world's short. You might have interleaving application/stop-the-world in a 50/50 ratio of computation time (that's something you can diagnose very easily with go tool trace btw). Having a higher GOGC threshold might help a lot there, since it will make stop-the-world's less frequent, while keeping their duration mostly unchanged (as that scales proportionally to live heap size). That's obviously just a guess based on the limited info I have though.

morelisp 4y ago
> it will make stop-the-world's less frequent, while keeping their duration mostly unchanged (as that scales proportionally to live heap size). Go's STW phases are mostly not proportional to live heap size; iirc one never is and the other is proportional to something variable but only weakly correlated with heap size (cleaning up cached spans). It's hard to figure out what GP is describing exactly but I don't think GOGC alone would necessarily address that. If latency was still good it was probably not over-triggering STW nor dragging other threads into mark assist. I think they may have been hitting the pathological case described where stack roots were not being counted so a little allocation with a lot of stack-heavy goroutines could get confused for a lot of allocation, trigger a CPU-intensive mark phase, but neither clean much real memory in absolute terms nor effectively count a larger live set after. Prior to 1.18, GOGC might have to be set dangerously large avoid that.