3 ms·

That does actually sound like it could be scenario one too. If you have a lot of small requests, with only few requests active at the same time, but many reque

That does actually sound like it could be scenario one too. If you have a lot of small requests, with only few requests active at the same time, but many requests per second overall, with each making a few allocations, you will have a small live heap size, while quickly reaching the threshold for another GC. This way you get a lot of GC runs. Latency isn't affected too much because Go is quite good at keeping the stop-the-world's short. You might have interleaving application/stop-the-world in a 50/50 ratio of computation time (that's something you can diagnose very easily with go tool trace btw). Having a higher GOGC threshold might help a lot there, since it will make stop-the-world's less frequent, while keeping their duration mostly unchanged (as that scales proportionally to live heap size). That's obviously just a guess based on the limited info I have though.

morelisp 4y ago
> it will make stop-the-world's less frequent, while keeping their duration mostly unchanged (as that scales proportionally to live heap size). Go's STW phases are mostly not proportional to live heap size; iirc one never is and the other is proportional to something variable but only weakly correlated with heap size (cleaning up cached spans). It's hard to figure out what GP is describing exactly but I don't think GOGC alone would necessarily address that. If latency was still good it was probably not over-triggering STW nor dragging other threads into mark assist. I think they may have been hitting the pathological case described where stack roots were not being counted so a little allocation with a lot of stack-heavy goroutines could get confused for a lot of allocation, trigger a CPU-intensive mark phase, but neither clean much real memory in absolute terms nor effectively count a larger live set after. Prior to 1.18, GOGC might have to be set dangerously large avoid that.

morelisp 4y ago

> it will make stop-the-world's less frequent, while keeping their duration mostly unchanged (as that scales proportionally to live heap size). Go's STW phases are mostly not proportional to live heap size; iirc one never is and the other is proportional to something variable but only weakly correlated with heap size (cleaning up cached spans). It's hard to figure out what GP is describing exactly but I don't think GOGC alone would necessarily address that. If latency was still good it was probably not over-triggering STW nor dragging other threads into mark assist. I think they may have been hitting the pathological case described where stack roots were not being counted so a little allocation with a lot of stack-heavy goroutines could get confused for a lot of allocation, trigger a CPU-intensive mark phase, but neither clean much real memory in absolute terms nor effectively count a larger live set after. Prior to 1.18, GOGC might have to be set dangerously large avoid that.