5 ms·
This doesn't read like a "GC is slow" problem to me, more like a "GC doesn't cope well in the face of a highly concurrent allocation workload". That's pretty ba
by dboreham 9y ago
This doesn't read like a "GC is slow" problem to me, more like a "GC doesn't cope well in the face of a highly concurrent allocation workload". That's pretty bad though. As soon as you go down the rabbit hole of tweaking knobs to cajole GC into doing what you expect, darkness lies beyond..
- jchw 9y agoIt sounds like a bug in the GC for it to run so often, but even so, Go's GC has only one knob and they're adament about keeping it that way. There's no rabbit hole to fall into here.
- Twirrim 9y agoThere can be advantages (and ZOMG so much confusion) in allowing multiple knobs though. Java takes it to an extreme and we're somewhat in the territory of needing a black box opitimiser, but you can adjust the GC to suit a really wide variety of workloads and requirements. By leaving it to one knob, Google/Go is heavily limiting the uses of the language.
- jerf 9y agoAll GC algorithms have pessimal cases; I believe there is a mathematical proof of that, though I can never find it on the net through the chatter of all the other stuff about garbage collection. This is a pessimal case for the current Go GC, but that alone doesn't prove much because all algorithms have pessimal cases. (Even "manual memory management" can end up with pessimal cases for the automatic part, which is why you might see an improvement with your "manually managed" program by swapping malloc for jemalloc or something. What people call "manual memory management" still generally includes a lot of automatic stuff.) I believe Go does something like set a threshold of memory use. Once that memory usage is hit, a collection is run. After that collection is run, it uses the new accurate amount of memory used to set the next threshold. The threshold of course start small and have an exponential increase of some sort. When the program first starts, the threshold is set fairly low. If you immediately start allocating lots of things and keeping them, a common case, it grows to a sensible size in a reasonable number of allocations and sweeps (which as you are growing into your initial footprint, amortize out pretty cheap), and in a lot of cases you won't think much about the GC until you are really pushing performance. If, however, you write a program that isn't holding on to much RAM in the steady state, such as a crypto algorithm (which for all their algorithmic complexity typically don't take much RAM to run, as compared to, say, loading an image, and I don't know for sure but it's possible this ends up all stack-allocated), and that algorithm allocates a lot of short-term stuff, perhaps even just allocates for its final answer, then you get into a situation where you are quickly generating enough garbage to cross the initial low threshold, but the GC immediately cleans it all up when it runs, so when it is deciding on what the new threshold should be it still picks a low number, causing it to repeat the cycle indefinitely. Fixes will immediately leap to your mind, but remember you must evaluate your fixes holistically, not as if this is the only case in the world. If it were the only case in the world, it would be an easy problem, and Go never would have exhibited the problem in the first place. In particular, as you consider your "fix", consider also that a real program that used real crypto would be much less likely to see this pessimal case occur, because it is very likely the rest of the program would push the memory thresholds up to something sensible, so the fix of "do nothing" is actually quite likely to work well on non-artifical-benchmark programs. It is likely that the most obvious fix that comes to mind will result in making a program that is deliberately running some small server process or something appearing to leak memory after running for a long period of time, for instance. (I suspect quite strongly in this case that the correct "fix" is to do nothing.)
- nkurz 9y agoI suspect quite strongly in this case that the correct "fix" is to do nothing. What about a fix that gives the user optional control over the thresholds, or even just a hint as to the desired behavior? Keep the defaults as they are, but add a “pragma” that would set the minimum threshold for GC? Or it could go a long way just to have an option to set a minimum time between collections. While it could easily make things worse for certain situations, I much prefer the option to hang myself over an immutable default that periodically does so without my input or consent.
- coldtea 9y ago>in the face of a highly concurrent allocation workload Isn't this "highly concurrent" the main selling point of Go?
- eternalban 9y agoThe original PR line taken by Go team was that langauges like Java "communicate by sharing" whereas in Go you would "share by communicating". It wasn't so much about Green vs Native threading as it was about the core semantics of Goroutines and Channels. This original PR also had annoying (to people such as myself) aspect of poopooing Java in context of concurrency (of all things). As it happens [lol] facts asserted themselves and (at least some in) Go community saw the reason as to why "communicating by sharing" paradigm ever got traction. "It's performance, stupid", to paraphrase Slick Willy, not because other language designers were thoughtless. So now it is "idiomatic" to see Go code using both paradigms (locking & passing channels). The original matra was quietly dropped along "systems programming PL" bit somewhere along the line. This OP type of article, in an interesting way, continues this pattern of "let's do it this [obvious] way" in face of dealing with concurrency. No less an authority than Cliff Click has asserted that lock-free type algorithms pretty much require a GC. Concurrency and memory management seem to be zero-sum games. Root cause is clearly Physics, and not the mechanism devised to deal with the physical realities.
- dullgiulio 9y agoChannels pass ownership of allocated objects, they don't perform copies or allocations themselves (once you've allocated the channel and the objects you want to pass). I don't think the article says anything about concurrent allocations; in fact, the various signers don't need to share anything (mutable) to sign the requests they get.
- eternalban 9y ago> Channels pass ownership of allocated objects, they don't perform copies or allocations themselves (once you've allocated the channel and the objects you want to pass). And? Did I say they did? > I don't think the article says anything about concurrent allocations No it doesn't. Point was that it is not a general purpose approach. GCs are.
- lucozade 9y agoIt's the default setting that didn't cope well. The final setting seemed to get pretty decent performance. And if you only have one knob then the rabbit hole is pretty shallow. The question not answered (as it wasn't asked) is how many workload profiles can be catered for? But as a sample of one, this appears successful.
- saas_co_de 9y agoThe question though is does the setting that worked for this particular micro-benchmark actually work well for more generic workloads that have similar performance issues.
- lucozade 9y agoDoes it matter? If they code up their target workload and get near linear scaling with collection on 5000% or 12000% it's still a win. The important question is whether or not they can get the scaling with some setting.