4 ms·
We upgraded to 2.6.1 about a week ago and switched to using the new(ish) parallel(ish) garbage collector. I still can't tell what the impact has been. Claude C
by dang 9mo ago
We upgraded to 2.6.1 about a week ago and switched to using the new(ish) parallel(ish) garbage collector. I still can't tell what the impact has been.
Claude Code (which is a wizard at analyzing log files but also, I fear, an incorrigible curve-fitter) insisted that it was a real breakthrough and an excellent choice! On the other hand there was a major slowdown last night, ending in SBCL dying from heap exhaustion. I haven't had a chance to dig into that yet.
- stackghost 9mo ago>SBCL dying from heap exhaustion Due to hitting the cap, or to framentation? My understanding is the new parallel GC compacts the heap rather infrequently.
- dang 9mo agoIf by the cap you mean the heap size passed in as the --dynamic-space-size argument, it didn't hit the cap. It was using about 2/3 of that. > My understanding is the new parallel GC compacts the heap rather infrequently Can you explain more?
- stackghost 9mo agoI'm going to caveat this by stating up front that obviously HN's source code is not public so I don't know what your hot path looks like, and that I'm not a domain expert on garbage collection, but I do write a fair amount of lisp for SBCL. Immix-style collectors, like the new GC in SBCL, only compact on an opportunistic basis and so you get fragmentation pressure under load. In that situation, you might be well under the dynamic space size cap but if it can't find a large enough contiguous chunk of free heap it will still die. So, fragmentation would be my prime suspect given what you described.
- dang 9mo agoSorry for suddenly clinging to you for support but might we be better off using the older GC in that case?
- stackghost 9mo agoNo problem. You might be better off moving back, yes. My understanding of immix-style collection is that it divides the heap into blocks and lines. A block is only compacted/reused if every object in it is dead, and so if you mix lifetimes (i.e. lots of short-lived requests, medium-life sessions, long-life db connections/caches/interned symbols) then you tend to fill up blocks with a mix of short and long-lived objects as users log in and make requests. When the requests get de-allocated the session remains (because the user closed the tab but didn't log out, for example, so the session is still valid) and so you end up with a bunch of blocks that are partially occupied by long-lived objects, and this is what drives fragmentation because live objects don't get moved/compacted/de-fragged very often. Eventually you fill up your entire heap with partially-allocated blocks and there is no single contiguous span of memory large enough to fit a new allocation and the allocator shits its pants. So if that's what the HN backend looks like architecturally (mixed lifetimes), then you'd probably benefit from the old GC because when it collects, it copies all live objects into new memory and you get defragmentation "for free" as a byproduct. Obviously it's doing more writing so pauses can be more pronounced, but I feel like for a webapp that might be a good trade-off. Alternatively you can allocate into dedicated arenas based on lifetime. That might be the best solution, at the expense of more engineering. Profiling and testing would tell you for sure.
- christophilus 9mo agoI love HN. This is gold.
- stackghost 9mo agoHey it's totally possible that I'm actually a golden retriever who has no idea what he's talking about woof woof bark wag wag
- rurban 9mo agoSBCL doesnt know when it's running low on available heap space? clisp uses libsigsegv, so it knows when to garbage collect really, and when it's not so needed.
- pjmlp 9mo agoAh, the enthusiasm to please from our AI minions. :)