4 ms·

> A high allocation rate of long-lived objects is not easy to do. This is exactly what async programs do on the hot path. Consider a 1M request per second pro

by hedora 2mo ago

> A high allocation rate of long-lived objects is not easy to do. This is exactly what async programs do on the hot path. Consider a 1M request per second process holding 64K of buffers per request. That’s 64GB of allocations per second. Now, assume the requests hit a remote database with 10ms latency. That’s 640MB of live heap in steady state, which ends up in the “long lived” part of most garbage collectors. Using RAM to save CPU is exactly the wrong tradeoff when such a system becomes CPU bound. It’s almost always the case that it is CPU bound due to an incoming request spike or elevated retry rates on the backend. Those tend to pile up, creating a 64GB/sec leak. The alternative is that the system is CPU bound because the heap is large. This is also very common. Unless each collection takes less work as the heap increases in size, backing off the GC rate to free CPU instantly drives the system into metastable failure, where the GC becomes more expensive because the GC is expensive. Instead of reasoning about this all the time, it’s much easier (for me, granted, I am not a typical java developer) to just jam the CPU intensive work on a low priority event queue so that it uses 100% CPU but never blocks low latency stuff, or things about to retire requests. (Or, stick it in a dedicated but small thread pool if I can’t touch the async event loops). This ends up being easier to deal with than java, since everything is thread safe, allocations are predictable, and there are CPU escape hatches I can use. C++ lets me use smart pointers that have exactly the semantics I want, and that are memory safe but racy in practice. Rust makes them actually memory and thread safe, but sometimes adds useless copies, initializations and thread synchronization (or requires unsafe).

pron 2mo ago
> That’s 640MB of live heap in steady state, which ends up in the “long lived” part of most garbage collectors. It won't, because that is exactly the thing good moving GCs detect and size the young-gen accordingly. > Using RAM to save CPU is exactly the wrong tradeoff when such a system becomes CPU bound. Did you mean to write something else, because it's pretty obvious that it's the right tradeoff? If something is CPU-bound, you want to reduce the CPU usage. > Unless each collection takes less work as the heap increases in size, backing off the GC rate to free CPU instantly drives the system into metastable failure, where the GC becomes more expensive because the GC is expensive. The whole point of moving collectors is that each collection takes the same amount of work, but you need to do it less frequently as the heap rises. So yes, as they heap grows, moving GCs are supposed to work less. The heap grows as a function of the allocation rate while the CPU devoted to memory management remains the same. That's precisely the optimisation that moving collectors bring. > Instead of reasoning about this all the time The whole point is that the GC is what "reasons" about this for you. > it’s much easier (for me, granted, I am not a typical java developer) to just jam the CPU intensive work on a low priority event queue so that it uses 100% CPU but never blocks low latency stuff, or things about to retire requests. That's orthogonal. You can do that at least as easily in Java. > This ends up being easier to deal with than java, since everything is thread safe, allocations are predictable, and there are CPU escape hatches I can use. Thread safety is orthogonal, and now with ZGC, memory management in Java is more predictable than malloc/free allocators. > C++ lets me use smart pointers that have exactly the semantics I want, and that are memory safe but racy in practice. Rust makes them actually memory and thread safe, but sometimes adds useless copies, initializations and thread synchronization (or requires unsafe). Yes, and it's also less efficient and less predictable in the memory management work as programs grow larger.

by hedora 2mo ago

pron 2mo ago

> That’s 640MB of live heap in steady state, which ends up in the “long lived” part of most garbage collectors. It won't, because that is exactly the thing good moving GCs detect and size the young-gen accordingly. > Using RAM to save CPU is exactly the wrong tradeoff when such a system becomes CPU bound. Did you mean to write something else, because it's pretty obvious that it's the right tradeoff? If something is CPU-bound, you want to reduce the CPU usage. > Unless each collection takes less work as the heap increases in size, backing off the GC rate to free CPU instantly drives the system into metastable failure, where the GC becomes more expensive because the GC is expensive. The whole point of moving collectors is that each collection takes the same amount of work, but you need to do it less frequently as the heap rises. So yes, as they heap grows, moving GCs are supposed to work less. The heap grows as a function of the allocation rate while the CPU devoted to memory management remains the same. That's precisely the optimisation that moving collectors bring. > Instead of reasoning about this all the time The whole point is that the GC is what "reasons" about this for you. > it’s much easier (for me, granted, I am not a typical java developer) to just jam the CPU intensive work on a low priority event queue so that it uses 100% CPU but never blocks low latency stuff, or things about to retire requests. That's orthogonal. You can do that at least as easily in Java. > This ends up being easier to deal with than java, since everything is thread safe, allocations are predictable, and there are CPU escape hatches I can use. Thread safety is orthogonal, and now with ZGC, memory management in Java is more predictable than malloc/free allocators. > C++ lets me use smart pointers that have exactly the semantics I want, and that are memory safe but racy in practice. Rust makes them actually memory and thread safe, but sometimes adds useless copies, initializations and thread synchronization (or requires unsafe). Yes, and it's also less efficient and less predictable in the memory management work as programs grow larger.