> The fallacy in that reasoning is that a program that's using a lot of CPU (especially if it's a huge MLOC-sized app) is most likely using up its CPU on memory

> The fallacy in that reasoning is that a program that's using a lot of CPU (especially if it's a huge MLOC-sized app) is most likely using up its CPU on memory throughput, not pure number-crunching compute! No, there's no such fallacy here because that assumption is not needed for the conclusion. The point is that CPU is needed to use RAM, and so if you use CPU for whatever reason - even to loop for an hour over some integers - you are consuming a resource that is needed to use RAM (by another program). So the use of CPU "captures" RAM whether it uses RAM or not, so it might as well use it. The extreme example I gave was that a program that uses 100% CPU (again, even if it uses zero RAM) effectively capures 100% of RAM because no other program can use any RAM while that program is running. This extreme example is just to build some intuition, but it scales to lower CPU utilisations. > Collections cycles are a huge burden on memory throughput I don't even know where to start. The whole point of moving collectors is that the can make the cost of memory management arbitrarily low, reduing the overhead compared to free list approaches. A collection cycle does a constant amount of work (per program & workload), but the frequency of the collections can be made arbitrarily low. This is memory management 101. The problem of moving collectors has traditionally been the impact on latency, not on throughput (they were always better on throughput than free lists) - until the advent of pauseless moving collectors. > Heap allocation may have marginally higher "CPU" use in the pure compute sense, but that's exactly the kind of CPU use that does trade off successfully with a lower RAM footprint. Except it doesn't given the actual economics of RAM and CPU. You'll need to wait for my talk to be posted to YouTube (I can't reproduce it all here), but in the meantime you can watch this one, by my colleague, which was a keynote at the most recent ISMM (International Symposium on Memory Management): https://youtu.be/mLNFVNXbw7I https://youtu.be/mLNFVNXbw7I The problem is that Erik is one of the world's leading experts on memory management, and he's talking to other experts, so his talk assumes quite a bit of familiarity with the subject. Also, his comparison focuses on tracing GCs, leaving the one to malloc/free implicit. > Similarly, non-moving concurrent garbage collectors like Go's also successfully navigate this tradeoff compared to moving/copying collectors Again, except they do not. Go users experience severe problems with the GC that Java users no longer do precisely because of the inefficiencies of their simple GC (the JDK used to have such a GC, but we removed it five years ago when newer, more sophisticated algorithms yielded better results. > On the other side of the tradeoff, arenas and caches increase memory footprint in a way that's low-impact on memory throughput (unlike pervasive use of a copying/moving GC) This is simply not true, and shows unfamiliarity with how modern moving collectors actually work (remember that the first open-source high throughput, paseuless moving collector was first released two and a half years ago). Moving collectors offer pretty much the same tradeoff as arenas. The key points are: 1. A generational design makes copying a relatively rare operation to begin with (only a relatively small number of objects are copied). 2. The frequency of collections can be made arbitrarily low. If the usage happens to be arena-like, i.e. no objects survive, nothing is copied (unrelated long-lived objects are already in the old gen, and because the old gen is untouched, there's no need to compact anything there). The reasons for not using moving collectors have nothing to do with throughput: 1. Latency used to suffer. Low-latency moving collectors were a very advanced technology. The first open-source one is younger than ChatGPT. 2. Moving collectors impact the design of FFI (with C, etc.), as C (etc.) does not support moving pointers for reasons having nothing to do with performance. It's very hard for languages that want a very direct and simple FFI (as FFI is very common in code) have a hard time implementing efficient moving collectors. 3. Good moving collectors require large expert teams (let alone pauseless moving collectors). The languages that have them (to varying degrees of sophistication and performance) are well funded ones. In particular, they are the JDK team, the .NET team, and the V8 team. Good allocators (for malloc/free) are also big and sophisticated beasts these days, but they're much easier to reuse in different languages (i.e. not only by C/C++/Zig/Rust, but also by Python). Effectively, large pieces of those languages' runtime is "offshored" to unrelated specialist teams. > The tradeoff is actually highly favorable to low-level languages, which commonly use arenas to manage challenges with heap allocation such as fragmentation. This is also not true (and I say this because I'm primarily a low-level programmer, and have been doing low-level programming for over 25 years). First, C++ and Rust in particular make arenas hard to use to their full power (which is one of the several reasons low-level programmers prefer Zig). Second, if you're not familiar with the severe costs of memory management in low level languages, I can only conclude you haven't been doing it for very long. Java was designed, among other things, to reduce the severe and hard-to-fix performance problems that many C++ programs had experienced (and do to this day). The hard problems concern both the limitations of AOT compilation and of free-list-based memory management. I'm not saying all C++ programs suffer from these issues, but a huge class of them do, which is one of the reasons large programs have migrated to Java.