14 ms·

> In rust, I usually just make sure stuff is not Box<>, and try to reuse buffers. That generally gets the memory allocator completely out of the way (except for

by pron 2mo ago

> In rust, I usually just make sure stuff is not Box<>, and try to reuse buffers. That generally gets the memory allocator completely out of the way (except for async). You say "just", but this is easy when programs are small. The problem is that this gets harder and harder and harder as programs grow large (the whole point of the JVM's design was to address the performance issues that plague large C++ programs). E.g. someone who works at one of the world's largest tech companies just told me that they have problems with Rust programs spending 30% of their CPU on memory management even when they're as small as a couple hundreds of thousands of LOC. > Is java able to stackify most new Object calls and elide default initialization of object members these days? No, the general idea is to just make memory management efficient (although some objects are "stackified" and the compiler will elide zeroing when non-defaults are passed to a constructor). Now, I say "just", but this used to come at the cost of GC pauses and larger footprint. Now it only comes at the cost of a larger footprint. But there is a definite choice here when it comes to performance. Low level languages give you control that means performance is attained through manual effort. Java takes away control to improve effort-per-performance. Roughly speaking, these tradeoffs mean that when programs are small and the extra effort is manageable, low-level languages are hard to beat, but when programs are large, it is Java that is hard to beat. > I’m surprised to hear there is no demand for compiler enforced/facilitated thread safety in Java. That was a major pain point in all the Java code bases I’ve worked with in the past, and is a headline safety feature for rust (which goes even further and enforces aliasing rules) and JS. This used to be a bigger problem when locks were the main mechanism for sharing data among threads. Now, with the wide selection of concurrent data structures, such problems don't occur as much. I'm not saying they don't occur at all, just not frequently enough to become a major priority. Also, safe Rust's data-race freedom comes at the cost of requiring unsafe for benign races, which are not uncommon in concurrent algorithms (i.e. it excludes even "good" races). This may be fine in languages whose view on performance is "with enough effort you can get good performance", but, as I said, Java is about making more "naive" programs fast with little effort.

stmw 2mo ago
It is fair that there are many ways to be slow in any number of programming languages. I'm surprised to hear "Rust programs spending 30% of their CPU on memory management even when they're as small as a couple hundreds of thousands of LOC", although I can visualize some unique workloads where that's unavoidable irrespective of language & runtime.
- pron 2mo ago
  You say unavoidable, but moving collectors are designed to reduce CPU at high allocation rates by increasing the heap size. Generational moving collectors have a pathological case - a high allocation rate of long-lived objects - but that's quite hard to get yourself into by accident. Their main downside (besides the inherent increased footprint) used to be unpredictable long pauses, which could have a very high impact on tail latencies, and that's just gone with ZGC. Generational moving collectors are a very powerful memory management optimisation, but because they necessarily require some "interesting" FFI layer between the ordinary heap and any passing of pointers between the program and the hardware/OS - the very thing low-level languages are designed not to have - this is a powerful, general optimisation (not perfect, but extremely useful in a wide class of programs) that is not available to low-level programming languages. And it's not the only one, BTW. JIT compilers are also designed for "global" average-case optimisations at the cost of precise low-level control over the worst case, which is another thing that low-level languages trade away. In the most simplistic way, I would say that the precise control that low-level languages are all about helps their performance when programs are small (and can be manually optimised globally) and hurts their performance as programs get large. They have to give up on some optimisations that come at the cost of ceding low-level control, and that includes moving collectors.
  - stmw 2mo ago
    When I say "unavoidable" I mean exactly things like your "a high allocation rate of long-lived objects (as one example), where the work that has to be done requires a significant number of CPU and memory cycles spent on memory management (manual or automatic). When I say tradeoffs", I mean exactly things like "inherent increased footprint", or my earlier "wasting RAM" point. At this point, I'm not sure that we're disagreeing about these details, but rather what we make of them... I view them as just years of continued tuning of tradeoffs (heap size vs CPU cycles vs memory cycles), while you them as major breakthrough that makes garbage collection much more desirable. Is that fair?
    - pron 2mo ago
      A high allocation rate of long-lived objects is not easy to do. Object "death" rate has to equal the allocation rate, so a high allocation rate of long-lived objects means that somehow you get to allocate, say, 1 GB/s of data that is kept for a long time and is discarded at a rate of 1 GB/s. > When I say tradeoffs", I mean exactly things like "inherent increased footprint", or my earlier "wasting RAM" point. Well, that is a real tradeoff, but there's a reason why it's a very attractive one for a huge class of applications. There are two ways of looking at this, which amount to the same thing: 1. Because both RAM and CPU are needed for computation, what matters isn't each of their utilisation values separately but only the more constraining or impactful of the two. 2. Because CPU is needed to use RAM, every CPU cycle you spend effectively takes away some other program's ability to use RAM. This means that for any amount of CPU utilisation, there is some amount of RAM that is effectively free (i.e. has no additional impact), and the more CPU a program consumes, the more RAM it can consume without it making additional impact. It's easiest to see in the extreme case of a program using 100% CPU: no other program can make progress, and so it doesn't matter how much of the available RAM your program is using - it effectively captures all of it whether it uses it or not. But this scales to any amount of CPU utilisation (not quite linearly). What wastes RAM is not using that "free RAM" to reduce the dominant resource, CPU. And what further determines the RAM/CPU "exchange rate" is the RAM/CPU ratio offered by the hardware, which is more RAM-heavy than some appreciate (it is very hard to find a metal or virtual deployment with less than 1 GB of RAM per core these days - taking into account partial cores in virtual machines - outside of embedded devices). This means that if you have a memory management algorithm that uses more RAM to help reduce CPU as CPU utilisation rises, that's usually a good thing. And moving collectors work exactly like that. The heap overhead in a generational moving collector is only a function of the allocation rate, and a high allocation rate also means high CPU usage. My colleague, the main developer of ZGC these days, gave a keynote about this very subject at ISMM: https://youtu.be/mLNFVNXbw7I https://youtu.be/mLNFVNXbw7I > I view them as just years of continued tuning of tradeoffs (heap size vs CPU cycles vs memory cycles), while you them as major breakthrough that makes garbage collection much more desirable. Is that fair? I say that for many years, the main practical, most "felt" tradeoff of moving collectors has been their STW pauses. With pauses eliminated, there is a qualitative change in the attractiveness of moving collectors, making them more appropriate than other memory management techniques for an even broader class of applications than before. Previously, applications that were very sensitive to tail latencies didn't want moving collectors; now, the tail latency is no longer an issue (unless your application's tail latency tolerance is such that a realtime OS is needed). In other words, I'm saying that moving collectors' most impactful tradeoff is now gone.

by pron 2mo ago

stmw 2mo ago

It is fair that there are many ways to be slow in any number of programming languages. I'm surprised to hear "Rust programs spending 30% of their CPU on memory management even when they're as small as a couple hundreds of thousands of LOC", although I can visualize some unique workloads where that's unavoidable irrespective of language & runtime.

pron 2mo ago
You say unavoidable, but moving collectors are designed to reduce CPU at high allocation rates by increasing the heap size. Generational moving collectors have a pathological case - a high allocation rate of long-lived objects - but that's quite hard to get yourself into by accident. Their main downside (besides the inherent increased footprint) used to be unpredictable long pauses, which could have a very high impact on tail latencies, and that's just gone with ZGC. Generational moving collectors are a very powerful memory management optimisation, but because they necessarily require some "interesting" FFI layer between the ordinary heap and any passing of pointers between the program and the hardware/OS - the very thing low-level languages are designed not to have - this is a powerful, general optimisation (not perfect, but extremely useful in a wide class of programs) that is not available to low-level programming languages. And it's not the only one, BTW. JIT compilers are also designed for "global" average-case optimisations at the cost of precise low-level control over the worst case, which is another thing that low-level languages trade away. In the most simplistic way, I would say that the precise control that low-level languages are all about helps their performance when programs are small (and can be manually optimised globally) and hurts their performance as programs get large. They have to give up on some optimisations that come at the cost of ceding low-level control, and that includes moving collectors.
- stmw 2mo ago
  When I say "unavoidable" I mean exactly things like your "a high allocation rate of long-lived objects (as one example), where the work that has to be done requires a significant number of CPU and memory cycles spent on memory management (manual or automatic). When I say tradeoffs", I mean exactly things like "inherent increased footprint", or my earlier "wasting RAM" point. At this point, I'm not sure that we're disagreeing about these details, but rather what we make of them... I view them as just years of continued tuning of tradeoffs (heap size vs CPU cycles vs memory cycles), while you them as major breakthrough that makes garbage collection much more desirable. Is that fair?
  - pron 2mo ago
    A high allocation rate of long-lived objects is not easy to do. Object "death" rate has to equal the allocation rate, so a high allocation rate of long-lived objects means that somehow you get to allocate, say, 1 GB/s of data that is kept for a long time and is discarded at a rate of 1 GB/s. > When I say tradeoffs", I mean exactly things like "inherent increased footprint", or my earlier "wasting RAM" point. Well, that is a real tradeoff, but there's a reason why it's a very attractive one for a huge class of applications. There are two ways of looking at this, which amount to the same thing: 1. Because both RAM and CPU are needed for computation, what matters isn't each of their utilisation values separately but only the more constraining or impactful of the two. 2. Because CPU is needed to use RAM, every CPU cycle you spend effectively takes away some other program's ability to use RAM. This means that for any amount of CPU utilisation, there is some amount of RAM that is effectively free (i.e. has no additional impact), and the more CPU a program consumes, the more RAM it can consume without it making additional impact. It's easiest to see in the extreme case of a program using 100% CPU: no other program can make progress, and so it doesn't matter how much of the available RAM your program is using - it effectively captures all of it whether it uses it or not. But this scales to any amount of CPU utilisation (not quite linearly). What wastes RAM is not using that "free RAM" to reduce the dominant resource, CPU. And what further determines the RAM/CPU "exchange rate" is the RAM/CPU ratio offered by the hardware, which is more RAM-heavy than some appreciate (it is very hard to find a metal or virtual deployment with less than 1 GB of RAM per core these days - taking into account partial cores in virtual machines - outside of embedded devices). This means that if you have a memory management algorithm that uses more RAM to help reduce CPU as CPU utilisation rises, that's usually a good thing. And moving collectors work exactly like that. The heap overhead in a generational moving collector is only a function of the allocation rate, and a high allocation rate also means high CPU usage. My colleague, the main developer of ZGC these days, gave a keynote about this very subject at ISMM: https://youtu.be/mLNFVNXbw7I https://youtu.be/mLNFVNXbw7I > I view them as just years of continued tuning of tradeoffs (heap size vs CPU cycles vs memory cycles), while you them as major breakthrough that makes garbage collection much more desirable. Is that fair? I say that for many years, the main practical, most "felt" tradeoff of moving collectors has been their STW pauses. With pauses eliminated, there is a qualitative change in the attractiveness of moving collectors, making them more appropriate than other memory management techniques for an even broader class of applications than before. Previously, applications that were very sensitive to tail latencies didn't want moving collectors; now, the tail latency is no longer an issue (unless your application's tail latency tolerance is such that a realtime OS is needed). In other words, I'm saying that moving collectors' most impactful tradeoff is now gone.