6 ms·

Concretely, what are current tail latencies, worst case? Ten years ago, “rewrite in C++” was definitely easier than getting the Java GC to stay up under server

by hedora 2mo ago

Concretely, what are current tail latencies, worst case? Ten years ago, “rewrite in C++” was definitely easier than getting the Java GC to stay up under server load. Most servers I work with run on big machines and are the only process, so figure a 100-250GB heap that lives for months, all async, small requests, so insane amounts of Future and String allocation spam. Optimizing that stuff away in Java is harder than writing Rust, so assume idiomatic Java. Also, is there any work on statically enforcing data race freedom in Java? That’s a bigger rust selling point than memory safety for me. I think swift has done some interesting work in that space. It would be nice to get those sorts of safety properties without manually writing borrow checker annotations.

pron 2mo ago
> Concretely, what are current tail latencies, worst case? Well under 1ms for ZGC (to the point that OS-caused hiccups are of similar magnitudes). > Ten years ago, “rewrite in C++” was definitely easier than getting the Java GC to stay up under server load. Both could have been hard in some cases, but open-source "pauseless" GCs are only 3 years old (and all of the JDK's GCs are nothing like what they were ten years ago). > Optimizing that stuff away in Java is harder than writing Rust, so assume idiomatic Java. Quite the opposite. Performance issues due to memory management are, in practice, more serious in Rust than they are in modern Java. > Also, is there any work on statically enforcing data race freedom in Java? There isn't much demand for that atm. If we see growing demand, we could prioritise it.
- hedora 2mo ago
  In rust, I usually just make sure stuff is not Box<>, and try to reuse buffers. That generally gets the memory allocator completely out of the way (except for async). The remaining allocator performance problems are mostly due to it zeroing allocated memory unless I use unsafe. Is java able to stackify most new Object calls and elide default initialization of object members these days? I’m surprised to hear there is no demand for compiler enforced/facilitated thread safety in Java. That was a major pain point in all the Java code bases I’ve worked with in the past, and is a headline safety feature for rust (which goes even further and enforces aliasing rules) and JS. Could you be seeing selection bias in your user base?
  - pron 2mo ago
    > In rust, I usually just make sure stuff is not Box<>, and try to reuse buffers. That generally gets the memory allocator completely out of the way (except for async). You say "just", but this is easy when programs are small. The problem is that this gets harder and harder and harder as programs grow large (the whole point of the JVM's design was to address the performance issues that plague large C++ programs). E.g. someone who works at one of the world's largest tech companies just told me that they have problems with Rust programs spending 30% of their CPU on memory management even when they're as small as a couple hundreds of thousands of LOC. > Is java able to stackify most new Object calls and elide default initialization of object members these days? No, the general idea is to just make memory management efficient (although some objects are "stackified" and the compiler will elide zeroing when non-defaults are passed to a constructor). Now, I say "just", but this used to come at the cost of GC pauses and larger footprint. Now it only comes at the cost of a larger footprint. But there is a definite choice here when it comes to performance. Low level languages give you control that means performance is attained through manual effort. Java takes away control to improve effort-per-performance. Roughly speaking, these tradeoffs mean that when programs are small and the extra effort is manageable, low-level languages are hard to beat, but when programs are large, it is Java that is hard to beat. > I’m surprised to hear there is no demand for compiler enforced/facilitated thread safety in Java. That was a major pain point in all the Java code bases I’ve worked with in the past, and is a headline safety feature for rust (which goes even further and enforces aliasing rules) and JS. This used to be a bigger problem when locks were the main mechanism for sharing data among threads. Now, with the wide selection of concurrent data structures, such problems don't occur as much. I'm not saying they don't occur at all, just not frequently enough to become a major priority. Also, safe Rust's data-race freedom comes at the cost of requiring unsafe for benign races, which are not uncommon in concurrent algorithms (i.e. it excludes even "good" races). This may be fine in languages whose view on performance is "with enough effort you can get good performance", but, as I said, Java is about making more "naive" programs fast with little effort.
    - stmw 2mo ago
      It is fair that there are many ways to be slow in any number of programming languages. I'm surprised to hear "Rust programs spending 30% of their CPU on memory management even when they're as small as a couple hundreds of thousands of LOC", although I can visualize some unique workloads where that's unavoidable irrespective of language & runtime.
stmw 2mo ago
As the OC, I think my view is somewhere in the middle - I am neither as optimistic about it being "great now" nor do I think that "rewrite in C++" 10 years ago was easier. My reason for disagreeing with the former view is that improvements in physical RAM available and tendency towards smaller workloads have allowed many Java (or other GC runtimes) to essentially "fix their problems because hardware got better". So you can waste more RAM, waste more cycles, but "it doesn't matter", and likely it is fine in many cases - but it's is not the same thing as claiming the GC algorithms are responsivle for that outcome. We have been 3 years away from GC solving memory management for at least 30 years. My reason for disagreeing with the latter view is that for those who don't have 100-250 GB long-lived heaps (or whatever the contemporary version of that is), the pain level is far lower than rewriting in C++ or Rust, or likely the pain level of hiring enough engineers who can do either. It's a completely different engineering culture.
- hedora 2mo ago
  It definitely was easier for the projects I worked on, but they are exactly the use case where the heap is long lived and most of the machine. I’ve also worked on systems with lots of small processes, and the operational issues that creates dwarfs GC problems: It takes one middle tier machine, and adds 64-128 network boundaries, and also creates an extremely difficult static memory allocation problem. I know people do it anyway, but it’s rare that they can articulate a decent technical reason for it, and it wastes something like 90% of the hardware (even in carefully optimized code bases / deployments). Anyway, I’m not the target market for such stuff.
- pron 2mo ago
  > So you can waste more RAM, waste more cycles Just to be clear, the main reason for the use of moving collectors in the first place is to waste less cycles on memory management (otherwise we wouldn't use them). They exist to serve as an optimisation. > We have been 3 years away from GC solving memory management for at least 30 years. It's now 3 years in the past (since Generational ZGC); e.g. see https://netflixtechblog.com/bending-pause-times-to-your-will-with-generational-zgc-256629c9386b https://netflixtechblog.com/bending-pause-times-to-your-will.... Of course, it doesn't solve all imaginable memory management issues, but in practice it makes it a non-issue for a large class of interesting and very common programs.

by hedora 2mo ago

pron 2mo ago

> Concretely, what are current tail latencies, worst case? Well under 1ms for ZGC (to the point that OS-caused hiccups are of similar magnitudes). > Ten years ago, “rewrite in C++” was definitely easier than getting the Java GC to stay up under server load. Both could have been hard in some cases, but open-source "pauseless" GCs are only 3 years old (and all of the JDK's GCs are nothing like what they were ten years ago). > Optimizing that stuff away in Java is harder than writing Rust, so assume idiomatic Java. Quite the opposite. Performance issues due to memory management are, in practice, more serious in Rust than they are in modern Java. > Also, is there any work on statically enforcing data race freedom in Java? There isn't much demand for that atm. If we see growing demand, we could prioritise it.

hedora 2mo ago
In rust, I usually just make sure stuff is not Box<>, and try to reuse buffers. That generally gets the memory allocator completely out of the way (except for async). The remaining allocator performance problems are mostly due to it zeroing allocated memory unless I use unsafe. Is java able to stackify most new Object calls and elide default initialization of object members these days? I’m surprised to hear there is no demand for compiler enforced/facilitated thread safety in Java. That was a major pain point in all the Java code bases I’ve worked with in the past, and is a headline safety feature for rust (which goes even further and enforces aliasing rules) and JS. Could you be seeing selection bias in your user base?
- pron 2mo ago
  > In rust, I usually just make sure stuff is not Box<>, and try to reuse buffers. That generally gets the memory allocator completely out of the way (except for async). You say "just", but this is easy when programs are small. The problem is that this gets harder and harder and harder as programs grow large (the whole point of the JVM's design was to address the performance issues that plague large C++ programs). E.g. someone who works at one of the world's largest tech companies just told me that they have problems with Rust programs spending 30% of their CPU on memory management even when they're as small as a couple hundreds of thousands of LOC. > Is java able to stackify most new Object calls and elide default initialization of object members these days? No, the general idea is to just make memory management efficient (although some objects are "stackified" and the compiler will elide zeroing when non-defaults are passed to a constructor). Now, I say "just", but this used to come at the cost of GC pauses and larger footprint. Now it only comes at the cost of a larger footprint. But there is a definite choice here when it comes to performance. Low level languages give you control that means performance is attained through manual effort. Java takes away control to improve effort-per-performance. Roughly speaking, these tradeoffs mean that when programs are small and the extra effort is manageable, low-level languages are hard to beat, but when programs are large, it is Java that is hard to beat. > I’m surprised to hear there is no demand for compiler enforced/facilitated thread safety in Java. That was a major pain point in all the Java code bases I’ve worked with in the past, and is a headline safety feature for rust (which goes even further and enforces aliasing rules) and JS. This used to be a bigger problem when locks were the main mechanism for sharing data among threads. Now, with the wide selection of concurrent data structures, such problems don't occur as much. I'm not saying they don't occur at all, just not frequently enough to become a major priority. Also, safe Rust's data-race freedom comes at the cost of requiring unsafe for benign races, which are not uncommon in concurrent algorithms (i.e. it excludes even "good" races). This may be fine in languages whose view on performance is "with enough effort you can get good performance", but, as I said, Java is about making more "naive" programs fast with little effort.
  - stmw 2mo ago
    It is fair that there are many ways to be slow in any number of programming languages. I'm surprised to hear "Rust programs spending 30% of their CPU on memory management even when they're as small as a couple hundreds of thousands of LOC", although I can visualize some unique workloads where that's unavoidable irrespective of language & runtime.

stmw 2mo ago

As the OC, I think my view is somewhere in the middle - I am neither as optimistic about it being "great now" nor do I think that "rewrite in C++" 10 years ago was easier. My reason for disagreeing with the former view is that improvements in physical RAM available and tendency towards smaller workloads have allowed many Java (or other GC runtimes) to essentially "fix their problems because hardware got better". So you can waste more RAM, waste more cycles, but "it doesn't matter", and likely it is fine in many cases - but it's is not the same thing as claiming the GC algorithms are responsivle for that outcome. We have been 3 years away from GC solving memory management for at least 30 years. My reason for disagreeing with the latter view is that for those who don't have 100-250 GB long-lived heaps (or whatever the contemporary version of that is), the pain level is far lower than rewriting in C++ or Rust, or likely the pain level of hiring enough engineers who can do either. It's a completely different engineering culture.

hedora 2mo ago
It definitely was easier for the projects I worked on, but they are exactly the use case where the heap is long lived and most of the machine. I’ve also worked on systems with lots of small processes, and the operational issues that creates dwarfs GC problems: It takes one middle tier machine, and adds 64-128 network boundaries, and also creates an extremely difficult static memory allocation problem. I know people do it anyway, but it’s rare that they can articulate a decent technical reason for it, and it wastes something like 90% of the hardware (even in carefully optimized code bases / deployments). Anyway, I’m not the target market for such stuff.
pron 2mo ago
> So you can waste more RAM, waste more cycles Just to be clear, the main reason for the use of moving collectors in the first place is to waste less cycles on memory management (otherwise we wouldn't use them). They exist to serve as an optimisation. > We have been 3 years away from GC solving memory management for at least 30 years. It's now 3 years in the past (since Generational ZGC); e.g. see https://netflixtechblog.com/bending-pause-times-to-your-will-with-generational-zgc-256629c9386b https://netflixtechblog.com/bending-pause-times-to-your-will.... Of course, it doesn't solve all imaginable memory management issues, but in practice it makes it a non-issue for a large class of interesting and very common programs.