7 ms·

Except in the special case where all memory can be easily handled in arenas, good tracing GCs have long ago surpassed manual memory management in throughput, an

by pron 3y ago

Except in the special case where all memory can be easily handled in arenas, good tracing GCs have long ago surpassed manual memory management in throughput, and more recently their latency impact is more than acceptable for the vast majority of applications (OpenJDK's ZGC has typical pause times measured in double/triple-digit microseconds, and the worst case rarely exceeds 1ms for a reasonable allocation rate -- the pauses are in the same ballpark as OS-induced ones). The only real and significant tradeoff is in memory footprint, and outside of specialty niches (where arenas just work for everything and worst-case latency is in the low microseconds range) that is the only high order question: is my application running in a memory-constrained environment (or it's really worth it to sacrifice other things to keep down RAM consumption) or not?

znpy 3y ago
> (OpenJDK's ZGC has typical pause times measured in double/triple-digit microseconds, and the worst case rarely exceeds 1ms for a reasonable allocation rate -- the pauses are in the same ballpark as OS-induced ones) We've been benchmarking ZGC and Shenandoah at work, and the p100 pause time is usually below 500us (micro-seconds). ZGC seems to be performing a bit better, as it seems to be performing less pauses than Shenandoah (hence doing more work/pause). We still have to run tests-in-production, but so far it seems that GC pauses are largely a solved issue when using ZGC (and Generational ZGC since Java21).
- deleted 3y ago
  [deleted]
- Galanwe 3y ago
  > We've been benchmarking ZGC and Shenandoah at work, and the p100 pause time is usually below 500us > so far it seems that GC pauses are largely a solved issue when using ZGC What? 500us is abysmally slow. Most projects I work on have a latency budget of less than 10us, the average being 2us. That is the budget for the whole wire in/wire out for a packet. Even for less latency sensitive workloads, 500us is a no go for most networking application.
  - pron 3y ago
    We're talking about occasional hiccups, not an average-case response-latency overhead. You can't get worst-case latency of 2-10us with a non-realtime kernel. Even a page fault could take longer than that.
    - Galanwe 3y ago
      > You can't get worst-case latency of 2-10us with a non-realtime kernel. Even a page fault could take longer than that. You obviously can, and this has nothing to do with the kernel being real-time or not. There is no situation I can think of where a page fault should occur on a properly setup system running a production networking software, meaning no swap, huge TLB, and proper memory management.
      - pron 3y ago
        If you can think of "no situation" where a server may incur a page fault, forced preemption, or need to perform any I/O to a service/database, then I hope you at least recognise that your world in no way represents the state of server software at large because none of these things is true for the vast majority of server software. In a former life I worked on some safety-critical onboard avionics software for an ultrasonic platform, and 2us was around the upper-limit worst-case latency (i.e. you'll kill someone if you miss that deadline); still, it's not the kind of requirements the vast majority of software finds itself under. When working over the internet, some of the very best services are at >10ms ping latency anyway, where a 500us hiccup is imperceptible.
        Galanwe 3y ago
        > If you can think of "no situation" where a server may incur a page fault, forced preemption, or need to perform any I/O to a service/database, then I hope you at least recognise that your world in no way represents the state of server software at large I won't deny that the majority of software out there is not latency sensitive, but the context of this article is specifically targeting those softwares that are _not_ using garbage collection, arguing that it is undeservedly overlooked. OP even adds that GC is a "solved problem" because some GC implementation is 500us worst case latency. My point is that the article author, and OP, are mistaken. Because if you are in the category of "I write server side software without GC" (e. g. C/C++), then 500us is horribly wrong. Your point being that 500us is fine for most software out there is surely true, but not relevant, because if that is the case, you're probably not using C/C++, thus this article is not targeting you. In _my world_ as you phrase it, traffic is unshaped. You need to be able to support line rate, otherwise packets are dropped, and hell breaks loose.
- pron 3y ago
  FYI, ZGC doesn't perform any collection work in the the stop-the-world pauses. They are only required to get all mutator threads to atomically observe the increment of the "GC epoch" for all threads. All actual work, both marking and compaction, is done by GC threads running concurrently with the mutator threads or by the mutator threads themselves as they run. It is only when allocation rate is very, very high that an allocating mutator thread will be paused ("throttled") for a significant amount of time to allow the GC to catch up by freeing up memory (and if you hit these cases, then you might be better off using a throughput-oriented collector).
  - hawk_ 3y ago
    Strangely in our workloads we have noticed generational ZGC latencies are better than G1 at ~99th-99.9th percentile or below but worse at percentiles above that. The allocation rate is moderate.
    - kaba0 3y ago
      Above that you might easily get measuring artifacts, like the OS swapping out your process once, or so.
      - hawk_ 3y ago
        Yes - but it's quite consistent. If it was 'noise' it wouldn't be.
    - loeg 3y ago
      It can be easy to game metrics like this -- trading off p100 for p99.9 or p99. E.g., defer all the expensive work to 1 in every 10,000 operations.
  - znpy 3y ago
    > FYI, ZGC doesn't perform any collection work in the the stop-the-world pauses. Yeah i know, but on a certain level I don’t care what it does or does not do. I care about my application not having latency spikes due to stop the world pauses :)
flohofwoe 3y ago
> Except in the special case ... IME it's the other way around, per-object individual lifetimes is a rare special case, in most real world code there will be many related objects of the same or very similar lifetimes. In such code, tracking individual object lifetimes is overkill (in the end, memory management is all about lifetimes, and fewer individual lifetimes instead of many is always better because it means less work, both in manual and automatic memory management). Not having to think about object lifetimes is just very convenient, that's why GC languages were successful despite the significant under-the-hood complexity of a good garbage collector.
- pron 3y ago
  That's not at all a rare special case in most server applications. One way to see that is to consider how much of a program's working set is in threads' stacks vs the heap. If the most of the working set is in the heap, there's usually some non-trivial object lifetimes involved (i.e. cases where lifetimes can be encapsulated and abstracted away from client code). Yes, sometimes all of these can be taken care of by arenas (and there are languages, like Zig, that strive to be very arena-friendly), but that -- i.e. the case where all objects are easily arena-able -- is not the more common scenario.
  - flohofwoe 3y ago
    Depends on the type of server I guess. I can easily imagine a situation where all allocations of a request are handled through a simple bump allocator, and once the request is done, the bump allocator is reset instead of 'freeing' each individual allocation.
    - cogman10 3y ago
      This would only work for fairly trivial applications. The moment you start adding things like http clients or databases you have to start considering having connection pools with lifetimes that don't strictly match a request lifecycle. Not saying such an application doesn't exist, it certainly does.
Thaxll 3y ago
Pauses are kind of solved but the CPU usage for the GC is still pretty high. You're still at the mercy of unpredictable tail latency and other corner cases.
- pron 3y ago
  That goes for manual memory management -- and certainly languages with a reference-counting GC, like Rust -- as well. The main difference is by far footprint overhead.
  - flohofwoe 3y ago
    I think the important thing to understand is that reference counting isn't any better (and often worse) than "regular" garbage collection. The point of manual memory management is to come up with problem-specific strategies to avoid or at least reduce dynamic memory allocation, not to insert manual release/free calls for individual objects ;)
    - pron 3y ago
      Reference counting is regular garbage collections. The two broad classes of GC algorithms are tracing and refcounting, and while they can converge to similar behaviour, usually the former optimises for throughput while the latter for memory footprint; latency is similar these days.
      - flohofwoe 3y ago
        > Reference counting is regular garbage collection. ...while I agree, for many C++ and Rust coders statements like this are pure heresy ;)
        pjmlp 3y ago
        They should read some CS literature, the kind that is used to write the compilers they rely on. :)
        3y ago
hyperpape 3y ago
Where are the measurements comparing throughput of tracing GCs and manual memory management? I'm aware of how incredibly hard this area is to measure, but it's a shame that the state of mainstream discussion is "most people just assume GC implies slow, but then again a handful of people say it's not."
- pron 3y ago
  Given that the no-GC-by-default market is ~10% of the global software market [1] with no signs of shift in either direction over the past couple of decades, which sounds about right to me (considering the percentage of programs that need to run in memory-constrained environment or must have precise control over memory), it seems that the number of those who may benefit significantly from a different choice is small and so it doesn't look like anyone wants or needs to be convinced of anything. "GC languages" already command ~90% of the market and have little to gain from such a small market of potential converts, and the others aren't trying or succeeding in increasing their market share, so who cares given the small stakes? [1]: https://www.devjobsscanner.com/blog/top-8-most-demanded-programming-languages/ https://www.devjobsscanner.com/blog/top-8-most-demanded-prog...
mtzet 3y ago
> special case where all memory can be easily handled in arenas That seems to be an unfair bar to set. If _most_ objects are easily allocated by an arena, then that still removes most of the need for GC. I like Jai's thesis that there's four types of memory allocations, from most common to least common: 1. Extremely short lived. Can be allocated on the function stack. 2. Short lived + well-defined lifetime (per frame/request). Can be allocated in a memory arena. 3. Long lived + well-defined owner. Can be managed by a subsystem-specific pool. 4. Long lived + unclear owner. Needs a dynamic memory management approach. If you want to make the claim that tracing GCs surpass manual memory management in general, you should compare to a system written with this in mind, not one that calls malloc/free all over the place. I guess it might be more fair if you compare tracing GC with modern c++/rust practices. I agree that for most systems, it's probably much more _practical_ to rely on tracing GC, but that's a very different statement.
- pron 3y ago
  I agree that all might be too high a bar, but most is too low, because even if most of your objects fall into categories 1-3, sufficiently many objects in category 4 would still make your life miserable. Furthermore, it's not like arenas take care of too much because they still require some careful thinking. E.g. Rust's lifetimes don't automatically ensure a correct use of arenas in all cases. A language like Zig is very friendly to arenas, but you still have to be careful about UAF. Now, I know very little about Jai, but bear in mind that its author doesn't have much experience at all with servers or with concurrent software in general, which is 1. where objects with uncertain lifetimes are common enough to be a serious problem, and 2. a huge portion of software being written these days. Games are a domain where it's unusually common for nearly all objects to have very clear, "statically analyzable" lifetimes.
- anamax 3y ago
  > 2. Short lived + well-defined lifetime (per frame/request). Can be allocated in a memory arena. You now have an "arena management" problem. Figuring out the correct arena to allocate from can be a problem. (You want to put all objects that become unused at the same time into the same arena, but that's not the same as all temporary objects allocated between time t and t+delta.)
epr 3y ago
> Except in the special case where all memory can be easily handled in arenas, good tracing GCs have long ago surpassed manual memory management in throughput Citation needed
pkolaczk 3y ago
> good tracing GCs have long ago surpassed manual memory management in throughput, and more recently their latency impact is more than acceptable for the vast majority of applications There is a very subtle problem you seem to be missing: the GCs with high throughput are not the same as the ones with low latency. So while technically it is possible that tracing could beat manual management on one metric - throughput, latency or memory overhead, it doesn’t do on all of them together.

by pron 3y ago

znpy 3y ago

> (OpenJDK's ZGC has typical pause times measured in double/triple-digit microseconds, and the worst case rarely exceeds 1ms for a reasonable allocation rate -- the pauses are in the same ballpark as OS-induced ones) We've been benchmarking ZGC and Shenandoah at work, and the p100 pause time is usually below 500us (micro-seconds). ZGC seems to be performing a bit better, as it seems to be performing less pauses than Shenandoah (hence doing more work/pause). We still have to run tests-in-production, but so far it seems that GC pauses are largely a solved issue when using ZGC (and Generational ZGC since Java21).

deleted 3y ago
[deleted]
Galanwe 3y ago
> We've been benchmarking ZGC and Shenandoah at work, and the p100 pause time is usually below 500us > so far it seems that GC pauses are largely a solved issue when using ZGC What? 500us is abysmally slow. Most projects I work on have a latency budget of less than 10us, the average being 2us. That is the budget for the whole wire in/wire out for a packet. Even for less latency sensitive workloads, 500us is a no go for most networking application.
- pron 3y ago
  We're talking about occasional hiccups, not an average-case response-latency overhead. You can't get worst-case latency of 2-10us with a non-realtime kernel. Even a page fault could take longer than that.
  - Galanwe 3y ago
    > You can't get worst-case latency of 2-10us with a non-realtime kernel. Even a page fault could take longer than that. You obviously can, and this has nothing to do with the kernel being real-time or not. There is no situation I can think of where a page fault should occur on a properly setup system running a production networking software, meaning no swap, huge TLB, and proper memory management.
    - pron 3y ago
      If you can think of "no situation" where a server may incur a page fault, forced preemption, or need to perform any I/O to a service/database, then I hope you at least recognise that your world in no way represents the state of server software at large because none of these things is true for the vast majority of server software. In a former life I worked on some safety-critical onboard avionics software for an ultrasonic platform, and 2us was around the upper-limit worst-case latency (i.e. you'll kill someone if you miss that deadline); still, it's not the kind of requirements the vast majority of software finds itself under. When working over the internet, some of the very best services are at >10ms ping latency anyway, where a 500us hiccup is imperceptible.
      - Galanwe 3y ago
        > If you can think of "no situation" where a server may incur a page fault, forced preemption, or need to perform any I/O to a service/database, then I hope you at least recognise that your world in no way represents the state of server software at large I won't deny that the majority of software out there is not latency sensitive, but the context of this article is specifically targeting those softwares that are _not_ using garbage collection, arguing that it is undeservedly overlooked. OP even adds that GC is a "solved problem" because some GC implementation is 500us worst case latency. My point is that the article author, and OP, are mistaken. Because if you are in the category of "I write server side software without GC" (e. g. C/C++), then 500us is horribly wrong. Your point being that 500us is fine for most software out there is surely true, but not relevant, because if that is the case, you're probably not using C/C++, thus this article is not targeting you. In _my world_ as you phrase it, traffic is unshaped. You need to be able to support line rate, otherwise packets are dropped, and hell breaks loose.
pron 3y ago
FYI, ZGC doesn't perform any collection work in the the stop-the-world pauses. They are only required to get all mutator threads to atomically observe the increment of the "GC epoch" for all threads. All actual work, both marking and compaction, is done by GC threads running concurrently with the mutator threads or by the mutator threads themselves as they run. It is only when allocation rate is very, very high that an allocating mutator thread will be paused ("throttled") for a significant amount of time to allow the GC to catch up by freeing up memory (and if you hit these cases, then you might be better off using a throughput-oriented collector).
- hawk_ 3y ago
  Strangely in our workloads we have noticed generational ZGC latencies are better than G1 at ~99th-99.9th percentile or below but worse at percentiles above that. The allocation rate is moderate.
  - kaba0 3y ago
    Above that you might easily get measuring artifacts, like the OS swapping out your process once, or so.
    - hawk_ 3y ago
      Yes - but it's quite consistent. If it was 'noise' it wouldn't be.
  - loeg 3y ago
    It can be easy to game metrics like this -- trading off p100 for p99.9 or p99. E.g., defer all the expensive work to 1 in every 10,000 operations.
- znpy 3y ago
  > FYI, ZGC doesn't perform any collection work in the the stop-the-world pauses. Yeah i know, but on a certain level I don’t care what it does or does not do. I care about my application not having latency spikes due to stop the world pauses :)

flohofwoe 3y ago

> Except in the special case ... IME it's the other way around, per-object individual lifetimes is a rare special case, in most real world code there will be many related objects of the same or very similar lifetimes. In such code, tracking individual object lifetimes is overkill (in the end, memory management is all about lifetimes, and fewer individual lifetimes instead of many is always better because it means less work, both in manual and automatic memory management). Not having to think about object lifetimes is just very convenient, that's why GC languages were successful despite the significant under-the-hood complexity of a good garbage collector.

pron 3y ago
That's not at all a rare special case in most server applications. One way to see that is to consider how much of a program's working set is in threads' stacks vs the heap. If the most of the working set is in the heap, there's usually some non-trivial object lifetimes involved (i.e. cases where lifetimes can be encapsulated and abstracted away from client code). Yes, sometimes all of these can be taken care of by arenas (and there are languages, like Zig, that strive to be very arena-friendly), but that -- i.e. the case where all objects are easily arena-able -- is not the more common scenario.
- flohofwoe 3y ago
  Depends on the type of server I guess. I can easily imagine a situation where all allocations of a request are handled through a simple bump allocator, and once the request is done, the bump allocator is reset instead of 'freeing' each individual allocation.
  - cogman10 3y ago
    This would only work for fairly trivial applications. The moment you start adding things like http clients or databases you have to start considering having connection pools with lifetimes that don't strictly match a request lifecycle. Not saying such an application doesn't exist, it certainly does.

Thaxll 3y ago

Pauses are kind of solved but the CPU usage for the GC is still pretty high. You're still at the mercy of unpredictable tail latency and other corner cases.

pron 3y ago
That goes for manual memory management -- and certainly languages with a reference-counting GC, like Rust -- as well. The main difference is by far footprint overhead.
- flohofwoe 3y ago
  I think the important thing to understand is that reference counting isn't any better (and often worse) than "regular" garbage collection. The point of manual memory management is to come up with problem-specific strategies to avoid or at least reduce dynamic memory allocation, not to insert manual release/free calls for individual objects ;)
  - pron 3y ago
    Reference counting is regular garbage collections. The two broad classes of GC algorithms are tracing and refcounting, and while they can converge to similar behaviour, usually the former optimises for throughput while the latter for memory footprint; latency is similar these days.
    - flohofwoe 3y ago
      > Reference counting is regular garbage collection. ...while I agree, for many C++ and Rust coders statements like this are pure heresy ;)
      - pjmlp 3y ago
        They should read some CS literature, the kind that is used to write the compilers they rely on. :)
      - 3y ago

hyperpape 3y ago

Where are the measurements comparing throughput of tracing GCs and manual memory management? I'm aware of how incredibly hard this area is to measure, but it's a shame that the state of mainstream discussion is "most people just assume GC implies slow, but then again a handful of people say it's not."

pron 3y ago
Given that the no-GC-by-default market is ~10% of the global software market [1] with no signs of shift in either direction over the past couple of decades, which sounds about right to me (considering the percentage of programs that need to run in memory-constrained environment or must have precise control over memory), it seems that the number of those who may benefit significantly from a different choice is small and so it doesn't look like anyone wants or needs to be convinced of anything. "GC languages" already command ~90% of the market and have little to gain from such a small market of potential converts, and the others aren't trying or succeeding in increasing their market share, so who cares given the small stakes? [1]: https://www.devjobsscanner.com/blog/top-8-most-demanded-programming-languages/ https://www.devjobsscanner.com/blog/top-8-most-demanded-prog...

mtzet 3y ago

> special case where all memory can be easily handled in arenas That seems to be an unfair bar to set. If _most_ objects are easily allocated by an arena, then that still removes most of the need for GC. I like Jai's thesis that there's four types of memory allocations, from most common to least common: 1. Extremely short lived. Can be allocated on the function stack. 2. Short lived + well-defined lifetime (per frame/request). Can be allocated in a memory arena. 3. Long lived + well-defined owner. Can be managed by a subsystem-specific pool. 4. Long lived + unclear owner. Needs a dynamic memory management approach. If you want to make the claim that tracing GCs surpass manual memory management in general, you should compare to a system written with this in mind, not one that calls malloc/free all over the place. I guess it might be more fair if you compare tracing GC with modern c++/rust practices. I agree that for most systems, it's probably much more _practical_ to rely on tracing GC, but that's a very different statement.

pron 3y ago
I agree that all might be too high a bar, but most is too low, because even if most of your objects fall into categories 1-3, sufficiently many objects in category 4 would still make your life miserable. Furthermore, it's not like arenas take care of too much because they still require some careful thinking. E.g. Rust's lifetimes don't automatically ensure a correct use of arenas in all cases. A language like Zig is very friendly to arenas, but you still have to be careful about UAF. Now, I know very little about Jai, but bear in mind that its author doesn't have much experience at all with servers or with concurrent software in general, which is 1. where objects with uncertain lifetimes are common enough to be a serious problem, and 2. a huge portion of software being written these days. Games are a domain where it's unusually common for nearly all objects to have very clear, "statically analyzable" lifetimes.
anamax 3y ago
> 2. Short lived + well-defined lifetime (per frame/request). Can be allocated in a memory arena. You now have an "arena management" problem. Figuring out the correct arena to allocate from can be a problem. (You want to put all objects that become unused at the same time into the same arena, but that's not the same as all temporary objects allocated between time t and t+delta.)

epr 3y ago

> Except in the special case where all memory can be easily handled in arenas, good tracing GCs have long ago surpassed manual memory management in throughput Citation needed

pkolaczk 3y ago

> good tracing GCs have long ago surpassed manual memory management in throughput, and more recently their latency impact is more than acceptable for the vast majority of applications There is a very subtle problem you seem to be missing: the GCs with high throughput are not the same as the ones with low latency. So while technically it is possible that tracing could beat manual management on one metric - throughput, latency or memory overhead, it doesn’t do on all of them together.