6 ms·

I used to work on a high-speed messaging system built in Java, and garbage collection was a killer. No matter how much we tweaked the code and the GC parameter

I used to work on a high-speed messaging system built in Java, and garbage collection was a killer. No matter how much we tweaked the code and the GC parameters, and this eventually got escalated pretty high up in Sun itself, we'd get these occasional multi-minute pauses where the whole thing would just freeze up, buffers would fill and stuff would start dropping on the floor. This was over 10 years ago, so things may have improved, but it would certainly give me pause about ever implementing anything time-critical in Java.

pelasaco 6y ago
It definitely improved. The GC is the area where we see more work between the Java versions
pvg 6y ago
It's changed quite a bit, with some of the biggest developments just in the last couple of years. It should still give you pause (depending on how time-critical) but now you have more choices in pause and more pause-knobs.
m_mueller 6y ago
Another user commented here a while ago that they solved this problem by just throwing it on the largest instance by memory (1TB or so), disabled GC and restart it after market closes.
- jbverschoor 6y ago
  Largest instance sounds like aws. There’s your nr 1 performance bottleneck
  - m_mueller 6y ago
    well, you choose the node infrastructure by whatever network latency you need to have guaranteed - obviously public clouds have their limits there.
    - jbverschoor 6y ago
      But it’s idiotic to discuss ns-optimizations, when your infrastructure and vms add many times more that. Aws is slow. Not talking about the network.
      - m_mueller 6y ago
        ehh I didn't even mention AWS - all I did was using the word 'instance' which seems to have thrown you off.
        LittlePeter 6y ago
        What else can "instance" mean, other than an instance of a virtual server? Granted may not be on AWS, but still is virtual which is a latency killer.
        m_mueller 6y ago
        IMO you can't automatically go from "we have a latency requirement" to "public cloud is out of the question". It depends on what latency requirement. If you can eat an intermittent delay of 20ms, AWS, to me still seems fine, you just need to plan ahead in terms of what resources you rent there. Between that, and the multiple seconds a GC pause could cost you, there is still a space where GC optimization on AWS would make sense. But if you're in HFT and target micro- or even nanosecond optimizations, you sure don't wanna run anything in that critical path on any public cloud.
thangalin 6y ago
Currently developing a radio communications system using Java, with JNI calls for hardware integration, running on a quad-core 32-bit ARM processor. We have constraints of around 20ms for real-time audio packet processing. Recently some stop-the-world pauses introduced by the GC resulted in calls dropping. A few GC parameter tweaks brought our latency below 20ms and the application no longer drops calls. This is using the GC that ships with JDK 11. Future improvements, such as Shenandoah and ZGC, may be able to further drop the interruptions to 5ms on average. The radio repeater is designed for real-time, safety-critical operations dealing with voice communications. Java is up for the job; like others have mentioned, it's more about the technical team and whether management understands what it takes to write and maintain an excellent code base than it is about Java versus C++.
- blub 6y ago
  It would be interesting to know what the trade-off for those tweaks was. Typically it's increased memory usage. Another relevant aspect is what happens when one hits GC pauses again. Will there be any tweaks left? Will they conflict with the previous ones? Using such high-level levers to fix performance problems in a specific component is great when it works, but when it doesn't you're out of options.
  - thangalin 6y ago
    Another option is to use a JVM that offers hard real-time constraints to achieve guaranteed performance characteristics. Such JVMs are in use in the automotive and airline industries---JamaicaVM, for example, offers deterministic garbage collection.
- logicchains 6y ago
  >A few GC parameter tweaks brought our latency below 20ms and the application no longer drops calls. Tick to trade latency for a modern software HFT stack needs to be under 10 _microseconds_, which is over three orders of magnitude more stringent.
  - ridaj 6y ago
patrec 6y ago
Why did your system dynamically allocate memory in the first place? High speed messaging sounds like an area where it should be doable to figure out at startup what resources you need to meet your latency and throughput requirements and then allocate them once and basically never gc anything. Was there some fundamental difficulty with that, or was it more that by the time your realized that GC would kill you, all your code was already written in a way that would have made it prohibitively expensive to switch to an architecture that avoids creating temporary objects?
- taneq 6y ago
  The best part about using a garbage collected language for high performance software is it's exactly the same as a non-garbage-collected language.
  - patrec 6y ago
    Sure, as long as your non-garbage-collected language from ten years ago had memory safety (and general lack of UB footguns), first class IDE support, excellent introspection and profiling facilities and a large potential hiring pool of elite programmers and wide acceptance in the industries you were trying to sell into.
    - taneq 6y ago
      Wait, are we still comparing C++ with Java here?
      - patrec 6y ago
        Yes? If you write zero or low allocation code in Java you pay a price for working against the language and you also lack several tools for writing fast code that C++ provides you with (as well as a bunch of useful abstractions), but your development experience will obviously be much better on a number of other dimensions.
    - fishermanbill 6y ago
jjav 6y ago
You should not have any full GC (pause) events, ever, in properly written production Java code. Source: I worked on several extremely high performance Java systems at Sun and also worked optimizing various Java servers at subsequent jobs based on my Sun experience. At one company (after Sun) I took on a backend service which was doing GC pauses every 3-5 minutes, the code was a dumpster fire mess. After cleaning things up we didn't hit a single full GC in two months of production use (they redeployed every two months so that's as long as the process lasted).
- blub 6y ago
  Extremely high performance properly written Java code has been mentioned so many times in Java performance discussions that it has acquired an almost legendary status, right there with the sufficiently smart compiler.
  - RhysU 6y ago
    I have observed the former, written by exceptional coworkers, but never the latter.
- zozbot234 6y ago
  Maybe, but that just raises the question...If you're coding your system under the constraint that you should not be having any GC-event, what's the point of picking a GC language in the first place?
  - tsimionescu 6y ago
    They said no full GC. The GC runs constantly in parallel with your program and cleans up what it can, but never has to pause execution of the whole program to reclaim even more memory. Not claiming I know how to do this, but I believe this is what they claiming.
  - jjav 6y ago
    As others said, no full GC events. Young generation GC is cheap, no need to avoid that. Productivity is the point. Even while paying attention to GC, it's much faster to write good code in Java than to worry about every malloc() in C. I love C, but if I need to churn out high performance code quickly, Java is the choice.
AtlasBarfed 6y ago
Cassandra has some of these issues as well, despite arena allocation, buffer reuse, and offheap manually allocated memory. 10 years has seen improvements to GC, but not eliminated the problem. Cass 4.0 finally enables ZGC, so we'll see how that goes. If you have stateless, you could do two JVMs and monitor the GC state, and reroute requests to a low-heap JVM and let the other clean up. Garbage collection is hard. I'm surprised that there aren't expert-level concepts like weak references and ways to sub-partition the heap, with say a new operator that can target a partition, so major GC doesn't have to do the whole heap. Although that starts getting into rust concepts and analysis.

pelasaco 6y ago

It definitely improved. The GC is the area where we see more work between the Java versions

pvg 6y ago

It's changed quite a bit, with some of the biggest developments just in the last couple of years. It should still give you pause (depending on how time-critical) but now you have more choices in pause and more pause-knobs.

m_mueller 6y ago

Another user commented here a while ago that they solved this problem by just throwing it on the largest instance by memory (1TB or so), disabled GC and restart it after market closes.

jbverschoor 6y ago
Largest instance sounds like aws. There’s your nr 1 performance bottleneck
- m_mueller 6y ago
  well, you choose the node infrastructure by whatever network latency you need to have guaranteed - obviously public clouds have their limits there.
  - jbverschoor 6y ago
    But it’s idiotic to discuss ns-optimizations, when your infrastructure and vms add many times more that. Aws is slow. Not talking about the network.
    - m_mueller 6y ago
      ehh I didn't even mention AWS - all I did was using the word 'instance' which seems to have thrown you off.
      - LittlePeter 6y ago
        What else can "instance" mean, other than an instance of a virtual server? Granted may not be on AWS, but still is virtual which is a latency killer.
        m_mueller 6y ago
        IMO you can't automatically go from "we have a latency requirement" to "public cloud is out of the question". It depends on what latency requirement. If you can eat an intermittent delay of 20ms, AWS, to me still seems fine, you just need to plan ahead in terms of what resources you rent there. Between that, and the multiple seconds a GC pause could cost you, there is still a space where GC optimization on AWS would make sense. But if you're in HFT and target micro- or even nanosecond optimizations, you sure don't wanna run anything in that critical path on any public cloud.

thangalin 6y ago

Currently developing a radio communications system using Java, with JNI calls for hardware integration, running on a quad-core 32-bit ARM processor. We have constraints of around 20ms for real-time audio packet processing. Recently some stop-the-world pauses introduced by the GC resulted in calls dropping. A few GC parameter tweaks brought our latency below 20ms and the application no longer drops calls. This is using the GC that ships with JDK 11. Future improvements, such as Shenandoah and ZGC, may be able to further drop the interruptions to 5ms on average. The radio repeater is designed for real-time, safety-critical operations dealing with voice communications. Java is up for the job; like others have mentioned, it's more about the technical team and whether management understands what it takes to write and maintain an excellent code base than it is about Java versus C++.

blub 6y ago
It would be interesting to know what the trade-off for those tweaks was. Typically it's increased memory usage. Another relevant aspect is what happens when one hits GC pauses again. Will there be any tweaks left? Will they conflict with the previous ones? Using such high-level levers to fix performance problems in a specific component is great when it works, but when it doesn't you're out of options.
- thangalin 6y ago
  Another option is to use a JVM that offers hard real-time constraints to achieve guaranteed performance characteristics. Such JVMs are in use in the automotive and airline industries---JamaicaVM, for example, offers deterministic garbage collection.
logicchains 6y ago
>A few GC parameter tweaks brought our latency below 20ms and the application no longer drops calls. Tick to trade latency for a modern software HFT stack needs to be under 10 _microseconds_, which is over three orders of magnitude more stringent.
- ridaj 6y ago

patrec 6y ago

Why did your system dynamically allocate memory in the first place? High speed messaging sounds like an area where it should be doable to figure out at startup what resources you need to meet your latency and throughput requirements and then allocate them once and basically never gc anything. Was there some fundamental difficulty with that, or was it more that by the time your realized that GC would kill you, all your code was already written in a way that would have made it prohibitively expensive to switch to an architecture that avoids creating temporary objects?

taneq 6y ago
The best part about using a garbage collected language for high performance software is it's exactly the same as a non-garbage-collected language.
- patrec 6y ago
  Sure, as long as your non-garbage-collected language from ten years ago had memory safety (and general lack of UB footguns), first class IDE support, excellent introspection and profiling facilities and a large potential hiring pool of elite programmers and wide acceptance in the industries you were trying to sell into.
  - taneq 6y ago
    Wait, are we still comparing C++ with Java here?
    - patrec 6y ago
      Yes? If you write zero or low allocation code in Java you pay a price for working against the language and you also lack several tools for writing fast code that C++ provides you with (as well as a bunch of useful abstractions), but your development experience will obviously be much better on a number of other dimensions.
  - fishermanbill 6y ago

jjav 6y ago

You should not have any full GC (pause) events, ever, in properly written production Java code. Source: I worked on several extremely high performance Java systems at Sun and also worked optimizing various Java servers at subsequent jobs based on my Sun experience. At one company (after Sun) I took on a backend service which was doing GC pauses every 3-5 minutes, the code was a dumpster fire mess. After cleaning things up we didn't hit a single full GC in two months of production use (they redeployed every two months so that's as long as the process lasted).

blub 6y ago
Extremely high performance properly written Java code has been mentioned so many times in Java performance discussions that it has acquired an almost legendary status, right there with the sufficiently smart compiler.
- RhysU 6y ago
  I have observed the former, written by exceptional coworkers, but never the latter.
zozbot234 6y ago
Maybe, but that just raises the question...If you're coding your system under the constraint that you should not be having any GC-event, what's the point of picking a GC language in the first place?
- tsimionescu 6y ago
  They said no full GC. The GC runs constantly in parallel with your program and cleans up what it can, but never has to pause execution of the whole program to reclaim even more memory. Not claiming I know how to do this, but I believe this is what they claiming.
- jjav 6y ago
  As others said, no full GC events. Young generation GC is cheap, no need to avoid that. Productivity is the point. Even while paying attention to GC, it's much faster to write good code in Java than to worry about every malloc() in C. I love C, but if I need to churn out high performance code quickly, Java is the choice.

AtlasBarfed 6y ago

Cassandra has some of these issues as well, despite arena allocation, buffer reuse, and offheap manually allocated memory. 10 years has seen improvements to GC, but not eliminated the problem. Cass 4.0 finally enables ZGC, so we'll see how that goes. If you have stateless, you could do two JVMs and monitor the GC state, and reroute requests to a low-heap JVM and let the other clean up. Garbage collection is hard. I'm surprised that there aren't expert-level concepts like weak references and ways to sub-partition the heap, with say a new operator that can target a partition, so major GC doesn't have to do the whole heap. Although that starts getting into rust concepts and analysis.