21 ms·

Sub-10 ms Latency in Java: Concurrent GC with Green Threads

154 points by haxen 6y ago

MaxBarraclough 6y ago
Doesn't seem helpful to use the term green threads here. This isn't a JVM with green threads (those are a thing of the past as I understand it). They're using plain old OpenJDK, and they're ensuring the GC gets a CPU core to itself. Neat that they were able to get a dramatic improvement in GC latencies on both G1 and ZGC. No mention of the Shenandoah GC. Would the same trick help out there too?
- brabel 6y ago
  > This isn't a JVM with green threads Exactly... I thought they may be talking about Project Loom's Virtual Threads (which are going to be true green Threads) which are available experimentally as of Java 15, given they did use Java 15, but nothing in the post indicates they used that.
  - haxen 6y ago
    We use the same technique of cooperative multithreading under any name, but without the low-level support to be able to write plain sequential Java code. However, even though that changes our internal programming model, the behavior with respect to native threads, interactions with the OS scheduler, CPU caches, etc., should be identical.
    - pron 6y ago
      Loom's virtual threads aren't cooperative, and the implementation actually does interact with the GCs in non-trivial ways. ZGC support was only added to Loom last week, and might still be unstable, but it would be interesting to test. Please report any finding -- be it performance or stability issues -- to the Loom mailing list.
      - haxen 6y ago
        "Virtual threads" are fibers? When you say they aren't cooperative, you mean there's JVM infrastructure that actually pre-empts a fiber?
        pron 6y ago
        > "Virtual threads" are fibers? Yes, but we found that the name "virtual thread" works better. For one, these fibres are actually an implementation of java.lang.Thread, so it's the same abstraction; for another, when people hear "fibres" they sometimes compare it to other implementations of fibres that are implemented very differently, and that causes confusion. > When you say they aren't cooperative, you mean there's JVM infrastructure that actually pre-empts a fiber? First, I mean that they don't require any explicit yields. Sometimes people confuse "preemptive" with "time-sliced preemption", so I'd rather just say "non-cooperative". Second, there is a capability to forcefully preempt a virtual thread even if it's in some computation loop, but that's up to a chosen scheduler, and the scheduler isn't part of the JVM. You can supply your own scheduler, written in Java, and you'll be able to choose to preempt a runaway thread. Currently, this capability is not exposed, but it will be eventually. See http://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part1.html http://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part1.h...
        haxen 6y ago
        > I mean that they don't require any explicit yields. So in this sense, are Kotlin coroutines also non-cooperative? They don't explicitly yield, but do have a mandatory API to use for "blocking" calls.
        pron 6y ago
        They do explicitly yield, although not with an explicit keyword or call, but with an explicit type. I guess you could say that the "await" is inferred by the compiler, but it's there. That's not the case with virtual threads, which behave more like Go's goroutines or Erlang's processes.
- haxen 6y ago
  We did measure on Shenandoah as well, it helped but not enough to be within 10 ms. Since this post is about Hazelcast Jet getting the best latency, we didn't report that.
  - MaxBarraclough 6y ago
    Interesting, thanks.
- tyingq 6y ago
  They say they are using this: https://hazelcast.com/blog/idle-green-threads-in-jet/ https://hazelcast.com/blog/idle-green-threads-in-jet/ And then say it's comparable. "This basic design is also present in the concepts of green threads and coroutines. In Hazelcast Jet we call them tasklets."
jakewins 6y ago
For years of my life, all I thought about was stuff like this. If you've ever ran latency sensitive systems on the JVM.. man is it ever a pain. Who was it that turned GC off entirely, minimized allocation and just restarted their VMs when they ran out of RAM every couple of hours, was that Netflix? Either way. It makes me excited for Rust and the languages it'll inspire, all this labor gone away.
- pron 6y ago
  In case you missed this post and the previous ones, here's the news: this labour is gone on the JVM. The GCs in JDK 14-15 are good enough pretty much out of the box, and they're getting better very quickly now. OTOH, if you think other languages let you do away with a GC without pretty significant extra work, especially in concurrent systems, well, then you haven't had experience with those languages.
  - VHRanger 6y ago
    In D, the GC is guaranteed to only ever run if you allocate new memory on the heap. This is still a lot of work for a video game (because you never want any latency, the only way to achieve this is with an arena allocator or going full @nogc). But for apps where the latency requirement is bounded, D doesn't make it hard and the language is nice and ergonomic.
    - chrisseaton 6y ago
      > In D, the GC is guaranteed to only ever run if you allocate new memory on the heap. Isn't that how most GCs work? Why would you do anything else? To release memory to the OS? That's not really a priority in most runtime systems, and I think wanting to do that is a pretty niche requirement.
      - frant-hartm 6y ago
        I want memory allocation to run in constant time, so I can achieve low latency. That's the point of concurrent GCs - they run in the background to free up memory.
ginko 6y ago
I would certainly hope that green threads have less than 10 megaseconds latency.
- hnarn 6y ago
  Was the title changed? Because it says "ms" both here and on jet-start.sh and that unit is milliseconds.
  - ginko 6y ago
    It used to be 'Ms'. Guess it's fixed now.
tupac_speedrap 6y ago
Glad to see some love for legacy programming languages like Java.
- shock 6y ago
  I don't think Java qualifies as legacy. I view it rather being in the 'mature' stage of programming languages evolution.
- pron 6y ago
  Well, that's understandable because there are still some legacy companies and organisations left that write a lot of new software in Java, like Apple, Amazon, Netflix, Twitter, Google, Alibaba, Tencent, NASA, GitHub, Microsoft, Facebook, Spotify and nearly all Fortune 500 companies. Plus, if you care about both performance and observability, there aren't many viable alternatives. BTW, many if not most of the cutting-edge advances in compilation, low-overhead deep profiling, and garbage collection are done on the Java platform, so it's still the technology leader in those areas.
- pjmlp 6y ago
  If Java is legacy with 25 years, what to say about C++ with 50, or C with 60, Python and Ruby both around 30. Ah and the beloved OS around here is reaching 60 as well. Plenty of legacy love.
- dionian 6y ago
  The Java language is Java's least appealing feature. Fortunately there are seamless alternatives!
ryanthedev 6y ago
Can we talk about no async/await support? Can't build scalable apps when I'm in callback hell. It's like ES5 all over again.
- capableweb 6y ago
  > Can't build scalable apps when I'm in callback hell What? We've (JavaScript and developers dealing with asynchronous patterns) been able to build scalable (in terms of code and its maintenance) for many many years, probably 10+. async/wait is simply syntactic sugar and doesn't drastically change anything, you still need to understand the asynchronicity underneath it all, and if you do, you won't have any problems building scalable apps using your knowledge.
  - ryanthedev 6y ago
    Pretty sure a callback vs synchronous style of coding is a little more than syntactic sugar. It's a complete shift in application design... If I have to flatmap one more time...
    - capableweb 6y ago
      > Pretty sure a callback vs synchronous style of coding is a little more than syntactic sugar Absolutely, asynchronous and synchronous are two very different patterns when programming with very different trade offs for both of them, but async/await is not synchronous, it only make that particular call _look_ synchronous, while actually being asynchronous. Failing to understand that async/await is just syntactic sugar for dealing with asynchronous programming will sooner or later bite you. Here is a gist that shows why async/await is not really synchronous (blocking) programming: https://gist.github.com/matt-mcalister/3f060bf32d292cfebc9448e210e043e2 https://gist.github.com/matt-mcalister/3f060bf32d292cfebc944...
- kasperni 6y ago
  Java is getting virtual threads/fibers instead of async/await. From https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.html https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.ht... ------------- An alternative solution to that of fibers to concurrency's simplicity vs. performance issue is known as async/await, and has been adopted by C# and Node.js, and will likely be adopted by standard JavaScript. Continuations and fibers dominate async/await in the sense that async/await is easily implemented with continuations (in fact, it can be implemented with a weak form of delimited continuations known as stackless continuations, that don't capture an entire call-stack but only the local context of a single subroutine), but not vice-versa. While implementing async/await is easier than full-blown continuations and fibers, that solution falls far too short of addressing the problem. While async/await makes code simpler and gives it the appearance of normal, sequential code, like asynchronous code it still requires significant changes to existing code, explicit support in libraries, and does not interoperate well with synchronous code. In other words, it does not solve what's known as the "colored function" problem.
hansdieter1337 6y ago
I want to see Java on mission critical computers in space!
- pjmlp 6y ago
  Not on space, but still mission critical, "Aegis Battleship Weapons System" http://www.artist-embedded.org/docs/Events/2011/JTRES/Slides/JTRES-Winkler.pdf http://www.artist-embedded.org/docs/Events/2011/JTRES/Slides... "French radar system for ballistic missile tracking and measurement" https://www.militaryaerospace.com/defense-executive/article/16720700/thales-chooses-aonix-perc-virtual-machine-software-for-ballistic-missile-radar https://www.militaryaerospace.com/defense-executive/article/... "NASA Ground Control Station for Multiple UAVs Flight Simulation" https://www.semanticscholar.org/paper/Ground-Control-Station-for-Multiple-UAVs-Flight-Angonese-Rosa/12e4a305fd7251803c743f1ad96718c72d198165 https://www.semanticscholar.org/paper/Ground-Control-Station...
- Rebelgecko 6y ago
  I've seen plenty of ground station and mission control software in Java. Actually in space a bit less likely though... A lot of that is running in RTOS environments that Java isn't well suited for (forget garbage collection, some flight software projects go as far as to ban dynamic memory allocation)
jjav 6y ago
Even on ten year old hardware single digit ms latency in Java server apps wasn't very special. Java (JVM) is an extremely performant platform so I always find odd how a meme has somehow built up on it being otherwise. True that one can end up writing terribly inefficient Java code, but one can write terrible code in any language. If I need to write server code where performance is particularly important and I don't want to deal with the cost (in debug time and dev expertise) of C or C++, Java would be my first choice. Also I'm of the school of thought that performance always matters. Autoscaling in cloud providers sure makes it easy to scale horizontally to make up for slow server code, but once you reach certain size, go have a chat with the finance team about the AWS bill.
widulola 6y ago
Like operator overloading and implicit conversations. https://newpornsites.pro/category/porn-sharing-sites/ https://newpornsites.pro/category/porn-sharing-sites/

154 points by haxen 6y ago

Doesn't seem helpful to use the term green threads here. This isn't a JVM with green threads (those are a thing of the past as I understand it). They're using plain old OpenJDK, and they're ensuring the GC gets a CPU core to itself. Neat that they were able to get a dramatic improvement in GC latencies on both G1 and ZGC. No mention of the Shenandoah GC. Would the same trick help out there too?

brabel 6y ago
> This isn't a JVM with green threads Exactly... I thought they may be talking about Project Loom's Virtual Threads (which are going to be true green Threads) which are available experimentally as of Java 15, given they did use Java 15, but nothing in the post indicates they used that.
- haxen 6y ago
  We use the same technique of cooperative multithreading under any name, but without the low-level support to be able to write plain sequential Java code. However, even though that changes our internal programming model, the behavior with respect to native threads, interactions with the OS scheduler, CPU caches, etc., should be identical.
  - pron 6y ago
    Loom's virtual threads aren't cooperative, and the implementation actually does interact with the GCs in non-trivial ways. ZGC support was only added to Loom last week, and might still be unstable, but it would be interesting to test. Please report any finding -- be it performance or stability issues -- to the Loom mailing list.
    - haxen 6y ago
      "Virtual threads" are fibers? When you say they aren't cooperative, you mean there's JVM infrastructure that actually pre-empts a fiber?
      - pron 6y ago
        > "Virtual threads" are fibers? Yes, but we found that the name "virtual thread" works better. For one, these fibres are actually an implementation of java.lang.Thread, so it's the same abstraction; for another, when people hear "fibres" they sometimes compare it to other implementations of fibres that are implemented very differently, and that causes confusion. > When you say they aren't cooperative, you mean there's JVM infrastructure that actually pre-empts a fiber? First, I mean that they don't require any explicit yields. Sometimes people confuse "preemptive" with "time-sliced preemption", so I'd rather just say "non-cooperative". Second, there is a capability to forcefully preempt a virtual thread even if it's in some computation loop, but that's up to a chosen scheduler, and the scheduler isn't part of the JVM. You can supply your own scheduler, written in Java, and you'll be able to choose to preempt a runaway thread. Currently, this capability is not exposed, but it will be eventually. See http://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part1.html http://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part1.h...
        haxen 6y ago
        > I mean that they don't require any explicit yields. So in this sense, are Kotlin coroutines also non-cooperative? They don't explicitly yield, but do have a mandatory API to use for "blocking" calls.
        pron 6y ago
        They do explicitly yield, although not with an explicit keyword or call, but with an explicit type. I guess you could say that the "await" is inferred by the compiler, but it's there. That's not the case with virtual threads, which behave more like Go's goroutines or Erlang's processes.
haxen 6y ago
We did measure on Shenandoah as well, it helped but not enough to be within 10 ms. Since this post is about Hazelcast Jet getting the best latency, we didn't report that.
- MaxBarraclough 6y ago
  Interesting, thanks.
tyingq 6y ago
They say they are using this: https://hazelcast.com/blog/idle-green-threads-in-jet/ https://hazelcast.com/blog/idle-green-threads-in-jet/ And then say it's comparable. "This basic design is also present in the concepts of green threads and coroutines. In Hazelcast Jet we call them tasklets."

jakewins 6y ago

For years of my life, all I thought about was stuff like this. If you've ever ran latency sensitive systems on the JVM.. man is it ever a pain. Who was it that turned GC off entirely, minimized allocation and just restarted their VMs when they ran out of RAM every couple of hours, was that Netflix? Either way. It makes me excited for Rust and the languages it'll inspire, all this labor gone away.

pron 6y ago
In case you missed this post and the previous ones, here's the news: this labour is gone on the JVM. The GCs in JDK 14-15 are good enough pretty much out of the box, and they're getting better very quickly now. OTOH, if you think other languages let you do away with a GC without pretty significant extra work, especially in concurrent systems, well, then you haven't had experience with those languages.
- VHRanger 6y ago
  In D, the GC is guaranteed to only ever run if you allocate new memory on the heap. This is still a lot of work for a video game (because you never want any latency, the only way to achieve this is with an arena allocator or going full @nogc). But for apps where the latency requirement is bounded, D doesn't make it hard and the language is nice and ergonomic.
  - chrisseaton 6y ago
    > In D, the GC is guaranteed to only ever run if you allocate new memory on the heap. Isn't that how most GCs work? Why would you do anything else? To release memory to the OS? That's not really a priority in most runtime systems, and I think wanting to do that is a pretty niche requirement.
    - frant-hartm 6y ago
      I want memory allocation to run in constant time, so I can achieve low latency. That's the point of concurrent GCs - they run in the background to free up memory.

ginko 6y ago

I would certainly hope that green threads have less than 10 megaseconds latency.

hnarn 6y ago
Was the title changed? Because it says "ms" both here and on jet-start.sh and that unit is milliseconds.
- ginko 6y ago
  It used to be 'Ms'. Guess it's fixed now.

tupac_speedrap 6y ago

Glad to see some love for legacy programming languages like Java.

shock 6y ago
I don't think Java qualifies as legacy. I view it rather being in the 'mature' stage of programming languages evolution.
pron 6y ago
Well, that's understandable because there are still some legacy companies and organisations left that write a lot of new software in Java, like Apple, Amazon, Netflix, Twitter, Google, Alibaba, Tencent, NASA, GitHub, Microsoft, Facebook, Spotify and nearly all Fortune 500 companies. Plus, if you care about both performance and observability, there aren't many viable alternatives. BTW, many if not most of the cutting-edge advances in compilation, low-overhead deep profiling, and garbage collection are done on the Java platform, so it's still the technology leader in those areas.
pjmlp 6y ago
If Java is legacy with 25 years, what to say about C++ with 50, or C with 60, Python and Ruby both around 30. Ah and the beloved OS around here is reaching 60 as well. Plenty of legacy love.
dionian 6y ago
The Java language is Java's least appealing feature. Fortunately there are seamless alternatives!

ryanthedev 6y ago

Can we talk about no async/await support? Can't build scalable apps when I'm in callback hell. It's like ES5 all over again.

capableweb 6y ago
> Can't build scalable apps when I'm in callback hell What? We've (JavaScript and developers dealing with asynchronous patterns) been able to build scalable (in terms of code and its maintenance) for many many years, probably 10+. async/wait is simply syntactic sugar and doesn't drastically change anything, you still need to understand the asynchronicity underneath it all, and if you do, you won't have any problems building scalable apps using your knowledge.
- ryanthedev 6y ago
  Pretty sure a callback vs synchronous style of coding is a little more than syntactic sugar. It's a complete shift in application design... If I have to flatmap one more time...
  - capableweb 6y ago
    > Pretty sure a callback vs synchronous style of coding is a little more than syntactic sugar Absolutely, asynchronous and synchronous are two very different patterns when programming with very different trade offs for both of them, but async/await is not synchronous, it only make that particular call _look_ synchronous, while actually being asynchronous. Failing to understand that async/await is just syntactic sugar for dealing with asynchronous programming will sooner or later bite you. Here is a gist that shows why async/await is not really synchronous (blocking) programming: https://gist.github.com/matt-mcalister/3f060bf32d292cfebc9448e210e043e2 https://gist.github.com/matt-mcalister/3f060bf32d292cfebc944...
kasperni 6y ago
Java is getting virtual threads/fibers instead of async/await. From https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.html https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.ht... ------------- An alternative solution to that of fibers to concurrency's simplicity vs. performance issue is known as async/await, and has been adopted by C# and Node.js, and will likely be adopted by standard JavaScript. Continuations and fibers dominate async/await in the sense that async/await is easily implemented with continuations (in fact, it can be implemented with a weak form of delimited continuations known as stackless continuations, that don't capture an entire call-stack but only the local context of a single subroutine), but not vice-versa. While implementing async/await is easier than full-blown continuations and fibers, that solution falls far too short of addressing the problem. While async/await makes code simpler and gives it the appearance of normal, sequential code, like asynchronous code it still requires significant changes to existing code, explicit support in libraries, and does not interoperate well with synchronous code. In other words, it does not solve what's known as the "colored function" problem.

hansdieter1337 6y ago

I want to see Java on mission critical computers in space!

pjmlp 6y ago
Not on space, but still mission critical, "Aegis Battleship Weapons System" http://www.artist-embedded.org/docs/Events/2011/JTRES/Slides/JTRES-Winkler.pdf http://www.artist-embedded.org/docs/Events/2011/JTRES/Slides... "French radar system for ballistic missile tracking and measurement" https://www.militaryaerospace.com/defense-executive/article/16720700/thales-chooses-aonix-perc-virtual-machine-software-for-ballistic-missile-radar https://www.militaryaerospace.com/defense-executive/article/... "NASA Ground Control Station for Multiple UAVs Flight Simulation" https://www.semanticscholar.org/paper/Ground-Control-Station-for-Multiple-UAVs-Flight-Angonese-Rosa/12e4a305fd7251803c743f1ad96718c72d198165 https://www.semanticscholar.org/paper/Ground-Control-Station...
Rebelgecko 6y ago
I've seen plenty of ground station and mission control software in Java. Actually in space a bit less likely though... A lot of that is running in RTOS environments that Java isn't well suited for (forget garbage collection, some flight software projects go as far as to ban dynamic memory allocation)

jjav 6y ago

Even on ten year old hardware single digit ms latency in Java server apps wasn't very special. Java (JVM) is an extremely performant platform so I always find odd how a meme has somehow built up on it being otherwise. True that one can end up writing terribly inefficient Java code, but one can write terrible code in any language. If I need to write server code where performance is particularly important and I don't want to deal with the cost (in debug time and dev expertise) of C or C++, Java would be my first choice. Also I'm of the school of thought that performance always matters. Autoscaling in cloud providers sure makes it easy to scale horizontally to make up for slow server code, but once you reach certain size, go have a chat with the finance team about the AWS bill.

widulola 6y ago

Like operator overloading and implicit conversations. https://newpornsites.pro/category/porn-sharing-sites/ https://newpornsites.pro/category/porn-sharing-sites/