14 ms·

Virtual threads are going to be great, but they're still limited (still starved the pool when used with 'synchronized' blocks), and they aren't the structured c

by adra 3y ago

Virtual threads are going to be great, but they're still limited (still starved the pool when used with 'synchronized' blocks), and they aren't the structured concurrency power houses like kotlin coroutines, but its an invaluable tool that will only continue to accelerate as the ecosystem moves to adopt them. Expect a lot of libraries to start release versions that are java 21 baseline because of this feature alone. We're in for a little bit of dependency hell for the short while. Thankfully, devs have been exposed to a mostly final loom for a year, so my hope is that at least the big projects are well on their way to quick adoptions. Unlike the 8->11 migration which largely brought pain, the 8->21 release brings with it a ton of value that i think will encourage most shops to actually pull the trigger and finally abandon 8.

tantamounta 3y ago
With the API being nearly the same, I keep just thinking that Virtual Threads are basically identical to Platform Threads except that they use far less memory (so you can have lots more of them). Are there any other actual differences? Better Peformance?
- noelwelsh 3y ago
  The context switch time is much smaller, so yes, better performance.
- pron 3y ago
  The relationship between throughput, latency, and concurrency in servers is expressed via Little's theorem. If your server is written in the thread-per-request style -- the only style for which the platform offers built-in language, VM, and tooling support -- then the most important factor affecting maximum throughput is the number of threads you can have (until, of course, the hardware is fully utilised). Being able to support many threads is the most effective improvement to server throughput you can offer. See: Why User-Mode Threads Are Good for Performance https://youtu.be/07V08SB1l8c https://youtu.be/07V08SB1l8c
  - tantamounta 3y ago
    Thanks for the video. I feel like there's a bit of conflation between the terms "performance(latency)" and "throughput", but I see the point. I'd be interested to see that latency graph (Time marker 15:38) between platform and virtual threads in the case where the server doesn't manufacture a 100ms delay (say, in the case of a caching reverse-proxy). Also - millions of Java programmers thank you for not going to async/await. What an evil source-code virus (among other things that is). I tried to watch it at 1.25x speed as I normally do, but you already talk at 1.25x speed, so no need !
    - pron 3y ago
      To understand what happens when the server doesn't perform IO, apply Little's formula to the CPU only. Clearly, the maximum concurrency would be equal to the number of cores, which means that in that situation there would be no benefit to more threads than cores. What you would see in the graph would be that the server fails once L is equal to the number of cores. The average ratio between IO and CPU time as portions of the average duration would give you an upper limit on how much more throughput you gain by having more threads. That's what I explain at 11:34. Also, both throughput and latency are performance metrics.
  - Svenskunganka 3y ago
    I watched the video and thoroughly enjoyed it, thank you for sharing it! I have a question that is perhaps not entirely related to the video, but it touches the topic of context switches. I've read this post [1] by Chris Hegarty, which explains that when calling the traditionally blocking network I/O APIs in the Java stdlib from a virtual thread, it uses asynchronous/poll-based kernel syscalls (IOCP, kqueue, epoll on Windows, Mac and Linux respectively) which I assume is to avoid blocking the carrier threads. That post was written in 2021, does it still hold true today in Java 21? Reading that, it also makes me wonder what happens for disk I/O? Many other runtimes, both "green thread" ones like Golang and asynchronous like libuv/tokio, use a blocking thread pool (static or elastic) to offload these kernel syscalls to because, from what I've read, those syscalls are not easily made non-blocking like e.g epoll is. Does Java Virtual Threads do the same, or does disk I/O block the carrier threads? For curiosity, does Java file APIs use io_uring on Linux if it is available? It is a fairly recently added kernel API for achieving truly non-blocking I/O, including disk I/O. It doesn't seem to bring much over epoll in terms of performance, but has been a boon for disk I/O and in general can reduce context switches with the kernel by reducing the amount of syscalls needed. [1]: https://inside.java/2021/05/10/networking-io-with-virtual-threads/ https://inside.java/2021/05/10/networking-io-with-virtual-th...
- Mawr 3y ago
  "With the currency being the same, I keep thinking a salary of $50k/year is the same as that of $500k/year. Are there any other actual differences?" Just as with performance improvements [1][2][3][4], the actual impact on the user experience is non-linear and often hard to predict. In the case of virtual threads, you go from needing to consciously work around a limited amount of available threads to spawning one per request and moving on. [1]: https://youtu.be/4XpnKHJAok8?t=3026 https://youtu.be/4XpnKHJAok8?t=3026 [2]: "These tests are fast enough that I can hit enter (my test-running keystroke) and have a response before I have time to think. It means that the flow of my thoughts never breaks." - https://news.ycombinator.com/item?id=7676948 https://news.ycombinator.com/item?id=7676948 [3]: https://news.ycombinator.com/item?id=37277885 https://news.ycombinator.com/item?id=37277885 [4]: "Go’s execution tracer has suffered from high overhead since its inception in 2014. Historically this has forced potential users to worry about up to 20% of CPU overhead when turning it on. Due to this, it's mostly been used in test environments or tricky situations rather than gaining adoption as a continuous profiling signal in production." - https://blog.felixge.de/waiting-for-go1-21-execution-tracing-with-less-than-one-percent-overhead/ https://blog.felixge.de/waiting-for-go1-21-execution-tracing...
pron 3y ago
Structured concurrency in JDK 21 is not only a powerful and flexible library feature, but one that is built deep into the runtime in a way that allows observability into the relationships among threads: https://openjdk.org/jeps/453 https://openjdk.org/jeps/453
- deleted 3y ago
  [deleted]
- logicchains 3y ago
  Do you know if clojure will be adopting these virtual threads, or still using the macro-based approach?
  - deleted 3y ago
    [deleted]
  - bcrosby95 3y ago
    It depends upon what you mean here. You could always set the backing thread pools for core.async and agents in Clojure. That gives you the ability to use virtual threads right now. But in order to avoid thread pinning, there will need to be some code changes to convert some uses of synchronized to ReentrantLock. How fast that happens will depend upon the given library maintainer. Here's an issue for making some of these changes in Clojure's core library: https://clojure.atlassian.net/browse/CLJ-2771 https://clojure.atlassian.net/browse/CLJ-2771 I've tested Clojure's agents with the new virtual threads for my targeted use case they're significantly faster than before - I can spin up tens of thousands of mostly idle agents and reach performance close enough to core.async for me.
- cogman10 3y ago
  It's somewhat unfortunate that structured concurrency ended up being a preview feature in 21. I agree that it's a great addition but man it'd be nice if it made the LTS. As it stands, probably won't be heavily used until Java 25.
  - 3y ago
larperdoodle 3y ago
Why haven't places updated already? It's not that much work to update. Where I work we always go to the new LTS version as soon as it's supported by gradle. Doesn't cross anyone's mind to _not_ upgrade.
- Brystephor 3y ago
  1) dependencies need to be upgraded. for example, not all versions of Gradle support all Java versions. So you need to upgrade Gradle to upgrade Java. 2) other things are deemed to have higher priority. 3) people are satisfied with existing features and don't want to spend energy to upgrade to something that doesn't provide immediate value. 4) folks aren't educated on what the benefit of switching would be so why would it be prioritized? This is a case of "they don't know what they don't know". I work on a team using Java 8 daily. It's fine. It's got things I wish it didn't (no null in switch statements for example) but I don't care about that so much that I'm going to go through the pain of upgrading 7-9 services in the mono repo, their dependencies, and then test them all to be on a new version of Java.
  - krzyk 3y ago
    Gradle/groovy is a liability for any jdk upgrades (similarly like lombok, but it usually supports new JDK at release, not before). We ditched spock because of groovy, and never looked back. Now at jdk 21, previously at 20.
    - brabel 3y ago
      That's pretty disingenuous. Groovy has always run on newer JDK versions before they even get released. One year ago, Gpars already supported Virtual Threads: https://groovy.apache.org/blog/gpars-meets-virtual-threads https://groovy.apache.org/blog/gpars-meets-virtual-threads As a heavy user of Groovy/Spock, though, I agree that upgrading Groovy itself can be challenging, unfortunately. Really depends though on how many edgy Groovy features you relied on :).
      - 3y ago
krzyk 3y ago
You forgot about 8->17 which ads really nice language features, records alone are greatest feature after lambdas. And 21 brings patterns in switch and records.
skwirl 3y ago
Do you have to baseline on Java 21 if you want to add support for virtual threads? Couldn't you continue using heavyweight threads on older versions of Java? My understanding is that both use the same Thread abstraction.
- adra 3y ago
  From an API perpective, you can always use reflection to cheat past the option to create virtual threads in pre-21 (without previews) java bytecode, but you need to do more to your code than just flip the switch to support virtual threads. A virtual thread thread pool by definition is unbound. If you're binding data to a thread (eg. Thread locals, you now have a seemingly unbound list of threads that is now effectively a memory leak). I bumped into that one a few months ago with Netty that has a per thread cache for some things (thankfully you can turn off that cache). It was creating a significantly large waste of RAM that slowed down the application alone. The other big one is as I mentioned the synchronized limitation. If you assume naively that anything can run in a virtual thread without worries, you're opening yourself up to deadlocks or at least significantly low performance code if you're relying on libraries/code that are synchronized using java monitors. There may be more examples of gotchas, but these two are the most notable examples I have right now.
- Quekid5 3y ago
  I believe, e.g. ZIO 2.next is doing something like this, dynamically deciding whether running something async or just doing the blocking thing depending on the availability of VThreads... but of course that's Scala, so YMMV. Without a way to trampoline computation (or transform code appropriately) it's probably impractical to do anything like that. (And of course, still many caveats as the sibling post points out.)
brabel 3y ago
> Expect a lot of libraries to start release versions that are java 21 baseline because of this feature alone. Java has had multi-version jars since 11 I think... that allows library authors to ship code that benefits from new features in newer versions of the JDK while still supporting older ones as well. Hopefully library authors can leverage that, though I'm aware something like Virtual Threads may be very difficult to design around for older versions.
- tadfisher 3y ago
  mrjars are a terrible idea, because libraries are very rarely a single jar, and require dependencies. Often that dependency set is different depending on the runtime version because of backports of newer Java APIs. So either you don't care and make your consumers use proguard to remove the unnecessary backports, or you create a Maven package with variants for each runtime version, which 99% of your downstream will end up using anyway.

by adra 3y ago

tantamounta 3y ago

With the API being nearly the same, I keep just thinking that Virtual Threads are basically identical to Platform Threads except that they use far less memory (so you can have lots more of them). Are there any other actual differences? Better Peformance?

noelwelsh 3y ago
The context switch time is much smaller, so yes, better performance.
pron 3y ago
The relationship between throughput, latency, and concurrency in servers is expressed via Little's theorem. If your server is written in the thread-per-request style -- the only style for which the platform offers built-in language, VM, and tooling support -- then the most important factor affecting maximum throughput is the number of threads you can have (until, of course, the hardware is fully utilised). Being able to support many threads is the most effective improvement to server throughput you can offer. See: Why User-Mode Threads Are Good for Performance https://youtu.be/07V08SB1l8c https://youtu.be/07V08SB1l8c
- tantamounta 3y ago
  Thanks for the video. I feel like there's a bit of conflation between the terms "performance(latency)" and "throughput", but I see the point. I'd be interested to see that latency graph (Time marker 15:38) between platform and virtual threads in the case where the server doesn't manufacture a 100ms delay (say, in the case of a caching reverse-proxy). Also - millions of Java programmers thank you for not going to async/await. What an evil source-code virus (among other things that is). I tried to watch it at 1.25x speed as I normally do, but you already talk at 1.25x speed, so no need !
  - pron 3y ago
    To understand what happens when the server doesn't perform IO, apply Little's formula to the CPU only. Clearly, the maximum concurrency would be equal to the number of cores, which means that in that situation there would be no benefit to more threads than cores. What you would see in the graph would be that the server fails once L is equal to the number of cores. The average ratio between IO and CPU time as portions of the average duration would give you an upper limit on how much more throughput you gain by having more threads. That's what I explain at 11:34. Also, both throughput and latency are performance metrics.
- Svenskunganka 3y ago
  I watched the video and thoroughly enjoyed it, thank you for sharing it! I have a question that is perhaps not entirely related to the video, but it touches the topic of context switches. I've read this post [1] by Chris Hegarty, which explains that when calling the traditionally blocking network I/O APIs in the Java stdlib from a virtual thread, it uses asynchronous/poll-based kernel syscalls (IOCP, kqueue, epoll on Windows, Mac and Linux respectively) which I assume is to avoid blocking the carrier threads. That post was written in 2021, does it still hold true today in Java 21? Reading that, it also makes me wonder what happens for disk I/O? Many other runtimes, both "green thread" ones like Golang and asynchronous like libuv/tokio, use a blocking thread pool (static or elastic) to offload these kernel syscalls to because, from what I've read, those syscalls are not easily made non-blocking like e.g epoll is. Does Java Virtual Threads do the same, or does disk I/O block the carrier threads? For curiosity, does Java file APIs use io_uring on Linux if it is available? It is a fairly recently added kernel API for achieving truly non-blocking I/O, including disk I/O. It doesn't seem to bring much over epoll in terms of performance, but has been a boon for disk I/O and in general can reduce context switches with the kernel by reducing the amount of syscalls needed. [1]: https://inside.java/2021/05/10/networking-io-with-virtual-threads/ https://inside.java/2021/05/10/networking-io-with-virtual-th...
Mawr 3y ago
"With the currency being the same, I keep thinking a salary of $50k/year is the same as that of $500k/year. Are there any other actual differences?" Just as with performance improvements [1][2][3][4], the actual impact on the user experience is non-linear and often hard to predict. In the case of virtual threads, you go from needing to consciously work around a limited amount of available threads to spawning one per request and moving on. [1]: https://youtu.be/4XpnKHJAok8?t=3026 https://youtu.be/4XpnKHJAok8?t=3026 [2]: "These tests are fast enough that I can hit enter (my test-running keystroke) and have a response before I have time to think. It means that the flow of my thoughts never breaks." - https://news.ycombinator.com/item?id=7676948 https://news.ycombinator.com/item?id=7676948 [3]: https://news.ycombinator.com/item?id=37277885 https://news.ycombinator.com/item?id=37277885 [4]: "Go’s execution tracer has suffered from high overhead since its inception in 2014. Historically this has forced potential users to worry about up to 20% of CPU overhead when turning it on. Due to this, it's mostly been used in test environments or tricky situations rather than gaining adoption as a continuous profiling signal in production." - https://blog.felixge.de/waiting-for-go1-21-execution-tracing-with-less-than-one-percent-overhead/ https://blog.felixge.de/waiting-for-go1-21-execution-tracing...

pron 3y ago

Structured concurrency in JDK 21 is not only a powerful and flexible library feature, but one that is built deep into the runtime in a way that allows observability into the relationships among threads: https://openjdk.org/jeps/453 https://openjdk.org/jeps/453

deleted 3y ago
[deleted]
logicchains 3y ago
Do you know if clojure will be adopting these virtual threads, or still using the macro-based approach?
- deleted 3y ago
  [deleted]
- bcrosby95 3y ago
  It depends upon what you mean here. You could always set the backing thread pools for core.async and agents in Clojure. That gives you the ability to use virtual threads right now. But in order to avoid thread pinning, there will need to be some code changes to convert some uses of synchronized to ReentrantLock. How fast that happens will depend upon the given library maintainer. Here's an issue for making some of these changes in Clojure's core library: https://clojure.atlassian.net/browse/CLJ-2771 https://clojure.atlassian.net/browse/CLJ-2771 I've tested Clojure's agents with the new virtual threads for my targeted use case they're significantly faster than before - I can spin up tens of thousands of mostly idle agents and reach performance close enough to core.async for me.
cogman10 3y ago
It's somewhat unfortunate that structured concurrency ended up being a preview feature in 21. I agree that it's a great addition but man it'd be nice if it made the LTS. As it stands, probably won't be heavily used until Java 25.
- 3y ago

larperdoodle 3y ago

Why haven't places updated already? It's not that much work to update. Where I work we always go to the new LTS version as soon as it's supported by gradle. Doesn't cross anyone's mind to _not_ upgrade.

Brystephor 3y ago
1) dependencies need to be upgraded. for example, not all versions of Gradle support all Java versions. So you need to upgrade Gradle to upgrade Java. 2) other things are deemed to have higher priority. 3) people are satisfied with existing features and don't want to spend energy to upgrade to something that doesn't provide immediate value. 4) folks aren't educated on what the benefit of switching would be so why would it be prioritized? This is a case of "they don't know what they don't know". I work on a team using Java 8 daily. It's fine. It's got things I wish it didn't (no null in switch statements for example) but I don't care about that so much that I'm going to go through the pain of upgrading 7-9 services in the mono repo, their dependencies, and then test them all to be on a new version of Java.
- krzyk 3y ago
  Gradle/groovy is a liability for any jdk upgrades (similarly like lombok, but it usually supports new JDK at release, not before). We ditched spock because of groovy, and never looked back. Now at jdk 21, previously at 20.
  - brabel 3y ago
    That's pretty disingenuous. Groovy has always run on newer JDK versions before they even get released. One year ago, Gpars already supported Virtual Threads: https://groovy.apache.org/blog/gpars-meets-virtual-threads https://groovy.apache.org/blog/gpars-meets-virtual-threads As a heavy user of Groovy/Spock, though, I agree that upgrading Groovy itself can be challenging, unfortunately. Really depends though on how many edgy Groovy features you relied on :).
    - 3y ago

krzyk 3y ago

You forgot about 8->17 which ads really nice language features, records alone are greatest feature after lambdas. And 21 brings patterns in switch and records.

skwirl 3y ago

Do you have to baseline on Java 21 if you want to add support for virtual threads? Couldn't you continue using heavyweight threads on older versions of Java? My understanding is that both use the same Thread abstraction.

adra 3y ago
From an API perpective, you can always use reflection to cheat past the option to create virtual threads in pre-21 (without previews) java bytecode, but you need to do more to your code than just flip the switch to support virtual threads. A virtual thread thread pool by definition is unbound. If you're binding data to a thread (eg. Thread locals, you now have a seemingly unbound list of threads that is now effectively a memory leak). I bumped into that one a few months ago with Netty that has a per thread cache for some things (thankfully you can turn off that cache). It was creating a significantly large waste of RAM that slowed down the application alone. The other big one is as I mentioned the synchronized limitation. If you assume naively that anything can run in a virtual thread without worries, you're opening yourself up to deadlocks or at least significantly low performance code if you're relying on libraries/code that are synchronized using java monitors. There may be more examples of gotchas, but these two are the most notable examples I have right now.
Quekid5 3y ago
I believe, e.g. ZIO 2.next is doing something like this, dynamically deciding whether running something async or just doing the blocking thing depending on the availability of VThreads... but of course that's Scala, so YMMV. Without a way to trampoline computation (or transform code appropriately) it's probably impractical to do anything like that. (And of course, still many caveats as the sibling post points out.)

brabel 3y ago

> Expect a lot of libraries to start release versions that are java 21 baseline because of this feature alone. Java has had multi-version jars since 11 I think... that allows library authors to ship code that benefits from new features in newer versions of the JDK while still supporting older ones as well. Hopefully library authors can leverage that, though I'm aware something like Virtual Threads may be very difficult to design around for older versions.

tadfisher 3y ago
mrjars are a terrible idea, because libraries are very rarely a single jar, and require dependencies. Often that dependency set is different depending on the runtime version because of backports of newer Java APIs. So either you don't care and make your consumers use proguard to remove the unnecessary backports, or you create a Maven package with variants for each runtime version, which 99% of your downstream will end up using anyway.