5 ms·

Java virtual threads hit with pinning issue

49 points by cempaka 3y ago

Traubenfuchs 3y ago
[flagged]
- pylua 3y ago
  Have there been any benchmarks compared to netty for real world use? Or any comparisons ? That would be useful for those of us considering a switch.
  - cogman10 3y ago
    https://www.infoq.com/articles/helidon-4-adopts-virtual-threads/ https://www.infoq.com/articles/helidon-4-adopts-virtual-thre...
    - pylua 3y ago
      That’s really interesting —- looks like there is a clear winner. I am curious as to why non blocking is slower. I also wonder if the reactive libraries could be refactored to use the virtual threading model. It would help with a lot of the debugging issues. I would be sad to see those libraries go by the wayside as they do make a lot of complicated tasks simpler once you understand it.
      - brabel 3y ago
        I would think the old async stuff is slower because it intrinsically requires much more memory allocation as you need to create objects and store state for everything. If you used Virtual Threads under the hood, those problems would not just disappear.
        cogman10 3y ago
        It's usually a wash, maybe even a little against virtual threads as they have to store stack information along side all the state for the current context. One of the major changes to the JVM to make virtual threads work revolved around setting up the stack to be as light weight as possible.
      - cogman10 3y ago
        It was only slightly slower. (Netty vs virtual threads) that could easily just come down to implementation differences. The take away, IMO, is that without major architecture you can use blocking code and get near (and sometimes over) the same performance as you'd get with non-blocking code. > I also wonder if the reactive libraries could be refactored to use the virtual threading model. You could, but ultimately the question is "why?". One of the benefits of virtual threads is that the reactive model isn't as appealing. That said, it's definitely possible for the reactive libraries to simply use virtual threads whenever they do anything in parallel. That'd allow the model to avoid potential issues when calling into blocking code. There might be a better avenue in the future if the JDK devs expose the underlying coroutine API (or parts of it) that powers virtual threads today. > I would be sad to see those libraries go by the wayside as they do make a lot of complicated tasks simpler once you understand it. The benefit of virtual threads is making things even simpler so you don't need to understand those complicated constructs. You just spawn new threads whenever you need to run things in parallel.
        pylua 3y ago
        > It was only slightly slower. (Netty vs virtual threads) that could easily just come down to implementation differences. I think this might be it. After my post I thought about it and guessed that maybe the workload was potentially not an optimal netty workload, or there are aspects to the workloak that are not well suited for netty. I am unsure, however. > That said, it's definitely possible for the reactive libraries to simply use virtual threads whenever they do anything in parallel. That'd allow the model to avoid potential issues when calling into blocking code. Definitely would be a great enhancement for the elastic thread pool. > The benefit of virtual threads is making things even simpler so you don't need to understand those complicated constructs. You just spawn new threads whenever you need to run things in parallel. Once you get used to it, in my opinion, those constructs are simpler and easier to use than the standard multithreading options in java, specifically flux and timed operations. However, some items like request scope or session scope is a problem and harder (impossible?) to do right.
- metaltyphoon 3y ago
  Unlike these folks from dotnet, which tested directly on ASP for real workload https://github.com/dotnet/runtimelab/issues/2398
  - sebazzz 3y ago
    > Interop with native code in green threads model is complex and comparatively slow . With a benchmark of a minimal P/Invoke, the cost of making 100,000,000 P/Invoke calls changed from 300ms to about 1800ms when running on a green thread. This was expected as similar issues impact other languages implementing green threads. We found that there are surprising functional issues in interactions with code which uses thread-local static variables or exposes native thread state. > Interactions with security mitigations such as shadow stacks intended to protect against return-oriented programming would be quite challenging.
xbar 3y ago
Alan Bateman's FOSDEM presentation that explains this issues and some solutions in the meanwhile: https://inside.java/2024/02/17/virtual-threads-next-steps/ https://inside.java/2024/02/17/virtual-threads-next-steps/
adrianmsmith 3y ago
This is not news, it's documented in the official description of the virtual threads: https://openjdk.org/jeps/444 https://openjdk.org/jeps/444
- bilbo0s 3y ago
  OK. Good. I’m glad someone else said this, since it means I’m not crazy. I was reading that thinking, man, this sounds like the way it’s outlined to work? Where’s the new issue?
  - smaudet 3y ago
    It's news if you are are uneducated about synchronization/threading in general (it's extremely difficult to do well so it stands to reason most people won't even understand the issues). It's also news that they are trying to provide some abstraction to help. Which is great, although as usual I suspect it will need to be applied per situation, not a one-size-fits all.
  - CHY872 3y ago
    Nah, it's not normal or really documented. Normally when languages have async, they end up with dual primitives - the synchronous one and the asynchronous one, and everyone learns that if they do the blocking operation on the async thread, they deserve their deadlock. Generally you then end up with two variants of the API, with the standard function colouring problem where you can call lockBlocking() from a non-async context, lockAsync() from an async, and woe betide you if you get mixed up. Ideally your language can help you avoid these issues sometimes (e.g. with Rust you can't hold a non-async lock over await points). Java instead did a big thing where they made all the primitives work fine in both cases with one API, something which is honestly really hard and really reflects the level of thought put into modern Java features. But there's a long tail of stuff that's still getting cleaned up (e.g. various weird I/O apis), and honestly I think it _is_ weird that an actual language keyword made it into the long tail. It's extra fun that it's actually quite hard to hit performance problems as a result of this, as the JVM will actually detect that it's getting to this problem and boot up threads to compensate.
- 3y ago
gudzpoz 3y ago
By the way, there was a recent HN discussion on a real-world example of deadlock caused by `synchronized` pinning ("We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres") [1]. [1] https://news.ycombinator.com/item?id=39008026 https://news.ycombinator.com/item?id=39008026
crustycoder 3y ago
I didn't watch the preso (TL;DW) but isn't this just a case of good old fashioned Priority Inversion? https://en.wikipedia.org/wiki/Priority_inversion https://en.wikipedia.org/wiki/Priority_inversion If so, it's pretty much an intractable problem, and is the reason (for example) why Solaris abandoned the two-level thread model and went back to a single level one? https://flylib.com/books/en/2.830.1.14/1/ https://flylib.com/books/en/2.830.1.14/1/ But then again, as I said, TL;DW ;-)
- CHY872 3y ago
  I think this is a bit more subtle and nasty than bog standard priority inversion. Specifically, in present virtual threads, if you take out a lock with synchronized, if you then do a normally-non-blocking asynchronous API call while you have the lock, you block one of your very few OS threads because the virtual thread can't now be migrated off the OS thread. IMO the compound thing is what makes it be nasty. E.g. you have a function `doSomething` which does some RPC, and that's all nicely non-blocking. But someone called map.computeIfAbsent(x, k -> doSomething(k)), and that uses synchronized on the inside so now your non-blocking API calls all magically became blocking, no further action required.
  - crustycoder 3y ago
    Thanks for the explanation and - Ugh :-(
CHY872 3y ago
The specific edge case which is most annoying here is that locking a Java object with synchronized converts all the non-blocking calls that Loom does further down the stack to be blocking ones, which then lock up your carrier threads. So you get this action as a distance thing. E.g. for (int i = 0; i < 500; i++) { newVirtualThread(() -> synchronized (new Object()) { Thread.sleep(100_000); }); } will blow up your JVM (modulo some compensation mechanisms that work definitely kinda) and that's odd.

49 points by cempaka 3y ago

Traubenfuchs 3y ago

[flagged]

pylua 3y ago
Have there been any benchmarks compared to netty for real world use? Or any comparisons ? That would be useful for those of us considering a switch.
- cogman10 3y ago
  https://www.infoq.com/articles/helidon-4-adopts-virtual-threads/ https://www.infoq.com/articles/helidon-4-adopts-virtual-thre...
  - pylua 3y ago
    That’s really interesting —- looks like there is a clear winner. I am curious as to why non blocking is slower. I also wonder if the reactive libraries could be refactored to use the virtual threading model. It would help with a lot of the debugging issues. I would be sad to see those libraries go by the wayside as they do make a lot of complicated tasks simpler once you understand it.
    - brabel 3y ago
      I would think the old async stuff is slower because it intrinsically requires much more memory allocation as you need to create objects and store state for everything. If you used Virtual Threads under the hood, those problems would not just disappear.
      - cogman10 3y ago
        It's usually a wash, maybe even a little against virtual threads as they have to store stack information along side all the state for the current context. One of the major changes to the JVM to make virtual threads work revolved around setting up the stack to be as light weight as possible.
    - cogman10 3y ago
      It was only slightly slower. (Netty vs virtual threads) that could easily just come down to implementation differences. The take away, IMO, is that without major architecture you can use blocking code and get near (and sometimes over) the same performance as you'd get with non-blocking code. > I also wonder if the reactive libraries could be refactored to use the virtual threading model. You could, but ultimately the question is "why?". One of the benefits of virtual threads is that the reactive model isn't as appealing. That said, it's definitely possible for the reactive libraries to simply use virtual threads whenever they do anything in parallel. That'd allow the model to avoid potential issues when calling into blocking code. There might be a better avenue in the future if the JDK devs expose the underlying coroutine API (or parts of it) that powers virtual threads today. > I would be sad to see those libraries go by the wayside as they do make a lot of complicated tasks simpler once you understand it. The benefit of virtual threads is making things even simpler so you don't need to understand those complicated constructs. You just spawn new threads whenever you need to run things in parallel.
      - pylua 3y ago
        > It was only slightly slower. (Netty vs virtual threads) that could easily just come down to implementation differences. I think this might be it. After my post I thought about it and guessed that maybe the workload was potentially not an optimal netty workload, or there are aspects to the workloak that are not well suited for netty. I am unsure, however. > That said, it's definitely possible for the reactive libraries to simply use virtual threads whenever they do anything in parallel. That'd allow the model to avoid potential issues when calling into blocking code. Definitely would be a great enhancement for the elastic thread pool. > The benefit of virtual threads is making things even simpler so you don't need to understand those complicated constructs. You just spawn new threads whenever you need to run things in parallel. Once you get used to it, in my opinion, those constructs are simpler and easier to use than the standard multithreading options in java, specifically flux and timed operations. However, some items like request scope or session scope is a problem and harder (impossible?) to do right.
metaltyphoon 3y ago
Unlike these folks from dotnet, which tested directly on ASP for real workload https://github.com/dotnet/runtimelab/issues/2398
- sebazzz 3y ago
  > Interop with native code in green threads model is complex and comparatively slow . With a benchmark of a minimal P/Invoke, the cost of making 100,000,000 P/Invoke calls changed from 300ms to about 1800ms when running on a green thread. This was expected as similar issues impact other languages implementing green threads. We found that there are surprising functional issues in interactions with code which uses thread-local static variables or exposes native thread state. > Interactions with security mitigations such as shadow stacks intended to protect against return-oriented programming would be quite challenging.

xbar 3y ago

Alan Bateman's FOSDEM presentation that explains this issues and some solutions in the meanwhile: https://inside.java/2024/02/17/virtual-threads-next-steps/ https://inside.java/2024/02/17/virtual-threads-next-steps/

adrianmsmith 3y ago

This is not news, it's documented in the official description of the virtual threads: https://openjdk.org/jeps/444 https://openjdk.org/jeps/444

bilbo0s 3y ago
OK. Good. I’m glad someone else said this, since it means I’m not crazy. I was reading that thinking, man, this sounds like the way it’s outlined to work? Where’s the new issue?
- smaudet 3y ago
  It's news if you are are uneducated about synchronization/threading in general (it's extremely difficult to do well so it stands to reason most people won't even understand the issues). It's also news that they are trying to provide some abstraction to help. Which is great, although as usual I suspect it will need to be applied per situation, not a one-size-fits all.
- CHY872 3y ago
  Nah, it's not normal or really documented. Normally when languages have async, they end up with dual primitives - the synchronous one and the asynchronous one, and everyone learns that if they do the blocking operation on the async thread, they deserve their deadlock. Generally you then end up with two variants of the API, with the standard function colouring problem where you can call lockBlocking() from a non-async context, lockAsync() from an async, and woe betide you if you get mixed up. Ideally your language can help you avoid these issues sometimes (e.g. with Rust you can't hold a non-async lock over await points). Java instead did a big thing where they made all the primitives work fine in both cases with one API, something which is honestly really hard and really reflects the level of thought put into modern Java features. But there's a long tail of stuff that's still getting cleaned up (e.g. various weird I/O apis), and honestly I think it _is_ weird that an actual language keyword made it into the long tail. It's extra fun that it's actually quite hard to hit performance problems as a result of this, as the JVM will actually detect that it's getting to this problem and boot up threads to compensate.
3y ago

gudzpoz 3y ago

By the way, there was a recent HN discussion on a real-world example of deadlock caused by `synchronized` pinning ("We switched to Java 21 virtual threads and got a deadlock in TPC-C for Postgres") [1]. [1] https://news.ycombinator.com/item?id=39008026 https://news.ycombinator.com/item?id=39008026

crustycoder 3y ago

I didn't watch the preso (TL;DW) but isn't this just a case of good old fashioned Priority Inversion? https://en.wikipedia.org/wiki/Priority_inversion https://en.wikipedia.org/wiki/Priority_inversion If so, it's pretty much an intractable problem, and is the reason (for example) why Solaris abandoned the two-level thread model and went back to a single level one? https://flylib.com/books/en/2.830.1.14/1/ https://flylib.com/books/en/2.830.1.14/1/ But then again, as I said, TL;DW ;-)

CHY872 3y ago
I think this is a bit more subtle and nasty than bog standard priority inversion. Specifically, in present virtual threads, if you take out a lock with synchronized, if you then do a normally-non-blocking asynchronous API call while you have the lock, you block one of your very few OS threads because the virtual thread can't now be migrated off the OS thread. IMO the compound thing is what makes it be nasty. E.g. you have a function `doSomething` which does some RPC, and that's all nicely non-blocking. But someone called map.computeIfAbsent(x, k -> doSomething(k)), and that uses synchronized on the inside so now your non-blocking API calls all magically became blocking, no further action required.
- crustycoder 3y ago
  Thanks for the explanation and - Ugh :-(

CHY872 3y ago

The specific edge case which is most annoying here is that locking a Java object with synchronized converts all the non-blocking calls that Loom does further down the stack to be blocking ones, which then lock up your carrier threads. So you get this action as a distance thing. E.g. for (int i = 0; i < 500; i++) { newVirtualThread(() -> synchronized (new Object()) { Thread.sleep(100_000); }); } will blow up your JVM (modulo some compensation mechanisms that work definitely kinda) and that's odd.