6 ms·

The problem with the naïve design is that even with virtual threads, you risk running out of (heap) memory if the threads ever block. Each task makes a bit of p

The problem with the naïve design is that even with virtual threads, you risk running out of (heap) memory if the threads ever block. Each task makes a bit of progress, allocates some objects, and then lets another one do the same thing. With virtual threads, you can limit the damage by using a semaphore, but you still need to tune the size. This isn't much different than sizing a traditional thread pool, and so I'm not sure what benefit virtual threads will really have in practice. You're swapping one config for another.

immibis 2y ago
Async does exactly the same by the way.
initplus 2y ago
The benefits from virtual threads come from the simple API that it presents to the programmer. It's not a performance optimization.
- hashmash 2y ago
  But that same benefit was always available with platform threads -- a simple API. What is the real gain by using virtual threads? It's either going to be performance or memory utilization.
  - groestl 2y ago
    It's combining the benefits from async models (state machines separated from os threads, thus more optimal for I/O bound workload), with the benefits from proper threading models (namely the simpler human interface). Memory utilization & performance is going to be similar to the async callback mess.
    - hashmash 2y ago
      Why is an async model better than using OS threads for an I/O bound workload? The OS is doing async stuff internally and shielding the complexity with threads. With virtual threads this work has shifted to the JVM. Can the JVM do threads better than the OS?
      - adgjlsfhk1 2y ago
        it can do a much better job because there isn't a security boundary. OS thread scheduling requires sys calls and invalidate a bunch of cache to prevent timing leaks
      - zokier 2y ago
        > Can the JVM do threads better than the OS? Yes. The JVM has far more opportunities for optimizing threads because it doesn't need to uphold 50 years of accumulated invariants and compatibility that current OSes do, and JVM has more visibilty on the application internals.
      - mrsilencedogood 2y ago
        "Why is an async model better than using OS threads for an I/O bound workload?" Because evented/callback-driven code is a nightmare to reason about and breaks lots of very basic tools, like the humble stack trace. Another big thing for me is resource management - try/finally don't work across callback boundaries, but do work within a virtual thread. I recently ported a netty-based evented system to virtual threads and a very long-standing issue - resource leakage - turned into one very nice try/finally block.
        deleted 2y ago
        [deleted]
  - lichtenberger 2y ago
    Throughput. The code can be "suspended" on a blocking call (I/O, where the platform thread usually is wasted, as the CPU has nothing to do during this time). So, the platform thread can do other work in the meantime.
  - CrimsonRain 2y ago
    Create 100k platform threads and you'll find out.
packetlost 2y ago
Yeah, and it's generally good to be RAM limited instead of CPU, no? The alternative is blowing a bunch of time on syscalls and OS scheduler overhead. Also the virtual threads run on a "traditional" thread pool to my understanding, so you can just tweak the number of worker threads to cap the total concurrency. The benefit is it's overall more efficient (in the general case) and lets you write linear blocking code (as opposed to function coloring). You don't have to use it, but it's nice that it's there. Now hopefully Valhalla actually makes it in eventually
- hashmash 2y ago
  The OS scheduler is still there (for the carrier threads), but now you've added on top of that FJ pool based scheduler overhead. Although virtual threads don't have the syscall overhead when they block, there's a new cost caused by allocating the internal continuation object, and copying state into it. This puts more pressure on the garbage collector. Context switching cost due to CPU cache thrashing doesn't go away regardless of which type of thread you're using. I've not yet seen a study that shows that virtual threads offer a huge benefit. The Open Liberty study suggests that they're worse than the existing platform threads.
  - zokier 2y ago
    > The OS scheduler is still there (for the carrier threads), but now you've added on top of that FJ pool based scheduler overhead. Ideally carrier threads would be pinned to isolated cpu cores, which removes most aspects of OS scheduler from the picture
  - zokier 2y ago
    > I've not yet seen a study that shows that virtual threads offer a huge benefit. Not exactly Java virtual threads, but a study on how userland threads beat kernel threads. https://cs.uwaterloo.ca/~mkarsten/papers/sigmetrics2020.html https://cs.uwaterloo.ca/~mkarsten/papers/sigmetrics2020.html For quick results, check figures 11 and 15 from the (preprint) paper. Userland threads ("fred") have ~50% higher throughput while having orders of magnitude better latency at high load levels, in a real-world application (memcached).
dikei 2y ago
> The problem with the naïve design is that even with virtual threads, you risk running out of (heap) memory if the threads ever block. The key with virtual threads is they are so light weight that you can have thousands of them running concurrently: even when they block for I/O, it doesn't matter. It's similar to light weight coroutine in other language like Go or Kotlin.
imtringued 2y ago
What you are complaining about has nothing to do with thread pools or virtual threads. You're pointing out the fact that more parallelism will also need more hardware and that a finite hardware budget will need a back pressure strategy to keep resource consumption within a limit. While you might be correct that "sizing a traditional thread pool" is a back pressure strategy that can be applied to virtual threads, the problem with it is that IO bound threads will prevent CPU bound threads from making progress. You don't want to apply back pressure based on the number of tasks. You want back pressure to be in response to resource utilization, so that enough tasks get scheduled to max out the hardware. This is a common problem with people using Java parallel streams, because they by default share a single global thread pool and the way to use your own thread pool is also extremely counterintuitive, because it essentially relies on some implicit thread local magic to choose to distribute the stream in the thread pool that the parallel stream was launched on, instead of passing it as a parameter. It would be best if people came up with more dynamic back pressure strategies, because this is a more general problem that goes way beyond thread pools. In fact, one of the key problems of automatic parallelization is deciding at what point there is too much parallelization.

immibis 2y ago

Async does exactly the same by the way.

initplus 2y ago

The benefits from virtual threads come from the simple API that it presents to the programmer. It's not a performance optimization.

hashmash 2y ago
But that same benefit was always available with platform threads -- a simple API. What is the real gain by using virtual threads? It's either going to be performance or memory utilization.
- groestl 2y ago
  It's combining the benefits from async models (state machines separated from os threads, thus more optimal for I/O bound workload), with the benefits from proper threading models (namely the simpler human interface). Memory utilization & performance is going to be similar to the async callback mess.
  - hashmash 2y ago
    Why is an async model better than using OS threads for an I/O bound workload? The OS is doing async stuff internally and shielding the complexity with threads. With virtual threads this work has shifted to the JVM. Can the JVM do threads better than the OS?
    - adgjlsfhk1 2y ago
      it can do a much better job because there isn't a security boundary. OS thread scheduling requires sys calls and invalidate a bunch of cache to prevent timing leaks
    - zokier 2y ago
      > Can the JVM do threads better than the OS? Yes. The JVM has far more opportunities for optimizing threads because it doesn't need to uphold 50 years of accumulated invariants and compatibility that current OSes do, and JVM has more visibilty on the application internals.
    - mrsilencedogood 2y ago
      "Why is an async model better than using OS threads for an I/O bound workload?" Because evented/callback-driven code is a nightmare to reason about and breaks lots of very basic tools, like the humble stack trace. Another big thing for me is resource management - try/finally don't work across callback boundaries, but do work within a virtual thread. I recently ported a netty-based evented system to virtual threads and a very long-standing issue - resource leakage - turned into one very nice try/finally block.
      - deleted 2y ago
        [deleted]
- lichtenberger 2y ago
  Throughput. The code can be "suspended" on a blocking call (I/O, where the platform thread usually is wasted, as the CPU has nothing to do during this time). So, the platform thread can do other work in the meantime.
- CrimsonRain 2y ago
  Create 100k platform threads and you'll find out.

packetlost 2y ago

Yeah, and it's generally good to be RAM limited instead of CPU, no? The alternative is blowing a bunch of time on syscalls and OS scheduler overhead. Also the virtual threads run on a "traditional" thread pool to my understanding, so you can just tweak the number of worker threads to cap the total concurrency. The benefit is it's overall more efficient (in the general case) and lets you write linear blocking code (as opposed to function coloring). You don't have to use it, but it's nice that it's there. Now hopefully Valhalla actually makes it in eventually

hashmash 2y ago
The OS scheduler is still there (for the carrier threads), but now you've added on top of that FJ pool based scheduler overhead. Although virtual threads don't have the syscall overhead when they block, there's a new cost caused by allocating the internal continuation object, and copying state into it. This puts more pressure on the garbage collector. Context switching cost due to CPU cache thrashing doesn't go away regardless of which type of thread you're using. I've not yet seen a study that shows that virtual threads offer a huge benefit. The Open Liberty study suggests that they're worse than the existing platform threads.
- zokier 2y ago
  > The OS scheduler is still there (for the carrier threads), but now you've added on top of that FJ pool based scheduler overhead. Ideally carrier threads would be pinned to isolated cpu cores, which removes most aspects of OS scheduler from the picture
- zokier 2y ago
  > I've not yet seen a study that shows that virtual threads offer a huge benefit. Not exactly Java virtual threads, but a study on how userland threads beat kernel threads. https://cs.uwaterloo.ca/~mkarsten/papers/sigmetrics2020.html https://cs.uwaterloo.ca/~mkarsten/papers/sigmetrics2020.html For quick results, check figures 11 and 15 from the (preprint) paper. Userland threads ("fred") have ~50% higher throughput while having orders of magnitude better latency at high load levels, in a real-world application (memcached).

dikei 2y ago

> The problem with the naïve design is that even with virtual threads, you risk running out of (heap) memory if the threads ever block. The key with virtual threads is they are so light weight that you can have thousands of them running concurrently: even when they block for I/O, it doesn't matter. It's similar to light weight coroutine in other language like Go or Kotlin.

imtringued 2y ago

What you are complaining about has nothing to do with thread pools or virtual threads. You're pointing out the fact that more parallelism will also need more hardware and that a finite hardware budget will need a back pressure strategy to keep resource consumption within a limit. While you might be correct that "sizing a traditional thread pool" is a back pressure strategy that can be applied to virtual threads, the problem with it is that IO bound threads will prevent CPU bound threads from making progress. You don't want to apply back pressure based on the number of tasks. You want back pressure to be in response to resource utilization, so that enough tasks get scheduled to max out the hardware. This is a common problem with people using Java parallel streams, because they by default share a single global thread pool and the way to use your own thread pool is also extremely counterintuitive, because it essentially relies on some implicit thread local magic to choose to distribute the stream in the thread pool that the parallel stream was launched on, instead of passing it as a parameter. It would be best if people came up with more dynamic back pressure strategies, because this is a more general problem that goes way beyond thread pools. In fact, one of the key problems of automatic parallelization is deciding at what point there is too much parallelization.