15 ms·

Virtual Threads: New Foundations for High-Scale Java Applications

213 points by axelfontaine 4y ago

samsquire 4y ago
This is good. I implemented a userspace 1:M:N timeslicing thread, kernel thread to lightweight thread multiplexer in Java, Rust and C. I preempt hot for and while loops by setting the looping variable to the limit from the kernel multiplexing thread. It means threads cannot have resource starvation. https://github.com/samsquire/preemptible-thread https://github.com/samsquire/preemptible-thread The design is simple. But having native support as in Loom is really useful.
- mattgreenrocks 4y ago
  I like it! Do you have any sense for what the perf hit is for making those loops less hot to enable pre-emption?
  - samsquire 4y ago
    There is no if statement in the hot loop or in the kernel thread so there is no performance cost there. The multiplexing thread is separate to the kernel thread so you could say it's 1:M:N thread scheduling. I should have been clearer in my comment. There is 3 types of threads. The multiplexing thread timeslices the preemption of the lightweight threads and kernel threads every 10 milliseconds. That is it stops all the loops in the lightweight thread and in the lightweight thread and causes the next lightweight thread to execute. So there is no overhead except for a structure variable retrieval in the loop body Rather than For(int I = 0; I < 1000000; I++) { } We have Register_loop(thread_loops, 0, 0, 1000000); For(; thread_loops[0].index < thread_loops[0].limit; thread_loops[0].index++) { } Handle_virtual_interrupt() And in the thread multiplexer scheduler, we do this thread_loops[0].index = thread_loops[0].limit
    - ReactiveJelly 4y ago
      But that means `thread_loops [0].index` must be read / written with atomic ops, right? I saw that's how wasmtime solves pre-emption of wasm threads with their "epoch-based interruption". [1] I see your Rust implementation does indeed use atomic ops. Have you _measured_ that there is no overhead to atomics? [1] https://docs.rs/wasmtime/1.0.1/wasmtime/struct.Config.html#method.epoch_interruption https://docs.rs/wasmtime/1.0.1/wasmtime/struct.Config.html#m... (Also it says that malicious guest code cannot skip the check, which is nice)
      - samsquire 4y ago
        I didn't use atomic ops in Java or C. It still works. The risk is that a thread fails to reschedule/be preempted/end the loop in this timeslice but if the exact same interleaving happens to every other timeslice after, it shall never preempt. But this is unlikely in practice. The chance of an incrementing at the end of a loop data race with setting the index to the limit and a preventing preemption a unlikely but possible. I assume the body of the loop has the most execution time, not the loop conditional and increment. I've not measured the overhead of the atomics. In theory the atomics are on the slow path of the loop - they happen during preemption and at the end of a hot loop iteration The loop itself is the hot part that we don't want to slow down The worst we do is slow down the execution of the next hot loop with atomic set. Atomic reads are cheap. I'm not sure if atomic writes are cheap if uncontended
        mattgreenrocks 4y ago
        The data race on the volatile version of the C code is technically UB, so you are at the mercy of the architecture, optimization settings and phase of the moon. This is a great answer for more detail: https://stackoverflow.com/a/60482370 https://stackoverflow.com/a/60482370
        samsquire 4y ago
        Thanks Matt. I shall need to do more testing. The right answer is probably use atomics but that has a performance cost. I am still trying to find out if atomics suffer from contendedness. My perfect scenario is the atomic increment should be as cheap as the non atomic increment. But I think the bus pausing and cache coherence protocols mean that the data is flushed from the store buffer to the memory, which is slow. I don't know if it acts as a lock and has an uncontended option
ccooffee 4y ago
This is a great writeup, and reignites my interest in Java. (I've long considered "Java Concurrency in Practice" to be the _best_ Java book ever written.) I haven't been able to figure out how the "unmount" of a virtual thread works. As stated in this article: > Nearly all blocking points in the JDK have been adapted so that when encountering a blocking operation on a virtual thread, the virtual thread is unmounted from its carrier instead of blocking. How would I implement this logic in my own libraries? The underlying JEP 425[0] doesn't seem to list any explicit APIs for that, but it does give other details not in the OP writeup. [0] https://openjdk.org/jeps/425 https://openjdk.org/jeps/425
- rlawson 4y ago
  Agree - Java Concurrency in Practice was a revelation when it came out. Well the whole concurrent api by Dr. Lea made it all so much saner. Very excited for virtual threads!
- geodel 4y ago
  Well one way is to replace "synchronized" blocks with ReentrantLocks where ever you can.
  - cvoss 4y ago
    I would guess that LockSupport.park() and friends have also been adapted to support virtual thread unmounting.
- ecshafer 4y ago
  Java Concurrency in Practice is a fantastic book. I had DL as a professor for about a half dozen courses in undergrad, including Concurrent and Parallel Programming. Absolutely fantastic professor, with a lot of insight into how parallel programming really works at the language level. One of the best courses I've taken.
  - anonymousDan 4y ago
    Yeah Java gets a lot of grief, but I learned a lot about concurrent programming from making sure I really understood every line of code in this book.
geodel 4y ago
I think it is really important development in Java space. One reason I plan to use it soon is because it does not bring in complex programing model of "reactive world" and hence dependency on tons of reactive libraries. I tried moving plain old Tomcat based service to scalable netty based reactive stack but it turned out to be too much work and an alien programing model. With Loom/Virtual thread, the only thing I will be looking for server supporting Virtual threads natively. Helidon Nima would fit the bill here as all other frameworks/app servers have so far just slapping virtual threads on their thread pool based system. And unsurprisingly it is not leading to great perf expected from Virtual thread based system.
- RcouF1uZ4gsC 4y ago
  > How long until OS vendors introduce abstractions to make this easier? Why aren't there OS-native green threads, or at the very least user-space scheduling affordances for runtimes that want to implement them without overhead in calling blocking code? Windows had has Fibers[0] for decades (IIRC since 1996 with Windows NT 4.0) 0. https://learn.microsoft.com/en-us/windows/win32/procthread/fibers https://learn.microsoft.com/en-us/windows/win32/procthread/f...
  - gpderetta 4y ago
    Fibers are just one part of it. You also need a scheduler and a way to hook into existing blocking APIs.
jeffbee 4y ago
" Operating systems typically allocate thread stacks as monolithic blocks of memory at thread creation time that cannot be resized later. This means that threads carry with them megabyte-scale chunks of memory to manage the native and Java call stacks." This extremely common misconception is not true of Linux or Windows. Both Windows and Linux have demand-paged thread stacks whose real size ("committed memory" in Windows) is minimal initially and grows when needed.
- uluyol 4y ago
  Do they shrink too? How many threads can be created before address space is exhausted (even if the memory isn't backed by pages, the address space is still reserved)?
  - tedunangst 4y ago
    The stack for any thread other than the first is just memory like any other allocation. You can free it, resize it, copy it elsewhere, whatever you want to do. Literally just a pointer in a register. People work up weird mythologies about it, but the stack can be anything you want if you're willing to write code to manage it.
  - jeffbee 4y ago
    You'll run out of physical memory for the first page of the stack long before you run out of room in the virtual address space.
mikece 4y ago
How does this compare to Processes in Elixir/Erlang -- is Java now as lightweight and performant?
- stefs 4y ago
  according to the article: > Virtual threads have more in common with the user-mode threads found in other languages, such as goroutines in Go or processes in Erlang -- but have the advantage of being semantically identical to the threads we already have.
  - anonymousDan 4y ago
    And the disadvantages of global shared memory, stop the world pauses etc. For sure it is a step in the right direction but it depends on where you stand wrt having shared mutable state in your programming model. It's also not clear to me whether virtual threads can lock up carrier threads (e.g. due to an infinite loop) or are somehow preemptible (c.f. erlang reduction counts).
    - kaba0 4y ago
      They can lock up carrier threads and are not preemptible. It is a bit subjective, but regarding threading Java often chooses to expose the basic primitives as is, and let you build on top. Erlang is an opinionated specialization of concurrent programming en large, which may be a better for for certain problems, but not for others. Also, I don’t think global shared memory is a problem, one is free to use ExtentLocals (threadlocals, but virtual thread friendly) in Java. Also, while small actor-scoped heaps are cool, I don’t think that they have anything on the state-of-the-art GCs Java employ, and also stop the world pauses are not really a problem that comes up, and if they do it is likely some programmer error.
    - pron 4y ago
      Java hasn't had significant stop-the-world pauses for a few years now. ZGC, the new low-latency collector, has <1ms pauses for heaps up to at least 4 TB. As to shared mutable state, virtual threads are completely orthogonal to that. You can write code with shared state or without, or, if you want language guarantees, you can use languages that greatly limit it, like Clojure. I would love to see a new implementation of Erlang for the Java platform.
thom 4y ago
So right now it seems like you can replace the thread pool Clojure uses for futures etc with virtual threads and go ham. You could even write an alternative go macro to replace the bits of core.async where you’re not supposed to block. Feels like Clojure could be poised to benefit the most here, and what a delight it is to have such a language on a modern runtime that still gets shiny new features!
- shaunxcode 4y ago
  Yep core async will work as is just use <!! and friends.
- KingMob 4y ago
  You wouldn’t need the go macro for anything with vthreads. If you continued to use core.async, you’d switch to the blocking variants, and ignore go entirely.
mgraczyk 4y ago
The section "What about async/await?", which compares these virtual threads to async/await is very weak. After reading this article, I came away with the impression that this is a dramatically worse way to solve this problem than async/await. The only benefit I see is that this will be simpler to use for the (increasingly rare) programmers who are not used to async programming. The first objection in the article is that with async/await you to may forget to use an async operation and could instead use a synchronous operation. This is not a real problem. Languages like JavaScript do not have any synchronous operations so you can't use them by mistake. Languages like python and C# solve this with simple lint rules that tell you if you make this mistake. The second objection is that you have to reimplement all library functions to support await. This is a bad objection because you also have to do this for virtual threads. Based on how long it took to add virtual threada to Java vs adding async/await to other languages, it seems like virtual threads were much more complicated to implement. The programming model here sounds analogous to using gevent with python vs python async/await. My opinion is that the gevent approach will die out completely as async/await becomes better supported and programmers become more familiar. EDIT: Looking more at the "Related Work" section at the bottom. I think I understand the problem here. The "Structured Concurrency" examples are unergonomical versions of async/await. I'm not sure what I'm missing but this seems like a strictly worse way to write structured concurrent code. Java example: Response handle() throws ExecutionException, InterruptedException { try (var scope = new StructuredTaskScope.ShutdownOnFailure()) { Future<String> user = scope.fork(() -> findUser()); Future<Integer> order = scope.fork(() -> fetchOrder()); scope.join(); // Join both forks scope.throwIfFailed(); // ... and propagate errors // Here, both forks have succeeded, so compose their results return new Response(user.resultNow(), order.resultNow()); } } Python equivalent async def handle() -> Response: # scope is implicit, throwing on failure is implicit. user, order = await asyncio.gather(findUser(), findOrder()) return Response(user, order) You could probably implement a similar abstraction in Java, but you would need to pass around and manage the the scope object, which seems cumbersome.
- wtetzner 4y ago
  I can see you having objections to their arguments against async/await, but what makes you say async/await is somehow the better solution?
gigatexal 4y ago
Reading through the source code examples has me rethinking my dislike for Java. It sure seems far less verbose and kinda nice actually.
- marginalia_nu 4y ago
  Modern Java is a lot less boilerplaty than old enterprise Java.
  - dboreham 4y ago
    Modern Java people are a lot less boilerplaty than old enterprise Java people.
    - gigatexal 4y ago
      What places are doing modern java development?
      - redditor98654 4y ago
        AWS, Netflix to name a few I know.
    - waynesonfire 4y ago
      And which modern web framework is least boilerplate? Springboot?
      - marginalia_nu 4y ago
        I'm a big fan of Spark. https://sparkjava.com/ https://sparkjava.com/
        ihateolives 4y ago
        Why Spark over Javalin? I switched to Javalin last year, it seems to have more active development.
Blackthorn 4y ago
So happy this is finally coming out! After years of using the library that inspired this (fibers), I'm so stoked this is coming to the wide outside world of Java. There's just no comparison in how understandable and easy to program and debug this is compared to callback and event based programming.
smasher164 4y ago
After all the hoopla surrounding concurrency models, it seems that languages are conceding that green threads are more ergonomic to work with. Go and Java have it, and now .NET is even experimenting with it. How long until OS vendors introduce abstractions to make this easier? Why aren't there OS-native green threads, or at the very least user-space scheduling affordances for runtimes that want to implement them without overhead in calling blocking code?
- halayli 4y ago
  bsd variants and sun os used to have n:m APIs but they got deprecated and removed long time ago. It's hard to provide a consistent behavior that works for everyone.
  - smasher164 4y ago
    I said in a sibling comment that scheduler activations may have been a flawed idea, but I don’t think the space of user-space scheduling APIs is fully explored. Google’s ghOSt work is an example of that. If io-uring is proof of anything, it’s that there’s still fundamental changes we can make in how we schedule work with the kernel.
- Jtsummers 4y ago
  > Why aren't there OS-native green threads, or at the very least user-space scheduling affordances for runtimes that want to implement them without overhead in calling blocking code? Green threads are, definitionally, not OS threads, they are user space threads. So you will never see OS-native green threads as it's an oxymoron. The way many green thread systems work is to either lie to you (you really only have one OS thread, the green threads exist to write concurrent code, which can be much simpler, but not parallel code using Pike's distinction), or to introduce multiple OS threads ("carrier threads" in the terms of this article) which green threads are distributed across (this is what Java is doing here, Go has done for a long time, BEAM languages for a long time, and many others). EDIT: To extend this, many people think of "green threads" as lightweight threading mechanisms. That's kind of accurate for many systems, but not always true. If that's the sense that's meant, then OS-native lightweight threads are certainly possible in the future. But there's probably not much reason to add them when user space lightweight concurrency mechanisms already exist, and there's no consensus on which ones are "best" (by whatever metric).
lenkite 4y ago
Really hope this makes it to Android. (probably need to wait for a decade or two though)
anonymousDan 4y ago
Copying virtual stacks on a context switch sounds kind of expensive. Any performance numbers available? Maybe for very deep stacks there are optimizations whereby you only copy in deeper frames lazily under the assumption they won't be used yet? Also, what is the story with preemption - if a virtual thread spins in an infinite loop, will it effectively hog the carrier thread or can it be descheduled? Finally, I would be really interested to see the impact on debugability. I did some related work where we were trying to get the JVM to run on top of a library operating system and a libc that contained a user level threading library. Debugging anything concurrency related became a complete nightmare since all the gdb tooling only really understood the underlying carrier threads. Having said all that, this sounds super cool and I think is 100% the way to go for Java. Would be interesting to revisit the implementation of something like Akka in light of this.
- aardvark179 4y ago
  Yes, lazy copying tricks are employed, and some work round stack frames delayed until the stack is moved by the GC on the assumption most will not live that long. There was a lot of work done on debugging so standard Java debuggers work well.
  - anonymousDan 4y ago
    That's pretty cool! What about the blocking issue? Presumably also if you are using JNI all bets are off?
    - aardvark179 4y ago
      If you are using JNI then that puts a native frame on the stack and pins the virtual thread. Loom’s strategy works because we know what can be on the Java stack, and that nothing external points into it, but we don’t know that for native frames.
- hawk_ 4y ago
  Stacks in idiomatic Java aren't usually that deep (pointers instead of in place values) so this isn't that big of a deal, unlike say in C++.
jsyolo 4y ago
Keeping familiarity aside, why would one use Java/JVM instead of nodejs for the server of a web app? I need to call SOAP services.
- anonymoushn 4y ago
  What are the benefits of using nodejs over Java?
  - jsyolo 4y ago
    SSR React, using the same language in the server as in the browser and not wrangling between to completely different ecosystems.
    - lmm 4y ago
      I get that the other way around by using Scala.js in the browser; I find it a much nicer language to work with than Javascript or even Typescript.
    - seinecle 4y ago
      Java has a solid frontend framework. I discussed it recently in relation to SSR React: https://nocodefunctions.com/blog/java-frontend-web-app/ https://nocodefunctions.com/blog/java-frontend-web-app/
- pjmlp 4y ago
  When one cares about performance.
  - kaba0 4y ago
    And maintainability. And quality of libraries.
stefs 4y ago
for the record, i really don't know much about threads, so the following questions are probably kinda stupid. first question: so, as the article states, the ONLY performance upside of virtual threads (versus os threads) is the number of inactive threads, thanks due to lower per-thread memory overhead. for some reason i was expecting to read something about context switching cost too. as far as i understand, virtual thread context switches are most likely between a lot cheaper and roughly as expensive than their carrier thread context switches, depending on how much memory has to be copied around and how to find the next thread to execute. the problem here is that virtual context switches may be cheaper, but have to be executed in addition to the os thread context switches, so the overall efficiency is actually lower because more work is spent scheduling (os vs. os+virtual). to minimize this it might be possible for privileged applications to disable os thread context switching for the carrier threads as long as there are active virtual threads. that way, the context switching and scheduling overhead is reduced from "os vs. os+virt" to "os vs. virt". i.e. as soon as there are active virtual threads the carrier thread is excluded for os scheduler until there aren't any active virtual threads anymore (or, alternatively, the virtual thread pool is empty). is this a thing? does this make sense? would it be worth it? do operating systems even support "manual" (i.e. by the app) thread scheduling hints? or are the carrier threads only rarely taken out of schedule because they're not really put to sleep as long as there are active virtual threads anyway, making this a non-issue? second question: as far as i understand blocking os threads, the scheduler stores which thread is waiting on which io resource and the appropriate thread gets woken up once a waited-on io resouce is available. this is not much of a problem with with a few hundred or thousand os threads, but now with virtual threads, the io resource must now be linked to the os thread for the virtual thread executor's scheduler by the os and then to the virtual thread waiting on the resource by the virtual thread scheduler. so for example if there are 100.000 inactive virtual threads waiting for a network response and one arrives, the os scheduler has to match it to an os thread first (the one the vt scheduler runs on) and then the vt scheduler has to match it to one of the virtual threads. i.e. two lookups in hashtables with 100.000 entries each (one io to os threads, the other io to vt). is this how it works or do i misunderstand this? as async models have the same issue but work fine i guess this isn't really a problem in practice. also, as far as i understand, the os thread woken up is given a kind of resouce id it's been woken up for, instead of "well, you went to sleep for a certain resource id so it's obvious which one you've been woken up for" in blocking IO).
rr808 4y ago
I spent the last 5 years learning reactive programming. I hate it. I'm looking forward to going back to something solid.
- s-macke 4y ago
  Java Reactive works around the shortcomings of the platform by basically inventing its own language inside Java. A high cost for the developer. Under the hood, the virtual thread feature does more or less the same as reactive. Async calls via epoll, a scheduler in user space, .....
bheadmaster 4y ago
It's funny how it took Java almost a decade to finally implement goroutines.
- kaba0 4y ago
  As opposed to what platform? Also, they did that in a completely backwards compatible way, so that programs that were written before Go was even a thing will also benefit, so if you will, Java has out of the doors orders of more “goroutine”-aware code as is, than Go will likely ever will.
  - bheadmaster 4y ago
    > As opposed to what platform? Go. > programs that were written before Go was even a thing will also benefit As far as I understand virtual threads, they have a new API for creating them, so older programs will at the very least need some source change and a recompilation in order to use them. Of course, if there's a JVM flag to automatically use virtual threads in place of OS threads, please correct me :) > Java has out of the doors orders of more “goroutine”-aware code as is, than Go will likely ever will. I honestly doubt that older code is "goroutine"-aware, as virtual threads are used in a different way than ordinary OS threads. I don't think there's any code that spawned hundreds of thousands of OS threads before, which would be the main benefit of using virtual threads.
- pjmlp 4y ago
  You mean the co-routines that Modula-2 already had in 1978, or Erlang in 1986, not such big achievement on Go's part. By the way, JVM green threads were available in 1996, however eventually red threads became the default implementation across all major JVM vendors.
  - bheadmaster 4y ago
    > You mean the co-routines that Modula-2 already had in 1978, or Erlang in 1986, not such big achievement on Go's part. What are you implying, that Go should've implemented goroutines before Go even existed?
polskibus 4y ago
Is this like async/await machinery in .NET?
- sebazzz 4y ago
  Yes but transparent

213 points by axelfontaine 4y ago

This is good. I implemented a userspace 1:M:N timeslicing thread, kernel thread to lightweight thread multiplexer in Java, Rust and C. I preempt hot for and while loops by setting the looping variable to the limit from the kernel multiplexing thread. It means threads cannot have resource starvation. https://github.com/samsquire/preemptible-thread https://github.com/samsquire/preemptible-thread The design is simple. But having native support as in Loom is really useful.

mattgreenrocks 4y ago
I like it! Do you have any sense for what the perf hit is for making those loops less hot to enable pre-emption?
- samsquire 4y ago
  There is no if statement in the hot loop or in the kernel thread so there is no performance cost there. The multiplexing thread is separate to the kernel thread so you could say it's 1:M:N thread scheduling. I should have been clearer in my comment. There is 3 types of threads. The multiplexing thread timeslices the preemption of the lightweight threads and kernel threads every 10 milliseconds. That is it stops all the loops in the lightweight thread and in the lightweight thread and causes the next lightweight thread to execute. So there is no overhead except for a structure variable retrieval in the loop body Rather than For(int I = 0; I < 1000000; I++) { } We have Register_loop(thread_loops, 0, 0, 1000000); For(; thread_loops[0].index < thread_loops[0].limit; thread_loops[0].index++) { } Handle_virtual_interrupt() And in the thread multiplexer scheduler, we do this thread_loops[0].index = thread_loops[0].limit
  - ReactiveJelly 4y ago
    But that means `thread_loops [0].index` must be read / written with atomic ops, right? I saw that's how wasmtime solves pre-emption of wasm threads with their "epoch-based interruption". [1] I see your Rust implementation does indeed use atomic ops. Have you _measured_ that there is no overhead to atomics? [1] https://docs.rs/wasmtime/1.0.1/wasmtime/struct.Config.html#method.epoch_interruption https://docs.rs/wasmtime/1.0.1/wasmtime/struct.Config.html#m... (Also it says that malicious guest code cannot skip the check, which is nice)
    - samsquire 4y ago
      I didn't use atomic ops in Java or C. It still works. The risk is that a thread fails to reschedule/be preempted/end the loop in this timeslice but if the exact same interleaving happens to every other timeslice after, it shall never preempt. But this is unlikely in practice. The chance of an incrementing at the end of a loop data race with setting the index to the limit and a preventing preemption a unlikely but possible. I assume the body of the loop has the most execution time, not the loop conditional and increment. I've not measured the overhead of the atomics. In theory the atomics are on the slow path of the loop - they happen during preemption and at the end of a hot loop iteration The loop itself is the hot part that we don't want to slow down The worst we do is slow down the execution of the next hot loop with atomic set. Atomic reads are cheap. I'm not sure if atomic writes are cheap if uncontended
      - mattgreenrocks 4y ago
        The data race on the volatile version of the C code is technically UB, so you are at the mercy of the architecture, optimization settings and phase of the moon. This is a great answer for more detail: https://stackoverflow.com/a/60482370 https://stackoverflow.com/a/60482370
        samsquire 4y ago
        Thanks Matt. I shall need to do more testing. The right answer is probably use atomics but that has a performance cost. I am still trying to find out if atomics suffer from contendedness. My perfect scenario is the atomic increment should be as cheap as the non atomic increment. But I think the bus pausing and cache coherence protocols mean that the data is flushed from the store buffer to the memory, which is slow. I don't know if it acts as a lock and has an uncontended option

ccooffee 4y ago

This is a great writeup, and reignites my interest in Java. (I've long considered "Java Concurrency in Practice" to be the _best_ Java book ever written.) I haven't been able to figure out how the "unmount" of a virtual thread works. As stated in this article: > Nearly all blocking points in the JDK have been adapted so that when encountering a blocking operation on a virtual thread, the virtual thread is unmounted from its carrier instead of blocking. How would I implement this logic in my own libraries? The underlying JEP 425[0] doesn't seem to list any explicit APIs for that, but it does give other details not in the OP writeup. [0] https://openjdk.org/jeps/425 https://openjdk.org/jeps/425

rlawson 4y ago
Agree - Java Concurrency in Practice was a revelation when it came out. Well the whole concurrent api by Dr. Lea made it all so much saner. Very excited for virtual threads!
geodel 4y ago
Well one way is to replace "synchronized" blocks with ReentrantLocks where ever you can.
- cvoss 4y ago
  I would guess that LockSupport.park() and friends have also been adapted to support virtual thread unmounting.
ecshafer 4y ago
Java Concurrency in Practice is a fantastic book. I had DL as a professor for about a half dozen courses in undergrad, including Concurrent and Parallel Programming. Absolutely fantastic professor, with a lot of insight into how parallel programming really works at the language level. One of the best courses I've taken.
- anonymousDan 4y ago
  Yeah Java gets a lot of grief, but I learned a lot about concurrent programming from making sure I really understood every line of code in this book.

geodel 4y ago

I think it is really important development in Java space. One reason I plan to use it soon is because it does not bring in complex programing model of "reactive world" and hence dependency on tons of reactive libraries. I tried moving plain old Tomcat based service to scalable netty based reactive stack but it turned out to be too much work and an alien programing model. With Loom/Virtual thread, the only thing I will be looking for server supporting Virtual threads natively. Helidon Nima would fit the bill here as all other frameworks/app servers have so far just slapping virtual threads on their thread pool based system. And unsurprisingly it is not leading to great perf expected from Virtual thread based system.

RcouF1uZ4gsC 4y ago
> How long until OS vendors introduce abstractions to make this easier? Why aren't there OS-native green threads, or at the very least user-space scheduling affordances for runtimes that want to implement them without overhead in calling blocking code? Windows had has Fibers[0] for decades (IIRC since 1996 with Windows NT 4.0) 0. https://learn.microsoft.com/en-us/windows/win32/procthread/fibers https://learn.microsoft.com/en-us/windows/win32/procthread/f...
- gpderetta 4y ago
  Fibers are just one part of it. You also need a scheduler and a way to hook into existing blocking APIs.

jeffbee 4y ago

" Operating systems typically allocate thread stacks as monolithic blocks of memory at thread creation time that cannot be resized later. This means that threads carry with them megabyte-scale chunks of memory to manage the native and Java call stacks." This extremely common misconception is not true of Linux or Windows. Both Windows and Linux have demand-paged thread stacks whose real size ("committed memory" in Windows) is minimal initially and grows when needed.

uluyol 4y ago
Do they shrink too? How many threads can be created before address space is exhausted (even if the memory isn't backed by pages, the address space is still reserved)?
- tedunangst 4y ago
  The stack for any thread other than the first is just memory like any other allocation. You can free it, resize it, copy it elsewhere, whatever you want to do. Literally just a pointer in a register. People work up weird mythologies about it, but the stack can be anything you want if you're willing to write code to manage it.
- jeffbee 4y ago
  You'll run out of physical memory for the first page of the stack long before you run out of room in the virtual address space.

mikece 4y ago

How does this compare to Processes in Elixir/Erlang -- is Java now as lightweight and performant?

stefs 4y ago
according to the article: > Virtual threads have more in common with the user-mode threads found in other languages, such as goroutines in Go or processes in Erlang -- but have the advantage of being semantically identical to the threads we already have.
- anonymousDan 4y ago
  And the disadvantages of global shared memory, stop the world pauses etc. For sure it is a step in the right direction but it depends on where you stand wrt having shared mutable state in your programming model. It's also not clear to me whether virtual threads can lock up carrier threads (e.g. due to an infinite loop) or are somehow preemptible (c.f. erlang reduction counts).
  - kaba0 4y ago
    They can lock up carrier threads and are not preemptible. It is a bit subjective, but regarding threading Java often chooses to expose the basic primitives as is, and let you build on top. Erlang is an opinionated specialization of concurrent programming en large, which may be a better for for certain problems, but not for others. Also, I don’t think global shared memory is a problem, one is free to use ExtentLocals (threadlocals, but virtual thread friendly) in Java. Also, while small actor-scoped heaps are cool, I don’t think that they have anything on the state-of-the-art GCs Java employ, and also stop the world pauses are not really a problem that comes up, and if they do it is likely some programmer error.
  - pron 4y ago
    Java hasn't had significant stop-the-world pauses for a few years now. ZGC, the new low-latency collector, has <1ms pauses for heaps up to at least 4 TB. As to shared mutable state, virtual threads are completely orthogonal to that. You can write code with shared state or without, or, if you want language guarantees, you can use languages that greatly limit it, like Clojure. I would love to see a new implementation of Erlang for the Java platform.

thom 4y ago

So right now it seems like you can replace the thread pool Clojure uses for futures etc with virtual threads and go ham. You could even write an alternative go macro to replace the bits of core.async where you’re not supposed to block. Feels like Clojure could be poised to benefit the most here, and what a delight it is to have such a language on a modern runtime that still gets shiny new features!

shaunxcode 4y ago
Yep core async will work as is just use <!! and friends.
KingMob 4y ago
You wouldn’t need the go macro for anything with vthreads. If you continued to use core.async, you’d switch to the blocking variants, and ignore go entirely.

mgraczyk 4y ago

The section "What about async/await?", which compares these virtual threads to async/await is very weak. After reading this article, I came away with the impression that this is a dramatically worse way to solve this problem than async/await. The only benefit I see is that this will be simpler to use for the (increasingly rare) programmers who are not used to async programming. The first objection in the article is that with async/await you to may forget to use an async operation and could instead use a synchronous operation. This is not a real problem. Languages like JavaScript do not have any synchronous operations so you can't use them by mistake. Languages like python and C# solve this with simple lint rules that tell you if you make this mistake. The second objection is that you have to reimplement all library functions to support await. This is a bad objection because you also have to do this for virtual threads. Based on how long it took to add virtual threada to Java vs adding async/await to other languages, it seems like virtual threads were much more complicated to implement. The programming model here sounds analogous to using gevent with python vs python async/await. My opinion is that the gevent approach will die out completely as async/await becomes better supported and programmers become more familiar. EDIT: Looking more at the "Related Work" section at the bottom. I think I understand the problem here. The "Structured Concurrency" examples are unergonomical versions of async/await. I'm not sure what I'm missing but this seems like a strictly worse way to write structured concurrent code. Java example: Response handle() throws ExecutionException, InterruptedException { try (var scope = new StructuredTaskScope.ShutdownOnFailure()) { Future<String> user = scope.fork(() -> findUser()); Future<Integer> order = scope.fork(() -> fetchOrder()); scope.join(); // Join both forks scope.throwIfFailed(); // ... and propagate errors // Here, both forks have succeeded, so compose their results return new Response(user.resultNow(), order.resultNow()); } } Python equivalent async def handle() -> Response: # scope is implicit, throwing on failure is implicit. user, order = await asyncio.gather(findUser(), findOrder()) return Response(user, order) You could probably implement a similar abstraction in Java, but you would need to pass around and manage the the scope object, which seems cumbersome.

wtetzner 4y ago
I can see you having objections to their arguments against async/await, but what makes you say async/await is somehow the better solution?

gigatexal 4y ago

Reading through the source code examples has me rethinking my dislike for Java. It sure seems far less verbose and kinda nice actually.

marginalia_nu 4y ago
Modern Java is a lot less boilerplaty than old enterprise Java.
- dboreham 4y ago
  Modern Java people are a lot less boilerplaty than old enterprise Java people.
  - gigatexal 4y ago
    What places are doing modern java development?
    - redditor98654 4y ago
      AWS, Netflix to name a few I know.
  - waynesonfire 4y ago
    And which modern web framework is least boilerplate? Springboot?
    - marginalia_nu 4y ago
      I'm a big fan of Spark. https://sparkjava.com/ https://sparkjava.com/
      - ihateolives 4y ago
        Why Spark over Javalin? I switched to Javalin last year, it seems to have more active development.

Blackthorn 4y ago

So happy this is finally coming out! After years of using the library that inspired this (fibers), I'm so stoked this is coming to the wide outside world of Java. There's just no comparison in how understandable and easy to program and debug this is compared to callback and event based programming.

smasher164 4y ago

After all the hoopla surrounding concurrency models, it seems that languages are conceding that green threads are more ergonomic to work with. Go and Java have it, and now .NET is even experimenting with it. How long until OS vendors introduce abstractions to make this easier? Why aren't there OS-native green threads, or at the very least user-space scheduling affordances for runtimes that want to implement them without overhead in calling blocking code?

halayli 4y ago
bsd variants and sun os used to have n:m APIs but they got deprecated and removed long time ago. It's hard to provide a consistent behavior that works for everyone.
- smasher164 4y ago
  I said in a sibling comment that scheduler activations may have been a flawed idea, but I don’t think the space of user-space scheduling APIs is fully explored. Google’s ghOSt work is an example of that. If io-uring is proof of anything, it’s that there’s still fundamental changes we can make in how we schedule work with the kernel.
Jtsummers 4y ago
> Why aren't there OS-native green threads, or at the very least user-space scheduling affordances for runtimes that want to implement them without overhead in calling blocking code? Green threads are, definitionally, not OS threads, they are user space threads. So you will never see OS-native green threads as it's an oxymoron. The way many green thread systems work is to either lie to you (you really only have one OS thread, the green threads exist to write concurrent code, which can be much simpler, but not parallel code using Pike's distinction), or to introduce multiple OS threads ("carrier threads" in the terms of this article) which green threads are distributed across (this is what Java is doing here, Go has done for a long time, BEAM languages for a long time, and many others). EDIT: To extend this, many people think of "green threads" as lightweight threading mechanisms. That's kind of accurate for many systems, but not always true. If that's the sense that's meant, then OS-native lightweight threads are certainly possible in the future. But there's probably not much reason to add them when user space lightweight concurrency mechanisms already exist, and there's no consensus on which ones are "best" (by whatever metric).

lenkite 4y ago

Really hope this makes it to Android. (probably need to wait for a decade or two though)

anonymousDan 4y ago

Copying virtual stacks on a context switch sounds kind of expensive. Any performance numbers available? Maybe for very deep stacks there are optimizations whereby you only copy in deeper frames lazily under the assumption they won't be used yet? Also, what is the story with preemption - if a virtual thread spins in an infinite loop, will it effectively hog the carrier thread or can it be descheduled? Finally, I would be really interested to see the impact on debugability. I did some related work where we were trying to get the JVM to run on top of a library operating system and a libc that contained a user level threading library. Debugging anything concurrency related became a complete nightmare since all the gdb tooling only really understood the underlying carrier threads. Having said all that, this sounds super cool and I think is 100% the way to go for Java. Would be interesting to revisit the implementation of something like Akka in light of this.

aardvark179 4y ago
Yes, lazy copying tricks are employed, and some work round stack frames delayed until the stack is moved by the GC on the assumption most will not live that long. There was a lot of work done on debugging so standard Java debuggers work well.
- anonymousDan 4y ago
  That's pretty cool! What about the blocking issue? Presumably also if you are using JNI all bets are off?
  - aardvark179 4y ago
    If you are using JNI then that puts a native frame on the stack and pins the virtual thread. Loom’s strategy works because we know what can be on the Java stack, and that nothing external points into it, but we don’t know that for native frames.
hawk_ 4y ago
Stacks in idiomatic Java aren't usually that deep (pointers instead of in place values) so this isn't that big of a deal, unlike say in C++.

jsyolo 4y ago

Keeping familiarity aside, why would one use Java/JVM instead of nodejs for the server of a web app? I need to call SOAP services.

anonymoushn 4y ago
What are the benefits of using nodejs over Java?
- jsyolo 4y ago
  SSR React, using the same language in the server as in the browser and not wrangling between to completely different ecosystems.
  - lmm 4y ago
    I get that the other way around by using Scala.js in the browser; I find it a much nicer language to work with than Javascript or even Typescript.
  - seinecle 4y ago
    Java has a solid frontend framework. I discussed it recently in relation to SSR React: https://nocodefunctions.com/blog/java-frontend-web-app/ https://nocodefunctions.com/blog/java-frontend-web-app/
pjmlp 4y ago
When one cares about performance.
- kaba0 4y ago
  And maintainability. And quality of libraries.

stefs 4y ago

for the record, i really don't know much about threads, so the following questions are probably kinda stupid. first question: so, as the article states, the ONLY performance upside of virtual threads (versus os threads) is the number of inactive threads, thanks due to lower per-thread memory overhead. for some reason i was expecting to read something about context switching cost too. as far as i understand, virtual thread context switches are most likely between a lot cheaper and roughly as expensive than their carrier thread context switches, depending on how much memory has to be copied around and how to find the next thread to execute. the problem here is that virtual context switches may be cheaper, but have to be executed in addition to the os thread context switches, so the overall efficiency is actually lower because more work is spent scheduling (os vs. os+virtual). to minimize this it might be possible for privileged applications to disable os thread context switching for the carrier threads as long as there are active virtual threads. that way, the context switching and scheduling overhead is reduced from "os vs. os+virt" to "os vs. virt". i.e. as soon as there are active virtual threads the carrier thread is excluded for os scheduler until there aren't any active virtual threads anymore (or, alternatively, the virtual thread pool is empty). is this a thing? does this make sense? would it be worth it? do operating systems even support "manual" (i.e. by the app) thread scheduling hints? or are the carrier threads only rarely taken out of schedule because they're not really put to sleep as long as there are active virtual threads anyway, making this a non-issue? second question: as far as i understand blocking os threads, the scheduler stores which thread is waiting on which io resource and the appropriate thread gets woken up once a waited-on io resouce is available. this is not much of a problem with with a few hundred or thousand os threads, but now with virtual threads, the io resource must now be linked to the os thread for the virtual thread executor's scheduler by the os and then to the virtual thread waiting on the resource by the virtual thread scheduler. so for example if there are 100.000 inactive virtual threads waiting for a network response and one arrives, the os scheduler has to match it to an os thread first (the one the vt scheduler runs on) and then the vt scheduler has to match it to one of the virtual threads. i.e. two lookups in hashtables with 100.000 entries each (one io to os threads, the other io to vt). is this how it works or do i misunderstand this? as async models have the same issue but work fine i guess this isn't really a problem in practice. also, as far as i understand, the os thread woken up is given a kind of resouce id it's been woken up for, instead of "well, you went to sleep for a certain resource id so it's obvious which one you've been woken up for" in blocking IO).

rr808 4y ago

I spent the last 5 years learning reactive programming. I hate it. I'm looking forward to going back to something solid.

s-macke 4y ago
Java Reactive works around the shortcomings of the platform by basically inventing its own language inside Java. A high cost for the developer. Under the hood, the virtual thread feature does more or less the same as reactive. Async calls via epoll, a scheduler in user space, .....

bheadmaster 4y ago

It's funny how it took Java almost a decade to finally implement goroutines.

kaba0 4y ago
As opposed to what platform? Also, they did that in a completely backwards compatible way, so that programs that were written before Go was even a thing will also benefit, so if you will, Java has out of the doors orders of more “goroutine”-aware code as is, than Go will likely ever will.
- bheadmaster 4y ago
  > As opposed to what platform? Go. > programs that were written before Go was even a thing will also benefit As far as I understand virtual threads, they have a new API for creating them, so older programs will at the very least need some source change and a recompilation in order to use them. Of course, if there's a JVM flag to automatically use virtual threads in place of OS threads, please correct me :) > Java has out of the doors orders of more “goroutine”-aware code as is, than Go will likely ever will. I honestly doubt that older code is "goroutine"-aware, as virtual threads are used in a different way than ordinary OS threads. I don't think there's any code that spawned hundreds of thousands of OS threads before, which would be the main benefit of using virtual threads.
pjmlp 4y ago
You mean the co-routines that Modula-2 already had in 1978, or Erlang in 1986, not such big achievement on Go's part. By the way, JVM green threads were available in 1996, however eventually red threads became the default implementation across all major JVM vendors.
- bheadmaster 4y ago
  > You mean the co-routines that Modula-2 already had in 1978, or Erlang in 1986, not such big achievement on Go's part. What are you implying, that Go should've implemented goroutines before Go even existed?

polskibus 4y ago

Is this like async/await machinery in .NET?

sebazzz 4y ago
Yes but transparent