6 ms·

On Java/JVM: Loom and Thread Fairness

98 points by lichtenberger 4y ago

lichtenberger 4y ago
The mentioned twitter discussion is also pretty interesting IMHO :-)
BenoitP 4y ago
Not a new kind of issue, and not a problem IMHO. Some GCs need to reach a safepoint before doing their work; and when a classic Thread is chugging along in a tight loop, that thread is blocking the whole system for a pending GC. One hacky fix is to provide an opportunity for a safepoint (System.out.print every 10k iterations). Other tools like JVM command line options allow for observing safepoint triggering. The same goes for virtual threads: inserting a Thread.sleep(1L) every nth iteration of a tight loop does it. Also there was talk specifying a custom scheduler for the virtual threads. That would not help in inserting switching opportunities, but it would give a way to define what kind of fairness you want.
- test-account 4y ago
  Agree. Let's document this trick and I'm happy with that unfair scheduling inconvenience for CPU heavy operations.
  - deleted 4y ago
    [deleted]
  - topspin 4y ago
    Seconded. Although I think Thread::yield should do what it says on the tin in the case of lightweight threads and it's hard to understand why they chose to not make it so, I don't care much. If the only problem I have to anticipate to get the full benefit of a well behaved lightweight threat scheduling is strategically sprinkling yield points inside those few tight loops that might actually threaten to hog a core then it's a net win.
    - anonymousDan 4y ago
      Having to do it manually is tedious and error prone IMHO. Better to have a reasonably good automated mechanism that you can disable if you wish).
      - topspin 4y ago
        > tedious and error prone IMHO It's neither. The frequency of non-blocking CPU bound sections of most applications is low and so the need to introduce scheduling hints is also infrequent. Aside from an endless, non-blocking loop that never yields omitting possibly beneficial yield points isn't an error; scheduling will not be ideal but the process will ultimately produce the same result.
    - topspin 4y ago
      Thinking about this further, a pragmatic enhancement to Thread::yield might look like this: enum Hint { ALWAYS, FREQUENT, SOMETIMES, INFREQUENT, NEVER } Thread::yield(Hint); The method could be intrinsic to the compiler+runtime to provide runtime optimization (loop unrolling, tuning, etc.) of the 2nd - 4th cases. This would neatly avoid the need to: long n = 0; while (true) { // stuff if (++n > SOME_LIMIT) { n = 0; // yield here } } ... or similar.
- anonymousDan 4y ago
  Seems like safepoints are an ideal way to do this. It sounds similar in ways to how the BEAM runtime for Elixir/Erlang will count reductions and preempts a lightweight process. I guess the interesting question is whether the same safepoint testing mechanism can be reused directly or whether some more complex logic is required on the critical path
- samsquire 4y ago
  Putting an if statement in your loops shall slow down your loops. Instead chunk/partition the loop and run it in chunks.
klabb3 4y ago
I only observe Java from a distance but everything I've read about project loom is amazing. Speaking as someone who has worked intricately on an async runtime in another language, I've more and more started to question the premises around the explicitly async paradigm. Lots of things we were "promised" with async turned out to be just as complicated in the threaded (i.e. blocking) programming models, and now we have two incompatible programming models. I'm now of the opinion that we should improve (instead of replace) our threaded models and do under-the-hood optimizations to sort out the performance issues. If I had a dime for everytime someone made a blocking call from an async function.. And who can blame them? Most blocking functions aren't even documented as such, you just have to know.
- sebazzz 4y ago
  > If I had a dime for everytime someone made a blocking call from an async function.. And who can blame them? Most blocking functions aren't even documented as such, you just have to know. Isn't that exactly what loom aims to resolve?
- lichtenberger 4y ago
  Yes, I think Loom is an amazing step forward and it's brilliant to just change the blocking methods and thus existing code might already benefit with minor changes.
- derriz 4y ago
  Yeah, I've used a bunch of non-blocking paradigms over the years and all become painful quickly once the system becomes non-trivial, except for green threads. I was very disappointed that Rust dropped this feature in favour of async.
  - dralley 4y ago
    It makes sense for Rust. Rust with a significant runtime wouldn't provide much value over Go / Kotlin / Java / C#, whereas Rust without a heavy runtime is the first legitimate competitor to the dominance of C / C++.
samsquire 4y ago
This is apt timing. I wrote a thought recently of chunking loops and yielding after a certain number of chunks. The Loom scheduler does NOT guarantee resource starvation avoidance for high CPU usage. You need to Thread.yield. I also thought of creating a mapping from synchronous code and rewriting it to a tree of LMAX disruptors. I ported a wait-free ringbuffer from C++ to Java today - it was written by Alexander Krizhanovsky[1] In LMAX disruptor you split each IO request in half - you have a disruptor that enqueuing events events to request the IO and to pass the event on to a callback you enqueue the response on a different disruptor. If a synchronous request handler has 27 lines it represents a tree of 27 disruptor threads independently scaled. We can pipeline the request with 27 threads all in an event loop that pipeline each task × core count and without blocking the servicing thread or other work. So event loops without blocking! So CPU heavy tasks do not block other heavy CPU tasks and do not block the servicing thread and are not blocked by IO. https://github.com/samsquire/ideas4#51-rewrite-synchronous-code-into-lmax-disruptor-thread-pools---event-loops-that-dont-block-on-cpu-usage https://github.com/samsquire/ideas4#51-rewrite-synchronous-c... [1]: https://www.linuxjournal.com/content/lock-free-multi-producer-multi-consumer-queue-ring-buffer https://www.linuxjournal.com/content/lock-free-multi-produce...
- cobbzilla 4y ago
  per the article, Thread.yield is a noop to Loom, which the author argues should be fixed. Currently you have to Thread.sleep(1) to force Loom to unmount the OS thread in a CPU-bound green thread.
ewlkjlergerg 4y ago
[flagged]
papercrane 4y ago
The "State of Loom" document from May 2020 discusses this sort of problem. Long-term I believe the solution is to allow preempting at any safe-point and custom schedulers, but neither of those will be delivered in the first preview. https://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part2.html https://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part2....
ferdowsi 4y ago
The Twitter discussion mentions this: > We can forcefully preempt virtual threads at any safepoint poll point What is a "safepoint poll point"? A sleep call?
- papercrane 4y ago
  A safepoint in the JVM is a state where all the objects on the heap are consistent and the GC roots are known. These safe points are important for the GC to able to safely work. A safepoint poll points are points in a threads execution where it's able to go into a safepoint and transfer control to the JVM so it polls to see if it should. https://openjdk.java.net/groups/hotspot/docs/HotSpotGlossary.html https://openjdk.java.net/groups/hotspot/docs/HotSpotGlossary... https://shipilev.net/jvm/anatomy-quarks/22-safepoint-polls/ https://shipilev.net/jvm/anatomy-quarks/22-safepoint-polls/
- native_samples 4y ago
  No, safepoints are added to your code by the JVM automatically. When bytecode is being interpreted, every bytecode instruction has a safepoint after it. In compiled code polls are scattered thoughout the code, for example on method entries and loop back edges (so the comment above about needing to add code in tight loops to help the GC is wrong).
  - papercrane 4y ago
    > so the comment above about needing to add code in tight loops to help the GC is wrong This is something where the answer is "it depends". Hotspot will omit the safe point poll in counted loops since it knows those loops will terminate.
anonymousDan 4y ago
Can anyone comment on the duebugability of Loom (e.g. how do stack traces work)? Can Loom threads jump between native threads? Is it easy to understand what is going on when I run all this under a debugger? Having worked with some user level threading systems in the past as part of some libos work the the fact the tooling wasn't really set up to handle such issues made it really difficult to understand what was going on when we ran into problems.
- kaba0 4y ago
  I can’t answer everything, but good debugging is a very important design goal of Loom, so the usual debug methods should work the exact same way they do with native threads.
- aardvark179 4y ago
  You normally see the virtual thread’s stack. Virtual threads are normally run on a fork join pool or other executor service, and can run on any OS thread in that pool. They are not tied to any single OS thread. Yes, it is easy to understand in a debugger as support has been added to the debugger interface and to JVMTI, along with some Useful JFR events. We hope that in the future structured concurrency will make it even easier to understand huge numbers of virtual threads in production.
  - anonymousDan 4y ago
    Right but the problem we had was that when you fired up the debugger (GDB) you could only see the stack traces of the currently active user level thread on each host thread. To understand what was going on in other user threads (e.g. who is holding what locks) you had to manually go into the scheduler data structures to find the descheduled threads etc, which was extremely tedious. If that all happens out of the box for Loom then that is great.

98 points by lichtenberger 4y ago

The mentioned twitter discussion is also pretty interesting IMHO :-)

Not a new kind of issue, and not a problem IMHO. Some GCs need to reach a safepoint before doing their work; and when a classic Thread is chugging along in a tight loop, that thread is blocking the whole system for a pending GC. One hacky fix is to provide an opportunity for a safepoint (System.out.print every 10k iterations). Other tools like JVM command line options allow for observing safepoint triggering. The same goes for virtual threads: inserting a Thread.sleep(1L) every nth iteration of a tight loop does it. Also there was talk specifying a custom scheduler for the virtual threads. That would not help in inserting switching opportunities, but it would give a way to define what kind of fairness you want.

test-account 4y ago
Agree. Let's document this trick and I'm happy with that unfair scheduling inconvenience for CPU heavy operations.
- deleted 4y ago
  [deleted]
- topspin 4y ago
  Seconded. Although I think Thread::yield should do what it says on the tin in the case of lightweight threads and it's hard to understand why they chose to not make it so, I don't care much. If the only problem I have to anticipate to get the full benefit of a well behaved lightweight threat scheduling is strategically sprinkling yield points inside those few tight loops that might actually threaten to hog a core then it's a net win.
  - anonymousDan 4y ago
    Having to do it manually is tedious and error prone IMHO. Better to have a reasonably good automated mechanism that you can disable if you wish).
    - topspin 4y ago
      > tedious and error prone IMHO It's neither. The frequency of non-blocking CPU bound sections of most applications is low and so the need to introduce scheduling hints is also infrequent. Aside from an endless, non-blocking loop that never yields omitting possibly beneficial yield points isn't an error; scheduling will not be ideal but the process will ultimately produce the same result.
  - topspin 4y ago
    Thinking about this further, a pragmatic enhancement to Thread::yield might look like this: enum Hint { ALWAYS, FREQUENT, SOMETIMES, INFREQUENT, NEVER } Thread::yield(Hint); The method could be intrinsic to the compiler+runtime to provide runtime optimization (loop unrolling, tuning, etc.) of the 2nd - 4th cases. This would neatly avoid the need to: long n = 0; while (true) { // stuff if (++n > SOME_LIMIT) { n = 0; // yield here } } ... or similar.
anonymousDan 4y ago
Seems like safepoints are an ideal way to do this. It sounds similar in ways to how the BEAM runtime for Elixir/Erlang will count reductions and preempts a lightweight process. I guess the interesting question is whether the same safepoint testing mechanism can be reused directly or whether some more complex logic is required on the critical path
samsquire 4y ago
Putting an if statement in your loops shall slow down your loops. Instead chunk/partition the loop and run it in chunks.

klabb3 4y ago

I only observe Java from a distance but everything I've read about project loom is amazing. Speaking as someone who has worked intricately on an async runtime in another language, I've more and more started to question the premises around the explicitly async paradigm. Lots of things we were "promised" with async turned out to be just as complicated in the threaded (i.e. blocking) programming models, and now we have two incompatible programming models. I'm now of the opinion that we should improve (instead of replace) our threaded models and do under-the-hood optimizations to sort out the performance issues. If I had a dime for everytime someone made a blocking call from an async function.. And who can blame them? Most blocking functions aren't even documented as such, you just have to know.

sebazzz 4y ago
> If I had a dime for everytime someone made a blocking call from an async function.. And who can blame them? Most blocking functions aren't even documented as such, you just have to know. Isn't that exactly what loom aims to resolve?
lichtenberger 4y ago
Yes, I think Loom is an amazing step forward and it's brilliant to just change the blocking methods and thus existing code might already benefit with minor changes.
derriz 4y ago
Yeah, I've used a bunch of non-blocking paradigms over the years and all become painful quickly once the system becomes non-trivial, except for green threads. I was very disappointed that Rust dropped this feature in favour of async.
- dralley 4y ago
  It makes sense for Rust. Rust with a significant runtime wouldn't provide much value over Go / Kotlin / Java / C#, whereas Rust without a heavy runtime is the first legitimate competitor to the dominance of C / C++.

samsquire 4y ago

This is apt timing. I wrote a thought recently of chunking loops and yielding after a certain number of chunks. The Loom scheduler does NOT guarantee resource starvation avoidance for high CPU usage. You need to Thread.yield. I also thought of creating a mapping from synchronous code and rewriting it to a tree of LMAX disruptors. I ported a wait-free ringbuffer from C++ to Java today - it was written by Alexander Krizhanovsky[1] In LMAX disruptor you split each IO request in half - you have a disruptor that enqueuing events events to request the IO and to pass the event on to a callback you enqueue the response on a different disruptor. If a synchronous request handler has 27 lines it represents a tree of 27 disruptor threads independently scaled. We can pipeline the request with 27 threads all in an event loop that pipeline each task × core count and without blocking the servicing thread or other work. So event loops without blocking! So CPU heavy tasks do not block other heavy CPU tasks and do not block the servicing thread and are not blocked by IO. https://github.com/samsquire/ideas4#51-rewrite-synchronous-code-into-lmax-disruptor-thread-pools---event-loops-that-dont-block-on-cpu-usage https://github.com/samsquire/ideas4#51-rewrite-synchronous-c... [1]: https://www.linuxjournal.com/content/lock-free-multi-producer-multi-consumer-queue-ring-buffer https://www.linuxjournal.com/content/lock-free-multi-produce...

cobbzilla 4y ago
per the article, Thread.yield is a noop to Loom, which the author argues should be fixed. Currently you have to Thread.sleep(1) to force Loom to unmount the OS thread in a CPU-bound green thread.

ewlkjlergerg 4y ago

[flagged]

papercrane 4y ago

The "State of Loom" document from May 2020 discusses this sort of problem. Long-term I believe the solution is to allow preempting at any safe-point and custom schedulers, but neither of those will be delivered in the first preview. https://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part2.html https://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part2....

ferdowsi 4y ago

The Twitter discussion mentions this: > We can forcefully preempt virtual threads at any safepoint poll point What is a "safepoint poll point"? A sleep call?

papercrane 4y ago
A safepoint in the JVM is a state where all the objects on the heap are consistent and the GC roots are known. These safe points are important for the GC to able to safely work. A safepoint poll points are points in a threads execution where it's able to go into a safepoint and transfer control to the JVM so it polls to see if it should. https://openjdk.java.net/groups/hotspot/docs/HotSpotGlossary.html https://openjdk.java.net/groups/hotspot/docs/HotSpotGlossary... https://shipilev.net/jvm/anatomy-quarks/22-safepoint-polls/ https://shipilev.net/jvm/anatomy-quarks/22-safepoint-polls/
native_samples 4y ago
No, safepoints are added to your code by the JVM automatically. When bytecode is being interpreted, every bytecode instruction has a safepoint after it. In compiled code polls are scattered thoughout the code, for example on method entries and loop back edges (so the comment above about needing to add code in tight loops to help the GC is wrong).
- papercrane 4y ago
  > so the comment above about needing to add code in tight loops to help the GC is wrong This is something where the answer is "it depends". Hotspot will omit the safe point poll in counted loops since it knows those loops will terminate.

anonymousDan 4y ago

Can anyone comment on the duebugability of Loom (e.g. how do stack traces work)? Can Loom threads jump between native threads? Is it easy to understand what is going on when I run all this under a debugger? Having worked with some user level threading systems in the past as part of some libos work the the fact the tooling wasn't really set up to handle such issues made it really difficult to understand what was going on when we ran into problems.

kaba0 4y ago
I can’t answer everything, but good debugging is a very important design goal of Loom, so the usual debug methods should work the exact same way they do with native threads.
aardvark179 4y ago
You normally see the virtual thread’s stack. Virtual threads are normally run on a fork join pool or other executor service, and can run on any OS thread in that pool. They are not tied to any single OS thread. Yes, it is easy to understand in a debugger as support has been added to the debugger interface and to JVMTI, along with some Useful JFR events. We hope that in the future structured concurrency will make it even easier to understand huge numbers of virtual threads in production.
- anonymousDan 4y ago
  Right but the problem we had was that when you fired up the debugger (GDB) you could only see the stack traces of the currently active user level thread on each host thread. To understand what was going on in other user threads (e.g. who is holding what locks) you had to manually go into the scheduler data structures to find the descheduled threads etc, which was extremely tedious. If that all happens out of the box for Loom then that is great.