5 ms·

seeing what java virtual threads can do it brings the question, are OS threads done the wrong way? why cannot the OS do what java is doing?

dboreham 3y ago
They're not quite the same kind of thread. But some OS do have support for lightweight threads, e.g. NT. Also OSes today do handle as many threads as most applications ever need.
pshirshov 3y ago
Java's Virtual Threads (as any other green threads/fibers/etc) are scheduled cooperatively. Effectively a fiber is a tiny data structure which contains a closure which has to be executed next and some metadata.
exDM69 3y ago
Virtual threads, or green threads use a co-operative threading/coroutine model on top of the preemptive threads provided by the OS. In addition to Java virtual threads, similar systems exist in Go goroutines, Rust async and Haskell green threads for example. There are upsides and downsides to cooperative userspace threads. An erratic or malicious green thread can hog all the CPU time and starve other threads for example. An operating system can not trust the userspace to behave, starvation due to misbehaving thread is unacceptable. So the OS can not implement green threads alone, it needs the userspace for the co-operative switching.
- yetihehe 3y ago
  In erlang you have preemptive green threads. They don't block* and having a million on one medium server is not unheard of. * I managed to block a whole erlang VM one day, spoke with Joe Armstrong on chat, but they won't fix this, as it's pretty niche use case.
  - darraghenright 3y ago
    I'm probably not the only one curious to hear more about this :)
    - yetihehe 3y ago
      Make a ordered_bag ets table and try to insert several thousand records in one ets call. It will block whole VM (spinning one core at 100%, doesn't matter how many you have) for several seconds. It does this because ordered bag needs to find each key in table as it inserts it, resulting in n^2 complexity and this needs to be done atomically. So for 1 000 keys you have 1 000 000 comparisons during a VM-wide lock. Solution - don't insert so many records in one call into ets table.
      - ninkendo 3y ago
        I’m more curious to understand how userspace threads can be preemptive. :) I can think of a few ways: - The VM just doesn’t JIT and can decide to stop executing a thread by just not interpreting the next piece of bytecode and switching to another green thread instead (this would be pretty slow due to the lack of JIT) - The VM JITs, but inserts a preamble before every function call saying “Before executing this function, should I switch to another green thread first?”, and thus, so long as you call functions frequently enough, you “preempt” yourself. This is how Go does it, and it’s a well known thing in Go that if you never call a function for a while (like just doing a really huge for loop), the current goroutine doesn’t yield execution and hogs the whole OS thread.
        messe 3y ago
        > - The VM JITs, but inserts a preamble before every function call saying “Before executing this function, should I switch to another green thread first?”, and thus, so long as you call functions frequently enough, you “preempt” yourself. This is how Go does it, and it’s a well known thing in Go that if you never call a function for a while (like just doing a really huge for loop), the current goroutine doesn’t yield execution and hogs the whole OS thread. I'm not sure of the implementation details, but this hasn't been true for a while in Go. As of Go 1.14, goroutines are asynchronously preemptible, so loops without function calls no longer deadlock the scheduler or GC: https://go.dev/doc/go1.14#runtime https://go.dev/doc/go1.14#runtime
tester756 3y ago
let's go even further Why do we even waste resources on OS when we can write e.g UEFI application and run it "directly" and let it manage resources without OS overhead Nowadays runtimes like Java's, etc, etc. are doing more and more tasks that previously were managed by OSes
- gpderetta 3y ago
  Sure, unikernels are a thing. But being able to run a full OS is convenient, and often still faster overall as there are magnitudes more people working at optimizing general purpose OSs than specialized unikernels.
- spease 3y ago
  By the time you got done writing device drivers, etc. you would just have another OS. https://xkcd.com/927/ https://xkcd.com/927/ Also “without OS overhead” - things sitting in memory don’t necessary detract from performance, but not having hardware acceleration for lack of drivers certainly can. The only overhead guaranteed during execution by the OS is context switching, and if you want to avoid that you’re basically writing an RTOS for an embedded system.
pdpi 3y ago
All else being equal, the one thing Java can do that the OS can’t is avoid paying for context switches. There is a cost to crossing the user space/kernel space boundary that you can avoid with user space lightweight threads.
- otabdeveloper4 3y ago
  False. The context will need to be switched regardless of where you're doing it. (Userspace or kernel.) The cost of crossing the userspace/kernel boundary happens on any system call and has nothing to do with context switching. (Modern OS's typically context switch during system calls for i/o, exactly the same as userspace threads.)
  - gpderetta 3y ago
    User/kernel space switch is a different kind of switch. Also when you do it cooperatively, context switches can be a bit more efficient as the runtime has better knowledge about the set of live registers at the switch point.
    - otabdeveloper4 3y ago
      The "different kind of switch" you talk about doesn't exist on any OS of the past 30 years.
      - gpderetta 3y ago
        I'm talking about raising a process privilege level, via whatever syscall mechanism. Which is a completely different code path than a process/thread context switch and, among other things, doesn't require going through the scheduler.
  - Sesse__ 3y ago
    The page tables change in a kernel context switch, whereas they don't in a user context switch. Depending on your CPU and OS, this may or may not matter; e.g., modern CPUs typically have caches that are tagged with the context (e.g. PCID on Intel), but the Spectre flavor of the day may require that you simply flush the entire L1 cache on context switch anyway if you're unlucky. The biggest cost of a context switch, though, is typically that your CPU goes to do something else, which reduces the efficiency of any caches (including branch prediction etc.). This is the same whether you have kernel or userspace context switches, assuming the threads you're switching between touch different code and/or data.
twic 3y ago
The significant difference between virtual and real threads is the stack. Real threads have to support programs written in languages which assume that the stack is a large, contiguous block of memory which they can use without cooperation with the operating system. In practice, that means a large slice of address space has to be allocated for each thread, along with page table entries, and at least a page of physical memory. Whereas virtual threads are deeply integrated with the compiler and runtime which uses them, and don't need to support such programs, meaning they can do more subtle things, with growable stacks, segmented stacks, moving frames off and on the platform thread stack, etc. I suppose you could implement these more subtle things with real threads, but it would require a richer interface between the program and the operating system, which would need to be supported in compilers and language runtimes. FreeBSD did something like this with kernel scheduled entities, but it never caught on. As an aside, i wouldn't call virtual threads cooperatively scheduled. To me, that means that user code has to be written with scheduling in mind, to make sure that it hits yield points where the thread can enter the scheduler. Virtual threads don't require that, because the JVM can ensure the necessary entries to the scheduler, since it controls interpretation and compilation. That said, i believe that in the current implementation of virtual threads on the JVM, it is possible that a tight loop that doesn't call any methods will not be rescheduled; i'm hazy about the details of this, because surely this interacts with safepoints. But maybe it's only very slightly cooperatively scheduled!
- im3w1l 3y ago
  Just thinking aloud here... So functions already (typically) have a bit of prologue for setting up the stackframe, where they juggle rsp and rbp etc. I wonder if you could change the convention for that into something that basically does RequestStackFrame(SIZE) for some known SIZE. RequestStackFrame could then either grow the stack linearly or get memory in some entirely different region depending on what is needed. Though doing such a function call for every tiny little function is a bit excessive of course, I'm thinking maybe the compiler should track for every function how much stack it will need in the worst case (usage for the function itself + max of usage for the functions it calls). Functions that use memory below some threshold just grow linearly, while those that use above it will go to the trouble of requesting memory. Now this idea assumes that function calls happen in a DAG, it breaks if functions can call themselves or each other in a cycle. But that's actually a good realization maybe. Like a recursive function should perhaps always explicitly request stack space. Security and robustness benefits! What about libraries? Ideally library functions would be annotated with how much stack space they need (as a side note I have long thought that C-headers are inadequate for specifying how a function should be called. Not annotating stack space requirements one deficiency here!). Absent such annotations, I suppose we need to go with the status quo of making some guess.

dboreham 3y ago

They're not quite the same kind of thread. But some OS do have support for lightweight threads, e.g. NT. Also OSes today do handle as many threads as most applications ever need.

pshirshov 3y ago

Java's Virtual Threads (as any other green threads/fibers/etc) are scheduled cooperatively. Effectively a fiber is a tiny data structure which contains a closure which has to be executed next and some metadata.

exDM69 3y ago

Virtual threads, or green threads use a co-operative threading/coroutine model on top of the preemptive threads provided by the OS. In addition to Java virtual threads, similar systems exist in Go goroutines, Rust async and Haskell green threads for example. There are upsides and downsides to cooperative userspace threads. An erratic or malicious green thread can hog all the CPU time and starve other threads for example. An operating system can not trust the userspace to behave, starvation due to misbehaving thread is unacceptable. So the OS can not implement green threads alone, it needs the userspace for the co-operative switching.

yetihehe 3y ago
In erlang you have preemptive green threads. They don't block* and having a million on one medium server is not unheard of. * I managed to block a whole erlang VM one day, spoke with Joe Armstrong on chat, but they won't fix this, as it's pretty niche use case.
- darraghenright 3y ago
  I'm probably not the only one curious to hear more about this :)
  - yetihehe 3y ago
    Make a ordered_bag ets table and try to insert several thousand records in one ets call. It will block whole VM (spinning one core at 100%, doesn't matter how many you have) for several seconds. It does this because ordered bag needs to find each key in table as it inserts it, resulting in n^2 complexity and this needs to be done atomically. So for 1 000 keys you have 1 000 000 comparisons during a VM-wide lock. Solution - don't insert so many records in one call into ets table.
    - ninkendo 3y ago
      I’m more curious to understand how userspace threads can be preemptive. :) I can think of a few ways: - The VM just doesn’t JIT and can decide to stop executing a thread by just not interpreting the next piece of bytecode and switching to another green thread instead (this would be pretty slow due to the lack of JIT) - The VM JITs, but inserts a preamble before every function call saying “Before executing this function, should I switch to another green thread first?”, and thus, so long as you call functions frequently enough, you “preempt” yourself. This is how Go does it, and it’s a well known thing in Go that if you never call a function for a while (like just doing a really huge for loop), the current goroutine doesn’t yield execution and hogs the whole OS thread.
      - messe 3y ago
        > - The VM JITs, but inserts a preamble before every function call saying “Before executing this function, should I switch to another green thread first?”, and thus, so long as you call functions frequently enough, you “preempt” yourself. This is how Go does it, and it’s a well known thing in Go that if you never call a function for a while (like just doing a really huge for loop), the current goroutine doesn’t yield execution and hogs the whole OS thread. I'm not sure of the implementation details, but this hasn't been true for a while in Go. As of Go 1.14, goroutines are asynchronously preemptible, so loops without function calls no longer deadlock the scheduler or GC: https://go.dev/doc/go1.14#runtime https://go.dev/doc/go1.14#runtime

tester756 3y ago

let's go even further Why do we even waste resources on OS when we can write e.g UEFI application and run it "directly" and let it manage resources without OS overhead Nowadays runtimes like Java's, etc, etc. are doing more and more tasks that previously were managed by OSes

gpderetta 3y ago
Sure, unikernels are a thing. But being able to run a full OS is convenient, and often still faster overall as there are magnitudes more people working at optimizing general purpose OSs than specialized unikernels.
spease 3y ago
By the time you got done writing device drivers, etc. you would just have another OS. https://xkcd.com/927/ https://xkcd.com/927/ Also “without OS overhead” - things sitting in memory don’t necessary detract from performance, but not having hardware acceleration for lack of drivers certainly can. The only overhead guaranteed during execution by the OS is context switching, and if you want to avoid that you’re basically writing an RTOS for an embedded system.

pdpi 3y ago

All else being equal, the one thing Java can do that the OS can’t is avoid paying for context switches. There is a cost to crossing the user space/kernel space boundary that you can avoid with user space lightweight threads.

otabdeveloper4 3y ago
False. The context will need to be switched regardless of where you're doing it. (Userspace or kernel.) The cost of crossing the userspace/kernel boundary happens on any system call and has nothing to do with context switching. (Modern OS's typically context switch during system calls for i/o, exactly the same as userspace threads.)
- gpderetta 3y ago
  User/kernel space switch is a different kind of switch. Also when you do it cooperatively, context switches can be a bit more efficient as the runtime has better knowledge about the set of live registers at the switch point.
  - otabdeveloper4 3y ago
    The "different kind of switch" you talk about doesn't exist on any OS of the past 30 years.
    - gpderetta 3y ago
      I'm talking about raising a process privilege level, via whatever syscall mechanism. Which is a completely different code path than a process/thread context switch and, among other things, doesn't require going through the scheduler.
- Sesse__ 3y ago
  The page tables change in a kernel context switch, whereas they don't in a user context switch. Depending on your CPU and OS, this may or may not matter; e.g., modern CPUs typically have caches that are tagged with the context (e.g. PCID on Intel), but the Spectre flavor of the day may require that you simply flush the entire L1 cache on context switch anyway if you're unlucky. The biggest cost of a context switch, though, is typically that your CPU goes to do something else, which reduces the efficiency of any caches (including branch prediction etc.). This is the same whether you have kernel or userspace context switches, assuming the threads you're switching between touch different code and/or data.

twic 3y ago

The significant difference between virtual and real threads is the stack. Real threads have to support programs written in languages which assume that the stack is a large, contiguous block of memory which they can use without cooperation with the operating system. In practice, that means a large slice of address space has to be allocated for each thread, along with page table entries, and at least a page of physical memory. Whereas virtual threads are deeply integrated with the compiler and runtime which uses them, and don't need to support such programs, meaning they can do more subtle things, with growable stacks, segmented stacks, moving frames off and on the platform thread stack, etc. I suppose you could implement these more subtle things with real threads, but it would require a richer interface between the program and the operating system, which would need to be supported in compilers and language runtimes. FreeBSD did something like this with kernel scheduled entities, but it never caught on. As an aside, i wouldn't call virtual threads cooperatively scheduled. To me, that means that user code has to be written with scheduling in mind, to make sure that it hits yield points where the thread can enter the scheduler. Virtual threads don't require that, because the JVM can ensure the necessary entries to the scheduler, since it controls interpretation and compilation. That said, i believe that in the current implementation of virtual threads on the JVM, it is possible that a tight loop that doesn't call any methods will not be rescheduled; i'm hazy about the details of this, because surely this interacts with safepoints. But maybe it's only very slightly cooperatively scheduled!

im3w1l 3y ago
Just thinking aloud here... So functions already (typically) have a bit of prologue for setting up the stackframe, where they juggle rsp and rbp etc. I wonder if you could change the convention for that into something that basically does RequestStackFrame(SIZE) for some known SIZE. RequestStackFrame could then either grow the stack linearly or get memory in some entirely different region depending on what is needed. Though doing such a function call for every tiny little function is a bit excessive of course, I'm thinking maybe the compiler should track for every function how much stack it will need in the worst case (usage for the function itself + max of usage for the functions it calls). Functions that use memory below some threshold just grow linearly, while those that use above it will go to the trouble of requesting memory. Now this idea assumes that function calls happen in a DAG, it breaks if functions can call themselves or each other in a cycle. But that's actually a good realization maybe. Like a recursive function should perhaps always explicitly request stack space. Security and robustness benefits! What about libraries? Ideally library functions would be annotated with how much stack space they need (as a side note I have long thought that C-headers are inadequate for specifying how a function should be called. Not annotating stack space requirements one deficiency here!). Absent such annotations, I suppose we need to go with the status quo of making some guess.