4 ms·

So maybe a more basic and fundamental question - but why can't OS threads be as good as these virtual threads? Why do we have to have two layers of thread abstr

by geokon 4y ago

So maybe a more basic and fundamental question - but why can't OS threads be as good as these virtual threads? Why do we have to have two layers of thread abstraction? My first reaction is "eww" quickly followed by "I'm prolly not understanding something here". Why can't we make the OS handle thousands of threads?

dboreham 4y ago
OS can handle thousands of threads easily. Kernel threading runs into scaling problems up at the 100's of thousands to millions level.
sidpatil 4y ago
My understanding is that OS thread operations require the processor/OS to switch between user mode and kernel mode, which is slow. Virtual threads don't require this, since they are managed entirely by the user process.
- belter 4y ago
  Correct. This is the main reason.
renox 4y ago
They could: Solaris had M:N threads at some point in time but 1) they're complicated 2) you pay a performance cost even if you don't use many threads..
- pjmlp 4y ago
  Called LWT, but they were eventually dropped exactly for those reasons.
Blackthorn 4y ago
We can, this was inspired by a talk given by Google about their own infrastructure. But it was apparently never open sourced.
verdagon 4y ago
Context switching costs are slightly higher for OS threads, but the real bottleneck is in the physical memory usage. An OS thread will start with an entire 4kb (page size on most OSs) for its stack and grow by 4kb as needed. A virtual thread will often have <1kb physical memory used, to start. In a program with lots of small OS threads, that's quite a bit of wasted space.
native_samples 4y ago
Thousands of threads is not a problem for any modern OS if you have enough RAM. Loom takes it up into the millions. As for why the JVM can do this and you can't with other runtimes, it's because it needs a complex runtime with very tight integration between the compiler, the GC, the threading library and various other subsystems. That's why the moment you make a native call the JVM loses the ability to switch between the virtual threads and has to start up more OS threads to compensate - it didn't create the code that's running on the stack at that point, so it can't work its usual magic. Fortunately the whole 'pure Java' movement of some decades ago means that actually you can do a heck of a lot of stuff without using native code. It's an ecosystem way less dependent on C extensions than most other languages.
duped 4y ago
I'm going to call them "green threads" in this context, don't read into it because the terminology is loose. - Since GT are known to a compiler/runtime they can be cooperative, as opposed to preemptive. OS threads cannot introspect into what your code is doing. That means GT may insert suspend and resume points at API boundaries, for example at any kind of IO. While an OS could implement this in theory (switch threads at i/o syscalls) in practice I don't think there are any thread schedulers that do this. - GT have orders of magnitude faster context switches than OS threads. All they need to do is swap a stack frame pointer. - GT often have growable stacks that start extremely small, OS threads have larger default stack sizes that don't grow. This makes starting and destroying GT cheaper, and they can use less memory than the equivalent number of OS threads. As for why OS threads don't just work the same, there are some disadvantages. You can't pin a GT to a single core, or set thread affinity. Context switching an OS thread is more expensive because it's doing more things, like saving the signal mask (which invokes a syscall, hence the cost). The large fixed stack size means that they don't have to perform memory allocation on a function call once the stack is too small. They have to be preemptive since the OS doesn't know anything about what the program is doing, other than a handful of i/o syscalls. They're essentially two different technologies for two use cases. The OS threads are more robust and closer to the metal, but they are expensive to start up and switch between. However GT are small and cheap. Another way to look at it is that there are only OS threads, and "virtual" or "green" threads are distinct concurrent tasks scheduled onto a pool of OS threads on the fly. The compiler inserts a bunch of book keeping to make scheduling those tasks easier. The only thing that makes them "thread like" is the API.
- zozbot234 4y ago
  Some operating systems implement standard OS threading API's on top of a pool of true "system threads". Notably, this even included some early Java implementations, before a 1-to-1 mapping between system and "user" threads became more popular for some reason.
  - duped 4y ago
    which makes a lot of sense once they have to map to logical cores
gpderetta 4y ago
M:N hybrid threads were in vogue at the turn of the millennium. Solaris had them, some of the *BSDs had them and IBM almost got them in Linux. There was also a lot of research on the topic (look for scheduler activation). They went out of fashion because they slowed down parallel programs (as opposed to highly concurrent programs) and required a lot additional complexity. Then the "10k Problem" article popularized poll-based architectures as opposed to threaded ones and the rest is history.
- zozbot234 4y ago
  Scheduler activation is not so different from the modern approach, using e.g. uring or IOCP (not really poll-based). The original terminology references "virtual processors" as opposed to system-level threads, and "user threads" as opposed to arbitrary tasks, but either way a task that's about to block must have its state saved and subsequently restored once the event has been handled, and these operations are managed by the user-level runtime not the OS.
  - gpderetta 4y ago
    I do think it is different. In fact I think that IOCP predates the scheduler activation work. Certainly a proactor event loop (uring iocp) is different from a reactor (poll) from an usage point of view, but you can implement one in term of the other. Scheduler activation is different and it is just a way for the kernel to notify userspace of all rescheduling events (including running out of the scheduling quantum or taking a page fault).
kerblang 4y ago
Since nobody else brought it up: One big item is non-blocking I/O. The idea of non-blocking is that you can manage thousands of network connections with only a handful of threads, because when you do thread-per-connection, you'll notice that your threads are mostly asleep in typical I/O heavy networked systems (like web-server/database stuff that most of us work on). You can do non-blocking in java now, but only with nasty callbacks and Futures and thread pools and such. In Go you get non-blocking "for free" so that it looks like blocking I/O but the runtime automatically suspends & resumes virtual threads as I/O buffers fill & drain. One of the big inspirations for Loom was Go jealousy. For all Go's faults, this is a really nice thing to have.

by geokon 4y ago

dboreham 4y ago

OS can handle thousands of threads easily. Kernel threading runs into scaling problems up at the 100's of thousands to millions level.

sidpatil 4y ago

My understanding is that OS thread operations require the processor/OS to switch between user mode and kernel mode, which is slow. Virtual threads don't require this, since they are managed entirely by the user process.

belter 4y ago
Correct. This is the main reason.

renox 4y ago

They could: Solaris had M:N threads at some point in time but 1) they're complicated 2) you pay a performance cost even if you don't use many threads..

pjmlp 4y ago
Called LWT, but they were eventually dropped exactly for those reasons.

Blackthorn 4y ago

We can, this was inspired by a talk given by Google about their own infrastructure. But it was apparently never open sourced.

verdagon 4y ago

Context switching costs are slightly higher for OS threads, but the real bottleneck is in the physical memory usage. An OS thread will start with an entire 4kb (page size on most OSs) for its stack and grow by 4kb as needed. A virtual thread will often have <1kb physical memory used, to start. In a program with lots of small OS threads, that's quite a bit of wasted space.

native_samples 4y ago

Thousands of threads is not a problem for any modern OS if you have enough RAM. Loom takes it up into the millions. As for why the JVM can do this and you can't with other runtimes, it's because it needs a complex runtime with very tight integration between the compiler, the GC, the threading library and various other subsystems. That's why the moment you make a native call the JVM loses the ability to switch between the virtual threads and has to start up more OS threads to compensate - it didn't create the code that's running on the stack at that point, so it can't work its usual magic. Fortunately the whole 'pure Java' movement of some decades ago means that actually you can do a heck of a lot of stuff without using native code. It's an ecosystem way less dependent on C extensions than most other languages.

duped 4y ago

I'm going to call them "green threads" in this context, don't read into it because the terminology is loose. - Since GT are known to a compiler/runtime they can be cooperative, as opposed to preemptive. OS threads cannot introspect into what your code is doing. That means GT may insert suspend and resume points at API boundaries, for example at any kind of IO. While an OS could implement this in theory (switch threads at i/o syscalls) in practice I don't think there are any thread schedulers that do this. - GT have orders of magnitude faster context switches than OS threads. All they need to do is swap a stack frame pointer. - GT often have growable stacks that start extremely small, OS threads have larger default stack sizes that don't grow. This makes starting and destroying GT cheaper, and they can use less memory than the equivalent number of OS threads. As for why OS threads don't just work the same, there are some disadvantages. You can't pin a GT to a single core, or set thread affinity. Context switching an OS thread is more expensive because it's doing more things, like saving the signal mask (which invokes a syscall, hence the cost). The large fixed stack size means that they don't have to perform memory allocation on a function call once the stack is too small. They have to be preemptive since the OS doesn't know anything about what the program is doing, other than a handful of i/o syscalls. They're essentially two different technologies for two use cases. The OS threads are more robust and closer to the metal, but they are expensive to start up and switch between. However GT are small and cheap. Another way to look at it is that there are only OS threads, and "virtual" or "green" threads are distinct concurrent tasks scheduled onto a pool of OS threads on the fly. The compiler inserts a bunch of book keeping to make scheduling those tasks easier. The only thing that makes them "thread like" is the API.

zozbot234 4y ago
Some operating systems implement standard OS threading API's on top of a pool of true "system threads". Notably, this even included some early Java implementations, before a 1-to-1 mapping between system and "user" threads became more popular for some reason.
- duped 4y ago
  which makes a lot of sense once they have to map to logical cores

gpderetta 4y ago

M:N hybrid threads were in vogue at the turn of the millennium. Solaris had them, some of the *BSDs had them and IBM almost got them in Linux. There was also a lot of research on the topic (look for scheduler activation). They went out of fashion because they slowed down parallel programs (as opposed to highly concurrent programs) and required a lot additional complexity. Then the "10k Problem" article popularized poll-based architectures as opposed to threaded ones and the rest is history.

zozbot234 4y ago
Scheduler activation is not so different from the modern approach, using e.g. uring or IOCP (not really poll-based). The original terminology references "virtual processors" as opposed to system-level threads, and "user threads" as opposed to arbitrary tasks, but either way a task that's about to block must have its state saved and subsequently restored once the event has been handled, and these operations are managed by the user-level runtime not the OS.
- gpderetta 4y ago
  I do think it is different. In fact I think that IOCP predates the scheduler activation work. Certainly a proactor event loop (uring iocp) is different from a reactor (poll) from an usage point of view, but you can implement one in term of the other. Scheduler activation is different and it is just a way for the kernel to notify userspace of all rescheduling events (including running out of the scheduling quantum or taking a page fault).

kerblang 4y ago

Since nobody else brought it up: One big item is non-blocking I/O. The idea of non-blocking is that you can manage thousands of network connections with only a handful of threads, because when you do thread-per-connection, you'll notice that your threads are mostly asleep in typical I/O heavy networked systems (like web-server/database stuff that most of us work on). You can do non-blocking in java now, but only with nasty callbacks and Futures and thread pools and such. In Go you get non-blocking "for free" so that it looks like blocking I/O but the runtime automatically suspends & resumes virtual threads as I/O buffers fill & drain. One of the big inspirations for Loom was Go jealousy. For all Go's faults, this is a really nice thing to have.