4 ms·

Operating system threads are always going to be (more) expensive

74 points by ingve 2y ago

deleted 2y ago
[deleted]
pjmlp 2y ago
Depends, given that operating systems like Solaris, macOS, Windows, also have userspace thread like APIs (tasks, dispach queues, concurrent runtime), that do the kernel/userspace M:N mapping in a language agnostic way.
- ArdentAardvark 2y ago
  The post literally mentions that a few paragraphs down in further detail. > The less that doing things with threads crosses the CPU's security boundary into (and out of) the kernel, the faster the threads go but the less we can really describe them as 'OS threads' and the harder it is to get things like forced thread preemption. And this applies not just for the 'OS threads' themselves but also to their activities. If you want 'OS threads' that perform 'synchronous IO through simple system calls', those IO operations are also transitioning into and out of the kernel. If you work to get around this purely through software, I suspect that what you wind up with is something that looks a lot like 'green' (user-space) threads with asynchronous IO once you peer behind the scenes of the abstractions that programs are seeing.
darkhelmet 2y ago
We took a shot at doing ultra-fast kernel threads on FreeBSD a few decades ago. For various reasons, it was reverted and removed a few major versions later. If you look a the old KSE work, the general gist was that if you were about to block in a syscall then you'd effectively get a signal-style longjmp back to your userland thread scheduler. You'd pick another thread and continue running all in the same process/task context. There were many problems with what we did and how we did it, but the unavoidable fundamental problem at the time was that it inverted assumptions about costs of low level primitives. Important(TM) software was optimized for the world where threads and blocking were expensive and things like pthread mutex operations were cheap. Our changes made threads and blocking trivially cheap but added non-trivial overhead to pthread mutex etc operations. Applications that made extensive use of pthread mutexes to coordinate work dispatching on a precious small pool of expensive threads were hit with devastating performance losses. Most critically, MySQL. We'd optimized for hundreds of thousands of threads rather than the case of multiplexing work over a few threads. It became apparent that this was going to be an eternal uphill battle and we eventually pulled the plug to do it the same way as Linux. We made a lot of mistakes with all of this.
- rjsw 2y ago
  NetBSD also tried it and reverted to kernel threads, some links here [1]. [1] https://en.wikipedia.org/wiki/Scheduler_activations https://en.wikipedia.org/wiki/Scheduler_activations
- rwmj 2y ago
  Why were mutexes more expensive?
  - tialaramex 2y ago
    Since they mention "a few decades ago" I'm guessing it's a very different mutex than the one you'd use today. Today I believe all of these operating systems (except MacOS, LOL) have either futex or an equivalent technology (the Windows thing works on bytes rather than aligned 32-bit values, so that's cool) and so it seems like only contention could be more expensive as the whole point of a futex is that uncontended acquisition is a single CPU store and that's userspace, the kernel is nowhere near it.
    - mmoskal 2y ago
      macOS has futexes it just doesn't talk about them... https://crates.io/crates/ulock-sys https://crates.io/crates/ulock-sys
      - tialaramex 2y ago
        I was not aware. I wonder if anybody tried asking Apple? That's what happened for SRWLOCK in Windows. Mara asked Microsoft if they can promise it actually does what it seems like it does, formally, and they did document that promise so then Rust used SRWLOCK [today your Rust does not use SRWLOCK, this year it was replaced for performance reasons, which is timely because it turns out SRWLOCK is faulty and so "Don't use SRWLOCK" is the most practical fix in the medium term, awkward for poor C++ though as they used SRWLOCK too]. Apple are even less talkative than Microsoft, but it is possible that the reason libc++ uses this is that it's actually guaranteed and if properly asked Apple would say so. It would be nice for (newer) macOS to get the same promises as the mainstream platforms with respect to these primitives.
fefe23 2y ago
This argument is basically hand waving, and it's factually wrong. Yes, switching from user to kernel mode and back is expensive. But let's count the syscalls. Model 1: Threads. 1 blocking read(), 1 blocking write(). Plus cost for the OS scheduler. Model 2: 1 "I want to read", 1 "can I read now?", 1 actual read, 1 "I want to write now", 1 "can I write now?", 1 actual write. Multiply as necessary if the read or write are only partial. Add IPC cost as necessary if you have more than one thread handling async events. Measuring by the syscall overhead, blocking I/O is much more efficient. There are other costs to it, though, so it usually consumes more resources per handled socket and turns out to be more expensive than async I/O. But it's not the syscall cost causing it.
- wilted-iris 2y ago
  This ignores the existence of batching capabilities like epoll and io uring. Async IO may need to do more syscalls than blocking IO for a single operation, but the story looks different if you consider a large number of operations that are awaited as a group.
  - bvrmn 2y ago
    It seems epoll could be slower than poll for async/await environments. It requires epoll_ctl (syscall) on each fd state transition. `poll` could wait in a single syscall for a small set of current blocked fds.
  - fefe23 2y ago
    io_uring actually changes this story because you can communicate events without crossing the syscall barrier, but only if you get enough incoming events to saturate your processing capabilities. epoll batching only works in respect to the event notifications from kernel to user space. The only other thing that could be considered batched is that you can switch from "I want to read" to "I want to write" without telling the kernel first that you don't want to read anymore. But that's hardly worth mentioning.
- bvrmn 2y ago
  Model 2 is wrong, most of async/await implements this: Nonblock read(), if EAGAIN: append fd into event waiting queue, wait event and read() again Nonblock write(), if EAGAIN: append fd into event waiting queue, wait event and write() again For loaded systems EAGAIN case is quite rare and syscall count is same.
deleted 2y ago
[deleted]
klabb3 2y ago
I don’t think we should parrot the syscalls-are-expensive too much. The overhead of thread syscalls is smaller than you’d think, and for say TCP and file IO our buffers are so big that we can mitigate the amounts in practice to saturate hardware (like NICs and SSDs) limitations, no problem. The issue has more to do with thread lifecycle management, ie creating, maintaining a stack and destroying the thread. When you have very lightweight tasks, those add up and have no direct mitigations. That’s a big reason we moved runtimes into userspace. I wish I could see the perf chart of these. If anyone finds them please link. It’s important to have objective facts to base these discussions in.
- sapiogram 2y ago
  > I wish I could see the perf chart of these. For thread creation, the author of the original post said "somewhere between 10 and 20 microseconds". This matches my own (very simple) benchmarks on my Ubuntu desktop, which was able to create and destroy just over 100k threads/second. Depending on your use case, using a thread pool could mitigate this cost entirely.
rwmj 2y ago
The problem with this analysis is that single system calls aren't the only way to invoke the operating system. You can also use an io_uring-style mechanism to batch your system calls. Or BPF to run mini-programs in the context of the OS, which is also a sort of batching. These don't have the same problematic overhead as system calls.
magicalhippo 2y ago
Related on-going thread: Asynchronous IO: the next billion-dollar mistake? https://news.ycombinator.com/item?id=41471707 https://news.ycombinator.com/item?id=41471707
Emjayen 2y ago
This is really just archaic *nix world problems; the NT kernel has had actual asynchronous I/O since inception -- even 30 years later, io_uring is mere batching to the kernel upper-half (and is a carbon-copy, down to the particular terminology, of RIO)
- FridgeSeal 2y ago
  > io_uring is mere batching to the kernel upper-half Your phrasing here suggests the NT kernel can/does have API’s that reach further down, is that correct, or am I reading too deeply? Haha

74 points by ingve 2y ago

deleted 2y ago

[deleted]

Depends, given that operating systems like Solaris, macOS, Windows, also have userspace thread like APIs (tasks, dispach queues, concurrent runtime), that do the kernel/userspace M:N mapping in a language agnostic way.

ArdentAardvark 2y ago
The post literally mentions that a few paragraphs down in further detail. > The less that doing things with threads crosses the CPU's security boundary into (and out of) the kernel, the faster the threads go but the less we can really describe them as 'OS threads' and the harder it is to get things like forced thread preemption. And this applies not just for the 'OS threads' themselves but also to their activities. If you want 'OS threads' that perform 'synchronous IO through simple system calls', those IO operations are also transitioning into and out of the kernel. If you work to get around this purely through software, I suspect that what you wind up with is something that looks a lot like 'green' (user-space) threads with asynchronous IO once you peer behind the scenes of the abstractions that programs are seeing.

darkhelmet 2y ago

We took a shot at doing ultra-fast kernel threads on FreeBSD a few decades ago. For various reasons, it was reverted and removed a few major versions later. If you look a the old KSE work, the general gist was that if you were about to block in a syscall then you'd effectively get a signal-style longjmp back to your userland thread scheduler. You'd pick another thread and continue running all in the same process/task context. There were many problems with what we did and how we did it, but the unavoidable fundamental problem at the time was that it inverted assumptions about costs of low level primitives. Important(TM) software was optimized for the world where threads and blocking were expensive and things like pthread mutex operations were cheap. Our changes made threads and blocking trivially cheap but added non-trivial overhead to pthread mutex etc operations. Applications that made extensive use of pthread mutexes to coordinate work dispatching on a precious small pool of expensive threads were hit with devastating performance losses. Most critically, MySQL. We'd optimized for hundreds of thousands of threads rather than the case of multiplexing work over a few threads. It became apparent that this was going to be an eternal uphill battle and we eventually pulled the plug to do it the same way as Linux. We made a lot of mistakes with all of this.

rjsw 2y ago
NetBSD also tried it and reverted to kernel threads, some links here [1]. [1] https://en.wikipedia.org/wiki/Scheduler_activations https://en.wikipedia.org/wiki/Scheduler_activations
rwmj 2y ago
Why were mutexes more expensive?
- tialaramex 2y ago
  Since they mention "a few decades ago" I'm guessing it's a very different mutex than the one you'd use today. Today I believe all of these operating systems (except MacOS, LOL) have either futex or an equivalent technology (the Windows thing works on bytes rather than aligned 32-bit values, so that's cool) and so it seems like only contention could be more expensive as the whole point of a futex is that uncontended acquisition is a single CPU store and that's userspace, the kernel is nowhere near it.
  - mmoskal 2y ago
    macOS has futexes it just doesn't talk about them... https://crates.io/crates/ulock-sys https://crates.io/crates/ulock-sys
    - tialaramex 2y ago
      I was not aware. I wonder if anybody tried asking Apple? That's what happened for SRWLOCK in Windows. Mara asked Microsoft if they can promise it actually does what it seems like it does, formally, and they did document that promise so then Rust used SRWLOCK [today your Rust does not use SRWLOCK, this year it was replaced for performance reasons, which is timely because it turns out SRWLOCK is faulty and so "Don't use SRWLOCK" is the most practical fix in the medium term, awkward for poor C++ though as they used SRWLOCK too]. Apple are even less talkative than Microsoft, but it is possible that the reason libc++ uses this is that it's actually guaranteed and if properly asked Apple would say so. It would be nice for (newer) macOS to get the same promises as the mainstream platforms with respect to these primitives.

fefe23 2y ago

This argument is basically hand waving, and it's factually wrong. Yes, switching from user to kernel mode and back is expensive. But let's count the syscalls. Model 1: Threads. 1 blocking read(), 1 blocking write(). Plus cost for the OS scheduler. Model 2: 1 "I want to read", 1 "can I read now?", 1 actual read, 1 "I want to write now", 1 "can I write now?", 1 actual write. Multiply as necessary if the read or write are only partial. Add IPC cost as necessary if you have more than one thread handling async events. Measuring by the syscall overhead, blocking I/O is much more efficient. There are other costs to it, though, so it usually consumes more resources per handled socket and turns out to be more expensive than async I/O. But it's not the syscall cost causing it.

wilted-iris 2y ago
This ignores the existence of batching capabilities like epoll and io uring. Async IO may need to do more syscalls than blocking IO for a single operation, but the story looks different if you consider a large number of operations that are awaited as a group.
- bvrmn 2y ago
  It seems epoll could be slower than poll for async/await environments. It requires epoll_ctl (syscall) on each fd state transition. `poll` could wait in a single syscall for a small set of current blocked fds.
- fefe23 2y ago
  io_uring actually changes this story because you can communicate events without crossing the syscall barrier, but only if you get enough incoming events to saturate your processing capabilities. epoll batching only works in respect to the event notifications from kernel to user space. The only other thing that could be considered batched is that you can switch from "I want to read" to "I want to write" without telling the kernel first that you don't want to read anymore. But that's hardly worth mentioning.
bvrmn 2y ago
Model 2 is wrong, most of async/await implements this: Nonblock read(), if EAGAIN: append fd into event waiting queue, wait event and read() again Nonblock write(), if EAGAIN: append fd into event waiting queue, wait event and write() again For loaded systems EAGAIN case is quite rare and syscall count is same.

deleted 2y ago

[deleted]

klabb3 2y ago

I don’t think we should parrot the syscalls-are-expensive too much. The overhead of thread syscalls is smaller than you’d think, and for say TCP and file IO our buffers are so big that we can mitigate the amounts in practice to saturate hardware (like NICs and SSDs) limitations, no problem. The issue has more to do with thread lifecycle management, ie creating, maintaining a stack and destroying the thread. When you have very lightweight tasks, those add up and have no direct mitigations. That’s a big reason we moved runtimes into userspace. I wish I could see the perf chart of these. If anyone finds them please link. It’s important to have objective facts to base these discussions in.

sapiogram 2y ago
> I wish I could see the perf chart of these. For thread creation, the author of the original post said "somewhere between 10 and 20 microseconds". This matches my own (very simple) benchmarks on my Ubuntu desktop, which was able to create and destroy just over 100k threads/second. Depending on your use case, using a thread pool could mitigate this cost entirely.

rwmj 2y ago

The problem with this analysis is that single system calls aren't the only way to invoke the operating system. You can also use an io_uring-style mechanism to batch your system calls. Or BPF to run mini-programs in the context of the OS, which is also a sort of batching. These don't have the same problematic overhead as system calls.

magicalhippo 2y ago

Related on-going thread: Asynchronous IO: the next billion-dollar mistake? https://news.ycombinator.com/item?id=41471707 https://news.ycombinator.com/item?id=41471707

Emjayen 2y ago

This is really just archaic *nix world problems; the NT kernel has had actual asynchronous I/O since inception -- even 30 years later, io_uring is mere batching to the kernel upper-half (and is a carbon-copy, down to the particular terminology, of RIO)

FridgeSeal 2y ago
> io_uring is mere batching to the kernel upper-half Your phrasing here suggests the NT kernel can/does have API’s that reach further down, is that correct, or am I reading too deeply? Haha