3 ms·

Is it just me or moving things out of kernel space improves performance in general? Like context switching, mutex or entire TCP stack. I wonder what else can be

Is it just me or moving things out of kernel space improves performance in general? Like context switching, mutex or entire TCP stack. I wonder what else can be moved into user space.

Jtsummers 1y ago
Most things can be moved into user space. https://en.wikipedia.org/wiki/Microkernel https://en.wikipedia.org/wiki/Microkernel
viega 1y ago
It depends on the relative cost of what you're doing of course, but in general, yes, the cost of entering the kernel can be impactful, if you are making a lot of calls into it, since a lot of work has to happen every time. There are definitely user-land TCP/IP implementations, thread implementations (Go's being particularly notable) and even file systems (FUSE).
- bicolao 1y ago
  fuse probably isn't a good example here because you still have to enter kernel space if i'm not mistaken, then out again to the fs driver in userspace then probably back to kernel space (block driver). fuse has many upsides, but I don't think performance is one of them.
  - viega 1y ago
    Well, you still have to enter the kernel to actually queue an OS-level thread w/ a futex. The kernel supporting being able to move stuff to userland sure doesn't guarantee better performance-- the main opportunity from that perspective is minimizing how often you cross the boundary. You're 100% right that there are plenty of other considerations, often positive for lifting things out, like minimization of ring 0 attack surface.
whstl 1y ago
You're right. With the caveat that it's the moving in and out of the kernel that's the expensive part, so putting things inside it also helps with performance... at the expense of security and reliability, of course, so userspace it is!
toast0 1y ago
Piling on with others; it's avoiding the context switching and copying between user and kernel that improves performance. If you can stay in user space, or stay in kernel space, that is likely to be better performance than going back and forth. This is why sendfile is great; rather than storage -> kernel memory -> user memory -> kernel memory -> network, you get storage -> kernel memory -> network. That's two less copies, and fewer context switches.
the8472 1y ago
io_uring alleviates a lot of those problems by essentially providing a batched, asynchronous syscall API, including futex calls, as mentioned in other comments.
senderista 1y ago
Futexes can be moved into user space! https://webkit.org/blog/6161/locking-in-webkit/ https://webkit.org/blog/6161/locking-in-webkit/
- tialaramex 1y ago
  While I'm sure it sounds great to Apple fans, this ends up just being a circle in practice. We need a way to fall asleep when contended. Apple calls this a "Parking Lot" Linux calls it a "Futex" and as we've seen Microsoft have a different name. Like Microsoft's Apple's approach is slightly smaller (one byte not four) and over a decade later.
  - gpderetta 1y ago
    Well, but with a parking_lot abstraction, you only need a wait primitive per thread (as a thread_local for example), so it doesn't need to be a futex. It could be a eventfd which is interesting for integrating with other event loops.
  - senderista 1y ago
    By putting the wait queues in userspace you can customize behavior with callbacks, as the article shows.
omgtehlion 1y ago
Yes it is. In hft (or any algo trading nowadays) it is commonplace to use usermode networking since like ~2010 or even earlier. The part of NICs address space is mapped to a process (control registers and memory rings), and whole driver just runs in your process without going through the kernel. Of course, it ignores usual stuff like firewalls, iptables, etc, but who cares when all you need is the lowest latency possible )

Is it just me or moving things out of kernel space improves performance in general? Like context switching, mutex or entire TCP stack. I wonder what else can be moved into user space.

Jtsummers 1y ago

Most things can be moved into user space. https://en.wikipedia.org/wiki/Microkernel https://en.wikipedia.org/wiki/Microkernel

viega 1y ago

It depends on the relative cost of what you're doing of course, but in general, yes, the cost of entering the kernel can be impactful, if you are making a lot of calls into it, since a lot of work has to happen every time. There are definitely user-land TCP/IP implementations, thread implementations (Go's being particularly notable) and even file systems (FUSE).

bicolao 1y ago
fuse probably isn't a good example here because you still have to enter kernel space if i'm not mistaken, then out again to the fs driver in userspace then probably back to kernel space (block driver). fuse has many upsides, but I don't think performance is one of them.
- viega 1y ago
  Well, you still have to enter the kernel to actually queue an OS-level thread w/ a futex. The kernel supporting being able to move stuff to userland sure doesn't guarantee better performance-- the main opportunity from that perspective is minimizing how often you cross the boundary. You're 100% right that there are plenty of other considerations, often positive for lifting things out, like minimization of ring 0 attack surface.

whstl 1y ago

You're right. With the caveat that it's the moving in and out of the kernel that's the expensive part, so putting things inside it also helps with performance... at the expense of security and reliability, of course, so userspace it is!

toast0 1y ago

Piling on with others; it's avoiding the context switching and copying between user and kernel that improves performance. If you can stay in user space, or stay in kernel space, that is likely to be better performance than going back and forth. This is why sendfile is great; rather than storage -> kernel memory -> user memory -> kernel memory -> network, you get storage -> kernel memory -> network. That's two less copies, and fewer context switches.

the8472 1y ago

io_uring alleviates a lot of those problems by essentially providing a batched, asynchronous syscall API, including futex calls, as mentioned in other comments.

senderista 1y ago

Futexes can be moved into user space! https://webkit.org/blog/6161/locking-in-webkit/ https://webkit.org/blog/6161/locking-in-webkit/

tialaramex 1y ago
While I'm sure it sounds great to Apple fans, this ends up just being a circle in practice. We need a way to fall asleep when contended. Apple calls this a "Parking Lot" Linux calls it a "Futex" and as we've seen Microsoft have a different name. Like Microsoft's Apple's approach is slightly smaller (one byte not four) and over a decade later.
- gpderetta 1y ago
  Well, but with a parking_lot abstraction, you only need a wait primitive per thread (as a thread_local for example), so it doesn't need to be a futex. It could be a eventfd which is interesting for integrating with other event loops.
- senderista 1y ago
  By putting the wait queues in userspace you can customize behavior with callbacks, as the article shows.

omgtehlion 1y ago

Yes it is. In hft (or any algo trading nowadays) it is commonplace to use usermode networking since like ~2010 or even earlier. The part of NICs address space is mapped to a process (control registers and memory rings), and whole driver just runs in your process without going through the kernel. Of course, it ignores usual stuff like firewalls, iptables, etc, but who cares when all you need is the lowest latency possible )