19 ms·

I hope one day async io-heavy programs will be written with a big user space part and many small ebpf parts running in kernel space communicating with userspace

I hope one day async io-heavy programs will be written with a big user space part and many small ebpf parts running in kernel space communicating with userspace using io_uring without wasting a single syscall. This could bring system efficiency to new levels.

eru 5y ago
Learning from exo-kernels might be the alternative.
- monocasa 5y ago
  io_uring combined with ebpf is learning from exokernels AFAICT. XOK for instance had at least three or four in kernel VMs exposed to user space depending on how you count them, and was sort of defined by user space using those vms to safely manipulate kernel contexts.
  - rwmj 5y ago
    Those VMs were used to decide who owns (eg) a network packet, which is important for exokernels since the tiny kernel is only there to multiplex the hardware so needs to know ownership, but is really nothing at all like eBPF.
    - monocasa 5y ago
      They did all sorts of other work too. For one example, their futex() equivalent was a program called a wake predicate that could read from that process's address space to decide if that thread should transition back to runnable. And deciding who owns a network packet is literally how "extended Berkley Packet Filter" go it's start as well.
alexgartrell 5y ago
We’re working on it :) [1] The interesting problem is that it’s still kind of a developer experience mess. We can get pretty far with libbpf skeleton, libbpf-rs, etc, but I think we’re still waiting on the “killer” framework for this (or some other kind of language support). [1] I work with Pavel, Jens, and Alexei
- fstrthnscnd 5y ago
  > We’re working on it :) [1] Here's a bunch of questions for you then: is the io_uring API similar to what we see in message based micro-kernels like Mach, where any syscall is implemented as a message posted to the kernel? If so, is this what Linux is heading towards? I also see a similarity between this recent development and what we see in GPU libraries, where early API like OpenGL did not provide any control of the graphic stack "internals", but gradually opened them with each new major revision, so that it became possible to tell the stack to allocate memory for vertex and textures buffers for instance. Are we going to see a similar trend with OS kernels? edit: added missing word (emphasized)
  - ithkuil 5y ago
    IIRC mach kernel implements IPC by making context switches (syscalls) for every IPC. There are also other microkernels with different IPC impl tradeoffs, though. Is there any that implements async rpc through pure message passing via shared memory ring buffers?
    - fstrthnscnd 5y ago
      I know that DragonflyBSD was born on the idea of implementing syscalls as a messages, while retaining the structure of a monolithic kernel (FreeBSD 4), but I don't know if the messages are kept in userland or there's still a context switch involved. The annoying part of having a relatively large piece of shared data between userland and kernel is that the kernel needs a way to ensure that this data hasn't been tampered with "illegally". I wonder how this is handled with io_uring.
      - atq2119 5y ago
        Is this really fundamentally different to existing syscalls? They already have to guard against the possibility that another thread does funny business with the memory accesses by the syscall. You add ring buffer control structures, sure, but apart from that it seems largely the same problem?
Veserv 5y ago
syscalls are really not that expensive. The hardware overhead to cross the privilege boundary both times is only like 100 nanoseconds on modern hardware with maybe a few hundred nanoseconds with full speculation mitigations enabled. Essentially everything else is work that someone in the system needs to do. That work would not go away even if you were to link everything into the kernel, so you really only save the hardware overhead by reducing the number of syscalls. So, unless your code is mainly hot loops <1-5 us in duration with a syscall needed per loop you would not really be getting that much efficiency by reducing the number of syscalls. To gain real benefits you need to be able to make the syscall implementations faster which is certainly possible with a different syscall paradigm.
- jcelerier 5y ago
  > The hardware overhead to cross the privilege boundary both times is only like 100 nanoseconds on modern hardware When your total time budget is 1 or 2 milliseconds that sounds really really expensive
  - Veserv 5y ago
    100 nanoseconds out of 1 millisecond is 1/10000 or ~0.01% which is really not very significant. As I said, if you have hot loops with a 1 microsecond budget then you might see material benefits with a kernel architecture that requires fewer privilege crossings. However, in most such cases there are fairly easy ways of reorganizing the code to use the system architecture more efficiently, such as by batching multiple writes in userspace, that will also result in fewer syscalls in your critical path (and thus fewer privilege crossings) without changing the kernel architecture. It is actually a fairly rare problem where the only solution is to issue a vast numbers of syscalls quickly and in those cases that is usually a limitation of the kernel data structures and syscall implementation. Just to clarify the distinction, in the former you issue the same number of "syscalls", but the new kernel architecture results in fewer privilege crossing events where as my statement here is to rearchitect so you need to issue fewer syscalls.
    - loopz 5y ago
flemhans 5y ago
If rsync was rewritten this way, would it bring big improvements when running from userspace as a normal user?
vidarh 5y ago
Given where it is now, I dream of seeing it go the next step towards what Alexia Massalin's Synthesis kernel did, and start JIT'ing across the user/kernel boundary. Of course, there be dragons.
- gpderetta 5y ago
  the specter of spectre raises is ugly head.
rektide 5y ago
> I hope one day async io-heavy programs will be written with a big user space part and many small ebpf parts running in kernel space communicating with userspace using io_uring without wasting a single syscall. Nicely stated elevator pitch. Big CSP / Communicating Sequential Processes[1] / process-calculus feel, from the lots of little parts working together description you have! We're somewhat re-primed to move in this direction from having done serverless "lambdas"/functions-as-a-service, too, I'd say. [1] https://en.wikipedia.org/wiki/Communicating_sequential_processes https://en.wikipedia.org/wiki/Communicating_sequential_proce...

eru 5y ago

Learning from exo-kernels might be the alternative.

monocasa 5y ago
io_uring combined with ebpf is learning from exokernels AFAICT. XOK for instance had at least three or four in kernel VMs exposed to user space depending on how you count them, and was sort of defined by user space using those vms to safely manipulate kernel contexts.
- rwmj 5y ago
  Those VMs were used to decide who owns (eg) a network packet, which is important for exokernels since the tiny kernel is only there to multiplex the hardware so needs to know ownership, but is really nothing at all like eBPF.
  - monocasa 5y ago
    They did all sorts of other work too. For one example, their futex() equivalent was a program called a wake predicate that could read from that process's address space to decide if that thread should transition back to runnable. And deciding who owns a network packet is literally how "extended Berkley Packet Filter" go it's start as well.

alexgartrell 5y ago

We’re working on it :) [1] The interesting problem is that it’s still kind of a developer experience mess. We can get pretty far with libbpf skeleton, libbpf-rs, etc, but I think we’re still waiting on the “killer” framework for this (or some other kind of language support). [1] I work with Pavel, Jens, and Alexei

fstrthnscnd 5y ago
> We’re working on it :) [1] Here's a bunch of questions for you then: is the io_uring API similar to what we see in message based micro-kernels like Mach, where any syscall is implemented as a message posted to the kernel? If so, is this what Linux is heading towards? I also see a similarity between this recent development and what we see in GPU libraries, where early API like OpenGL did not provide any control of the graphic stack "internals", but gradually opened them with each new major revision, so that it became possible to tell the stack to allocate memory for vertex and textures buffers for instance. Are we going to see a similar trend with OS kernels? edit: added missing word (emphasized)
- ithkuil 5y ago
  IIRC mach kernel implements IPC by making context switches (syscalls) for every IPC. There are also other microkernels with different IPC impl tradeoffs, though. Is there any that implements async rpc through pure message passing via shared memory ring buffers?
  - fstrthnscnd 5y ago
    I know that DragonflyBSD was born on the idea of implementing syscalls as a messages, while retaining the structure of a monolithic kernel (FreeBSD 4), but I don't know if the messages are kept in userland or there's still a context switch involved. The annoying part of having a relatively large piece of shared data between userland and kernel is that the kernel needs a way to ensure that this data hasn't been tampered with "illegally". I wonder how this is handled with io_uring.
    - atq2119 5y ago
      Is this really fundamentally different to existing syscalls? They already have to guard against the possibility that another thread does funny business with the memory accesses by the syscall. You add ring buffer control structures, sure, but apart from that it seems largely the same problem?

Veserv 5y ago

syscalls are really not that expensive. The hardware overhead to cross the privilege boundary both times is only like 100 nanoseconds on modern hardware with maybe a few hundred nanoseconds with full speculation mitigations enabled. Essentially everything else is work that someone in the system needs to do. That work would not go away even if you were to link everything into the kernel, so you really only save the hardware overhead by reducing the number of syscalls. So, unless your code is mainly hot loops <1-5 us in duration with a syscall needed per loop you would not really be getting that much efficiency by reducing the number of syscalls. To gain real benefits you need to be able to make the syscall implementations faster which is certainly possible with a different syscall paradigm.

jcelerier 5y ago
> The hardware overhead to cross the privilege boundary both times is only like 100 nanoseconds on modern hardware When your total time budget is 1 or 2 milliseconds that sounds really really expensive
- Veserv 5y ago
  100 nanoseconds out of 1 millisecond is 1/10000 or ~0.01% which is really not very significant. As I said, if you have hot loops with a 1 microsecond budget then you might see material benefits with a kernel architecture that requires fewer privilege crossings. However, in most such cases there are fairly easy ways of reorganizing the code to use the system architecture more efficiently, such as by batching multiple writes in userspace, that will also result in fewer syscalls in your critical path (and thus fewer privilege crossings) without changing the kernel architecture. It is actually a fairly rare problem where the only solution is to issue a vast numbers of syscalls quickly and in those cases that is usually a limitation of the kernel data structures and syscall implementation. Just to clarify the distinction, in the former you issue the same number of "syscalls", but the new kernel architecture results in fewer privilege crossing events where as my statement here is to rearchitect so you need to issue fewer syscalls.
  - loopz 5y ago

flemhans 5y ago

If rsync was rewritten this way, would it bring big improvements when running from userspace as a normal user?

vidarh 5y ago

Given where it is now, I dream of seeing it go the next step towards what Alexia Massalin's Synthesis kernel did, and start JIT'ing across the user/kernel boundary. Of course, there be dragons.

gpderetta 5y ago
the specter of spectre raises is ugly head.

rektide 5y ago

> I hope one day async io-heavy programs will be written with a big user space part and many small ebpf parts running in kernel space communicating with userspace using io_uring without wasting a single syscall. Nicely stated elevator pitch. Big CSP / Communicating Sequential Processes[1] / process-calculus feel, from the lots of little parts working together description you have! We're somewhat re-primed to move in this direction from having done serverless "lambdas"/functions-as-a-service, too, I'd say. [1] https://en.wikipedia.org/wiki/Communicating_sequential_processes https://en.wikipedia.org/wiki/Communicating_sequential_proce...