8 ms·

Descriptorless Files for Io_uring

113 points by signa11 5y ago

marcodiego 5y ago
I hope one day async io-heavy programs will be written with a big user space part and many small ebpf parts running in kernel space communicating with userspace using io_uring without wasting a single syscall. This could bring system efficiency to new levels.
- eru 5y ago
  Learning from exo-kernels might be the alternative.
  - monocasa 5y ago
    io_uring combined with ebpf is learning from exokernels AFAICT. XOK for instance had at least three or four in kernel VMs exposed to user space depending on how you count them, and was sort of defined by user space using those vms to safely manipulate kernel contexts.
    - rwmj 5y ago
      Those VMs were used to decide who owns (eg) a network packet, which is important for exokernels since the tiny kernel is only there to multiplex the hardware so needs to know ownership, but is really nothing at all like eBPF.
      - monocasa 5y ago
        They did all sorts of other work too. For one example, their futex() equivalent was a program called a wake predicate that could read from that process's address space to decide if that thread should transition back to runnable. And deciding who owns a network packet is literally how "extended Berkley Packet Filter" go it's start as well.
- alexgartrell 5y ago
  We’re working on it :) [1] The interesting problem is that it’s still kind of a developer experience mess. We can get pretty far with libbpf skeleton, libbpf-rs, etc, but I think we’re still waiting on the “killer” framework for this (or some other kind of language support). [1] I work with Pavel, Jens, and Alexei
  - fstrthnscnd 5y ago
    > We’re working on it :) [1] Here's a bunch of questions for you then: is the io_uring API similar to what we see in message based micro-kernels like Mach, where any syscall is implemented as a message posted to the kernel? If so, is this what Linux is heading towards? I also see a similarity between this recent development and what we see in GPU libraries, where early API like OpenGL did not provide any control of the graphic stack "internals", but gradually opened them with each new major revision, so that it became possible to tell the stack to allocate memory for vertex and textures buffers for instance. Are we going to see a similar trend with OS kernels? edit: added missing word (emphasized)
    - ithkuil 5y ago
      IIRC mach kernel implements IPC by making context switches (syscalls) for every IPC. There are also other microkernels with different IPC impl tradeoffs, though. Is there any that implements async rpc through pure message passing via shared memory ring buffers?
      - fstrthnscnd 5y ago
        I know that DragonflyBSD was born on the idea of implementing syscalls as a messages, while retaining the structure of a monolithic kernel (FreeBSD 4), but I don't know if the messages are kept in userland or there's still a context switch involved. The annoying part of having a relatively large piece of shared data between userland and kernel is that the kernel needs a way to ensure that this data hasn't been tampered with "illegally". I wonder how this is handled with io_uring.
- Veserv 5y ago
  syscalls are really not that expensive. The hardware overhead to cross the privilege boundary both times is only like 100 nanoseconds on modern hardware with maybe a few hundred nanoseconds with full speculation mitigations enabled. Essentially everything else is work that someone in the system needs to do. That work would not go away even if you were to link everything into the kernel, so you really only save the hardware overhead by reducing the number of syscalls. So, unless your code is mainly hot loops <1-5 us in duration with a syscall needed per loop you would not really be getting that much efficiency by reducing the number of syscalls. To gain real benefits you need to be able to make the syscall implementations faster which is certainly possible with a different syscall paradigm.
  - jcelerier 5y ago
    > The hardware overhead to cross the privilege boundary both times is only like 100 nanoseconds on modern hardware When your total time budget is 1 or 2 milliseconds that sounds really really expensive
    - Veserv 5y ago
      100 nanoseconds out of 1 millisecond is 1/10000 or ~0.01% which is really not very significant. As I said, if you have hot loops with a 1 microsecond budget then you might see material benefits with a kernel architecture that requires fewer privilege crossings. However, in most such cases there are fairly easy ways of reorganizing the code to use the system architecture more efficiently, such as by batching multiple writes in userspace, that will also result in fewer syscalls in your critical path (and thus fewer privilege crossings) without changing the kernel architecture. It is actually a fairly rare problem where the only solution is to issue a vast numbers of syscalls quickly and in those cases that is usually a limitation of the kernel data structures and syscall implementation. Just to clarify the distinction, in the former you issue the same number of "syscalls", but the new kernel architecture results in fewer privilege crossing events where as my statement here is to rearchitect so you need to issue fewer syscalls.
      - loopz 5y ago
- flemhans 5y ago
  If rsync was rewritten this way, would it bring big improvements when running from userspace as a normal user?
- vidarh 5y ago
  Given where it is now, I dream of seeing it go the next step towards what Alexia Massalin's Synthesis kernel did, and start JIT'ing across the user/kernel boundary. Of course, there be dragons.
  - gpderetta 5y ago
    the specter of spectre raises is ugly head.
- rektide 5y ago
  > I hope one day async io-heavy programs will be written with a big user space part and many small ebpf parts running in kernel space communicating with userspace using io_uring without wasting a single syscall. Nicely stated elevator pitch. Big CSP / Communicating Sequential Processes[1] / process-calculus feel, from the lots of little parts working together description you have! We're somewhat re-primed to move in this direction from having done serverless "lambdas"/functions-as-a-service, too, I'd say. [1] https://en.wikipedia.org/wiki/Communicating_sequential_processes https://en.wikipedia.org/wiki/Communicating_sequential_proce...
ylyn 5y ago
I never thought Linux could become any more monolithic. But somehow it is! (Not that it's a bad thing.)
turbinerneiter 5y ago
OT: What does io_uring stand for? I get the io part, but what is uring? Is that an abbreviation for something?
- benschulz 5y ago
  I always assumed it's the verb uring[1], but that might well be wrong. [1]: https://en.wiktionary.org/wiki/uring https://en.wiktionary.org/wiki/uring
- the_mitsuhiko 5y ago
  It’s a ring buffer. Not sure what the u is. Micro?
  - gpderetta 5y ago
    userspace?
  - yxhuvud 5y ago
    No, it is two ring buffers, with the kernel in between moving requests from one side to the other. My guess is that the name describes the shape.
- leotaku 5y ago
  The interface uses ringbuffers for communication between the kernel and userspace. That's how I have always understood it at least, not sure if it's actually correct.
- 10000truths 5y ago
  Userspace ring buffers.
IshKebab 5y ago
Does this allow you to get around limits on the number of open files? I work on a system that only allows 4096 open files which means VSCode's file watching code doesn't work on large projects. It would be great it this provided a workaround.
- 10000truths 5y ago
  I highly doubt it. Suppose you want to run untrusted user code with fork() + setrlimit(RLIMIT_NOFILE, &limit) + exec(...). It would be rather silly if the user code could just create an io_uring to bypass resource limits.
  - IshKebab 5y ago
    But the open file limits aren't a trust issue, they're a resource issue. If this allows avoiding the reason for the resource limitation in the first place then there's no problem as far as I can see.
    - 10000truths 5y ago
      > But the open file limits aren't a trust issue, they're a resource issue. Not necessarily. I might use open file limits to deny all OS resources to user code except for some pipes that I explicitly provide it with from the parent process, like the following pseudocode: pipe(child_stdin_pipe); pipe(child_stdout_pipe); pipe(child_stderr_pipe); pid = fork(); if(pid is child) { dup2(child_stdin_pipe[0], 0); dup2(child_stdout_pipe[1], 1); dup2(child_stderr_pipe[1], 2); close_range(3, INT_MAX); setrlimit(RLIMIT_NOFILE, 3); // maybe setrlimit a bunch of other stuff, like cpu time exec(...); } elif(pid is parent) { close(child_stdin_pipe[1]); close(child_stdin_pipe[0]); close(child_stdin_pipe[0]); // send info to untrusted user code write(child_stdin_pipe[0], ...); // retrieve info from untrusted user code read(child_stdout_pipe[1], ...); read(child_stderr_pipe[1], ...); wait(); } Of course, this can be easier done with platform-specific tools like seccomp, but the point remains.

113 points by signa11 5y ago

I hope one day async io-heavy programs will be written with a big user space part and many small ebpf parts running in kernel space communicating with userspace using io_uring without wasting a single syscall. This could bring system efficiency to new levels.

eru 5y ago
Learning from exo-kernels might be the alternative.
- monocasa 5y ago
  io_uring combined with ebpf is learning from exokernels AFAICT. XOK for instance had at least three or four in kernel VMs exposed to user space depending on how you count them, and was sort of defined by user space using those vms to safely manipulate kernel contexts.
  - rwmj 5y ago
    Those VMs were used to decide who owns (eg) a network packet, which is important for exokernels since the tiny kernel is only there to multiplex the hardware so needs to know ownership, but is really nothing at all like eBPF.
    - monocasa 5y ago
      They did all sorts of other work too. For one example, their futex() equivalent was a program called a wake predicate that could read from that process's address space to decide if that thread should transition back to runnable. And deciding who owns a network packet is literally how "extended Berkley Packet Filter" go it's start as well.
alexgartrell 5y ago
We’re working on it :) [1] The interesting problem is that it’s still kind of a developer experience mess. We can get pretty far with libbpf skeleton, libbpf-rs, etc, but I think we’re still waiting on the “killer” framework for this (or some other kind of language support). [1] I work with Pavel, Jens, and Alexei
- fstrthnscnd 5y ago
  > We’re working on it :) [1] Here's a bunch of questions for you then: is the io_uring API similar to what we see in message based micro-kernels like Mach, where any syscall is implemented as a message posted to the kernel? If so, is this what Linux is heading towards? I also see a similarity between this recent development and what we see in GPU libraries, where early API like OpenGL did not provide any control of the graphic stack "internals", but gradually opened them with each new major revision, so that it became possible to tell the stack to allocate memory for vertex and textures buffers for instance. Are we going to see a similar trend with OS kernels? edit: added missing word (emphasized)
  - ithkuil 5y ago
    IIRC mach kernel implements IPC by making context switches (syscalls) for every IPC. There are also other microkernels with different IPC impl tradeoffs, though. Is there any that implements async rpc through pure message passing via shared memory ring buffers?
    - fstrthnscnd 5y ago
      I know that DragonflyBSD was born on the idea of implementing syscalls as a messages, while retaining the structure of a monolithic kernel (FreeBSD 4), but I don't know if the messages are kept in userland or there's still a context switch involved. The annoying part of having a relatively large piece of shared data between userland and kernel is that the kernel needs a way to ensure that this data hasn't been tampered with "illegally". I wonder how this is handled with io_uring.
Veserv 5y ago
syscalls are really not that expensive. The hardware overhead to cross the privilege boundary both times is only like 100 nanoseconds on modern hardware with maybe a few hundred nanoseconds with full speculation mitigations enabled. Essentially everything else is work that someone in the system needs to do. That work would not go away even if you were to link everything into the kernel, so you really only save the hardware overhead by reducing the number of syscalls. So, unless your code is mainly hot loops <1-5 us in duration with a syscall needed per loop you would not really be getting that much efficiency by reducing the number of syscalls. To gain real benefits you need to be able to make the syscall implementations faster which is certainly possible with a different syscall paradigm.
- jcelerier 5y ago
  > The hardware overhead to cross the privilege boundary both times is only like 100 nanoseconds on modern hardware When your total time budget is 1 or 2 milliseconds that sounds really really expensive
  - Veserv 5y ago
    100 nanoseconds out of 1 millisecond is 1/10000 or ~0.01% which is really not very significant. As I said, if you have hot loops with a 1 microsecond budget then you might see material benefits with a kernel architecture that requires fewer privilege crossings. However, in most such cases there are fairly easy ways of reorganizing the code to use the system architecture more efficiently, such as by batching multiple writes in userspace, that will also result in fewer syscalls in your critical path (and thus fewer privilege crossings) without changing the kernel architecture. It is actually a fairly rare problem where the only solution is to issue a vast numbers of syscalls quickly and in those cases that is usually a limitation of the kernel data structures and syscall implementation. Just to clarify the distinction, in the former you issue the same number of "syscalls", but the new kernel architecture results in fewer privilege crossing events where as my statement here is to rearchitect so you need to issue fewer syscalls.
    - loopz 5y ago
flemhans 5y ago
If rsync was rewritten this way, would it bring big improvements when running from userspace as a normal user?
vidarh 5y ago
Given where it is now, I dream of seeing it go the next step towards what Alexia Massalin's Synthesis kernel did, and start JIT'ing across the user/kernel boundary. Of course, there be dragons.
- gpderetta 5y ago
  the specter of spectre raises is ugly head.
rektide 5y ago
> I hope one day async io-heavy programs will be written with a big user space part and many small ebpf parts running in kernel space communicating with userspace using io_uring without wasting a single syscall. Nicely stated elevator pitch. Big CSP / Communicating Sequential Processes[1] / process-calculus feel, from the lots of little parts working together description you have! We're somewhat re-primed to move in this direction from having done serverless "lambdas"/functions-as-a-service, too, I'd say. [1] https://en.wikipedia.org/wiki/Communicating_sequential_processes https://en.wikipedia.org/wiki/Communicating_sequential_proce...

ylyn 5y ago

I never thought Linux could become any more monolithic. But somehow it is! (Not that it's a bad thing.)

turbinerneiter 5y ago

OT: What does io_uring stand for? I get the io part, but what is uring? Is that an abbreviation for something?

benschulz 5y ago
I always assumed it's the verb uring[1], but that might well be wrong. [1]: https://en.wiktionary.org/wiki/uring https://en.wiktionary.org/wiki/uring
the_mitsuhiko 5y ago
It’s a ring buffer. Not sure what the u is. Micro?
- gpderetta 5y ago
  userspace?
- yxhuvud 5y ago
  No, it is two ring buffers, with the kernel in between moving requests from one side to the other. My guess is that the name describes the shape.
leotaku 5y ago
The interface uses ringbuffers for communication between the kernel and userspace. That's how I have always understood it at least, not sure if it's actually correct.
10000truths 5y ago
Userspace ring buffers.

IshKebab 5y ago

Does this allow you to get around limits on the number of open files? I work on a system that only allows 4096 open files which means VSCode's file watching code doesn't work on large projects. It would be great it this provided a workaround.

10000truths 5y ago
I highly doubt it. Suppose you want to run untrusted user code with fork() + setrlimit(RLIMIT_NOFILE, &limit) + exec(...). It would be rather silly if the user code could just create an io_uring to bypass resource limits.
- IshKebab 5y ago
  But the open file limits aren't a trust issue, they're a resource issue. If this allows avoiding the reason for the resource limitation in the first place then there's no problem as far as I can see.
  - 10000truths 5y ago
    > But the open file limits aren't a trust issue, they're a resource issue. Not necessarily. I might use open file limits to deny all OS resources to user code except for some pipes that I explicitly provide it with from the parent process, like the following pseudocode: pipe(child_stdin_pipe); pipe(child_stdout_pipe); pipe(child_stderr_pipe); pid = fork(); if(pid is child) { dup2(child_stdin_pipe[0], 0); dup2(child_stdout_pipe[1], 1); dup2(child_stderr_pipe[1], 2); close_range(3, INT_MAX); setrlimit(RLIMIT_NOFILE, 3); // maybe setrlimit a bunch of other stuff, like cpu time exec(...); } elif(pid is parent) { close(child_stdin_pipe[1]); close(child_stdin_pipe[0]); close(child_stdin_pipe[0]); // send info to untrusted user code write(child_stdin_pipe[0], ...); // retrieve info from untrusted user code read(child_stdout_pipe[1], ...); read(child_stderr_pipe[1], ...); wait(); } Of course, this can be easier done with platform-specific tools like seccomp, but the point remains.