4 ms·

> What io_uring does do is provide a way to poll without needing to wait, but if you haven’t received new events when you poll, you’re not on the fast path any

> What io_uring does do is provide a way to poll without needing to wait, but if you haven’t received new events when you poll, you’re not on the fast path any more. Isn't "not on the fast path any more" a bit absolutist? io_uring's "slow" path is roughly one syscall per iteration, right? That's still many fewer syscalls than one syscall per IO operation (or more if any return EAGAIN/EWOULDBLOCK) as you'd be doing without it. I'm not sure I really care about eliminating that last syscall per iteration; it seems minor in comparison.

klodolph 6y ago
io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall. I am a bit baffled how this could possibly be considered an “absolutist” viewpoint—I am just saying that there exist scenarios where io_uring is not helpful. This should be uncontroversial.
- scottlamb 6y ago
  You mean when there's only one thing to do per iteration? I'd describe that as when mostly idle. As the system gets more loaded, the one syscall per iteration matters less and less.
- jorangreef 6y ago
  > io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall. That's not correct, io_uring was "absolutely" designed, at least in the technical sense, for zero syscalls in the slow path (if you want to): IORING_SETUP_SQPOLL When this flag is specified, a kernel thread is created to perform submission queue polling. An io_uring instance configured in this way enables an application to issue I/O without ever context switching into the kernel. By using the submission queue to fill in new submission queue entries and watching for completions on the completion queue, the application can submit and reap I/Os without doing a single system call. From the man page: https://manpages.debian.org/unstable/liburing-dev/io_uring_setup.2.en.html https://manpages.debian.org/unstable/liburing-dev/io_uring_s... This mode required privileges in early kernel versions but that's already changed. Things are moving fast.
volta83 6y ago
> io_uring's "slow" path is roughly one syscall per iteration, right? Right, so if a blocking API makes 1 syscall, io_uring would make N syscals for N iterations.
- jorangreef 6y ago
  Where 1 iteration and 1 syscall to io_uring_enter() is submitting 100s of I/O operations per io_uring_enter() syscall (and you can even run the ring buffers with the kernel set to poll so you can do 0 syscalls if that's not already enough). That's pretty huge amortization. Rough benchmarks we've done are showing double throughput for io_uring for 4096 byte AF sector write/fsync/read combos: https://github.com/coilhq/tigerbeetle/tree/master/demos/io_uring https://github.com/coilhq/tigerbeetle/tree/master/demos/io_u...
  - volta83 6y ago
    > Where 1 iteration and 1 syscall to io_uring_enter() is submitting 100s of I/O operations per io_uring_enter() syscall (and you can even run the ring buffers with the kernel set to poll so you can do 0 syscalls if that's not already enough). Same for the blocking case. If I do a syscall to read a whole file, its just 1 syscall creating millions of I/O operations.
    - jorangreef 6y ago
      Sure, but that's all you'll ever do with the blocking case: 1 syscall at a time, while your program sits and does nothing with the CPU, whereas with io_uring at least you can do CPU while you wait on your IO. So even ignoring the IORING_SQPOLL option that requires no io_uring_enter() syscall, a basic usage of io_uring is still going to be faster. io_uring is a bicycle for IO, and you can ride it as fast as you want to. But it's apples and oranges to blocking IO, which is always stuck in first gear.
      - volta83 6y ago
        > while your program sits and does nothing with the CPU The CPU can run other threads while the hardware does DMA transfers. The thread just yields when the transfer is started, and a hardware exception wakes it up when the DMA transfer finishes.

klodolph 6y ago

io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall. I am a bit baffled how this could possibly be considered an “absolutist” viewpoint—I am just saying that there exist scenarios where io_uring is not helpful. This should be uncontroversial.

scottlamb 6y ago
You mean when there's only one thing to do per iteration? I'd describe that as when mostly idle. As the system gets more loaded, the one syscall per iteration matters less and less.
jorangreef 6y ago
> io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall. That's not correct, io_uring was "absolutely" designed, at least in the technical sense, for zero syscalls in the slow path (if you want to): IORING_SETUP_SQPOLL When this flag is specified, a kernel thread is created to perform submission queue polling. An io_uring instance configured in this way enables an application to issue I/O without ever context switching into the kernel. By using the submission queue to fill in new submission queue entries and watching for completions on the completion queue, the application can submit and reap I/Os without doing a single system call. From the man page: https://manpages.debian.org/unstable/liburing-dev/io_uring_setup.2.en.html https://manpages.debian.org/unstable/liburing-dev/io_uring_s... This mode required privileges in early kernel versions but that's already changed. Things are moving fast.

volta83 6y ago

> io_uring's "slow" path is roughly one syscall per iteration, right? Right, so if a blocking API makes 1 syscall, io_uring would make N syscals for N iterations.

jorangreef 6y ago
Where 1 iteration and 1 syscall to io_uring_enter() is submitting 100s of I/O operations per io_uring_enter() syscall (and you can even run the ring buffers with the kernel set to poll so you can do 0 syscalls if that's not already enough). That's pretty huge amortization. Rough benchmarks we've done are showing double throughput for io_uring for 4096 byte AF sector write/fsync/read combos: https://github.com/coilhq/tigerbeetle/tree/master/demos/io_uring https://github.com/coilhq/tigerbeetle/tree/master/demos/io_u...
- volta83 6y ago
  > Where 1 iteration and 1 syscall to io_uring_enter() is submitting 100s of I/O operations per io_uring_enter() syscall (and you can even run the ring buffers with the kernel set to poll so you can do 0 syscalls if that's not already enough). Same for the blocking case. If I do a syscall to read a whole file, its just 1 syscall creating millions of I/O operations.
  - jorangreef 6y ago
    Sure, but that's all you'll ever do with the blocking case: 1 syscall at a time, while your program sits and does nothing with the CPU, whereas with io_uring at least you can do CPU while you wait on your IO. So even ignoring the IORING_SQPOLL option that requires no io_uring_enter() syscall, a basic usage of io_uring is still going to be faster. io_uring is a bicycle for IO, and you can ride it as fast as you want to. But it's apples and oranges to blocking IO, which is always stuck in first gear.
    - volta83 6y ago
      > while your program sits and does nothing with the CPU The CPU can run other threads while the hardware does DMA transfers. The thread just yields when the transfer is started, and a hardware exception wakes it up when the DMA transfer finishes.