3 ms·

> io_uring's "slow" path is roughly one syscall per iteration, right? Right, so if a blocking API makes 1 syscall, io_uring would make N syscals for N iteratio

> io_uring's "slow" path is roughly one syscall per iteration, right? Right, so if a blocking API makes 1 syscall, io_uring would make N syscals for N iterations.

jorangreef 6y ago
Where 1 iteration and 1 syscall to io_uring_enter() is submitting 100s of I/O operations per io_uring_enter() syscall (and you can even run the ring buffers with the kernel set to poll so you can do 0 syscalls if that's not already enough). That's pretty huge amortization. Rough benchmarks we've done are showing double throughput for io_uring for 4096 byte AF sector write/fsync/read combos: https://github.com/coilhq/tigerbeetle/tree/master/demos/io_uring https://github.com/coilhq/tigerbeetle/tree/master/demos/io_u...
- volta83 6y ago
  > Where 1 iteration and 1 syscall to io_uring_enter() is submitting 100s of I/O operations per io_uring_enter() syscall (and you can even run the ring buffers with the kernel set to poll so you can do 0 syscalls if that's not already enough). Same for the blocking case. If I do a syscall to read a whole file, its just 1 syscall creating millions of I/O operations.
  - jorangreef 6y ago
    Sure, but that's all you'll ever do with the blocking case: 1 syscall at a time, while your program sits and does nothing with the CPU, whereas with io_uring at least you can do CPU while you wait on your IO. So even ignoring the IORING_SQPOLL option that requires no io_uring_enter() syscall, a basic usage of io_uring is still going to be faster. io_uring is a bicycle for IO, and you can ride it as fast as you want to. But it's apples and oranges to blocking IO, which is always stuck in first gear.
    - volta83 6y ago
      > while your program sits and does nothing with the CPU The CPU can run other threads while the hardware does DMA transfers. The thread just yields when the transfer is started, and a hardware exception wakes it up when the DMA transfer finishes.
      - jorangreef 6y ago
        Sure, but we're comparing the efficiency of one of your program's single threads, because otherwise you could take that same argument you just used and turn it around and say fine, just run another thread then with another io_uring... and you're still ahead. You have to compare at the smallest unit of control plane. At the same time, multiple threads for a single program introduce context switches which are becoming horrendously expensive compared to the sheer number of IOPS that modern NVMe SSDs can do. Thread-per-core designs built around io_uring are the future of IO on Linux.

> io_uring's "slow" path is roughly one syscall per iteration, right? Right, so if a blocking API makes 1 syscall, io_uring would make N syscals for N iterations.

jorangreef 6y ago

Where 1 iteration and 1 syscall to io_uring_enter() is submitting 100s of I/O operations per io_uring_enter() syscall (and you can even run the ring buffers with the kernel set to poll so you can do 0 syscalls if that's not already enough). That's pretty huge amortization. Rough benchmarks we've done are showing double throughput for io_uring for 4096 byte AF sector write/fsync/read combos: https://github.com/coilhq/tigerbeetle/tree/master/demos/io_uring https://github.com/coilhq/tigerbeetle/tree/master/demos/io_u...

volta83 6y ago
> Where 1 iteration and 1 syscall to io_uring_enter() is submitting 100s of I/O operations per io_uring_enter() syscall (and you can even run the ring buffers with the kernel set to poll so you can do 0 syscalls if that's not already enough). Same for the blocking case. If I do a syscall to read a whole file, its just 1 syscall creating millions of I/O operations.
- jorangreef 6y ago
  Sure, but that's all you'll ever do with the blocking case: 1 syscall at a time, while your program sits and does nothing with the CPU, whereas with io_uring at least you can do CPU while you wait on your IO. So even ignoring the IORING_SQPOLL option that requires no io_uring_enter() syscall, a basic usage of io_uring is still going to be faster. io_uring is a bicycle for IO, and you can ride it as fast as you want to. But it's apples and oranges to blocking IO, which is always stuck in first gear.
  - volta83 6y ago
    > while your program sits and does nothing with the CPU The CPU can run other threads while the hardware does DMA transfers. The thread just yields when the transfer is started, and a hardware exception wakes it up when the DMA transfer finishes.
    - jorangreef 6y ago
      Sure, but we're comparing the efficiency of one of your program's single threads, because otherwise you could take that same argument you just used and turn it around and say fine, just run another thread then with another io_uring... and you're still ahead. You have to compare at the smallest unit of control plane. At the same time, multiple threads for a single program introduce context switches which are becoming horrendously expensive compared to the sheer number of IOPS that modern NVMe SSDs can do. Thread-per-core designs built around io_uring are the future of IO on Linux.