6 ms·

Linux kernel: Add io_uring IO interface

125 points by jorangreef 8y ago

htfy96 8y ago
Introduction to io_uring on lwn: https://lwn.net/Articles/776703/ https://lwn.net/Articles/776703/
tick_tock_tick 8y ago
Interesting has anyone run any tests comparing this interface's performance to epoll?
- htfy96 8y ago
  > Axboe claims that it is far more efficient, but no benchmark results have been included yet to back up that claim. Among other things, this interface can do asynchronous buffered I/O without a context switch in cases where the desired data is in the page cache; buffered I/O has always been a bit of a sore spot for Linux AIO.
- the8472 8y ago
  This isn't just for sockets and pipes. You can use it for disk IO too. This is the One Ring.
mmastrac 8y ago
This will be great when you can add splice ops to the ring in addition to read/write/sync. Zero-copy between sockets with no syscall!
newnewpdro 8y ago
With spectre/meltdown mitigations increasing the cost of syscalls I imagine this becomes quite significant. Somewhat orthogonally, does anyone happen to know if the hardware fixes eliminating the need for software mitigations still increase syscall cost substantially? I imagine if you have to inhibit speculation across context boundaries the cost is still higher than a pre-spectre world, even if done in hardware...
- nightfly 8y ago
  Last time I looked, the hardware mitigations were almost exactly as expensive as the software ones.
  - wtallis 8y ago
    I'm not sure if there are even any true hardware mitigations actually shipping yet. I think the closest we have are products that ship with stock microcode implementing the same changes that aftermarket microcode updates applied to existing products.
    - ac29 8y ago
      Some hardware fixes are already in place: https://www.anandtech.com/show/13301/spectre-and-meltdown-in-hardware-intel-clarifies-whiskey-lake-and-amber-lake https://www.anandtech.com/show/13301/spectre-and-meltdown-in...
  - saagarjha 8y ago
    That's because they effectively ossified the software fix in hardware, right? At some point we might be able to come up with a real way to fix the issue.
  - bonzini 8y ago
    For most of them (not Spectre v1) hardware fixes are easy, just adding some extra bits to cache or BTB tags. Problem is it's whack-a-mole.
- caf 8y ago
  The Meltdown workaround - separate kernel and userspace page tables with %cr3 switched on kernel entry and exit - is responsible for the bulk of the syscall cost increase. That one is entirely fixable in hardware without a significant cost penalty (after all, AMD wasn't affected by that one). The various kinds of Spectre are harder (and are wider than just the kernel, things like Javascript VMs are also affected), but the cost there is more diffuse.
grandinj 8y ago
This is impressive, but it adds yet another very complex io api to the Linux list. Compared to this Windows is looking more and more impressive and their api is decades old. I’m sure this is higher performing in some benchmark setup, but very little cpu time is spent on kernel overhead for any heavy io application. It would be nicer just to have a clean set of flexible asynchronous apis that all shared a relatively simpler model.
- devbug 8y ago
  It takes ~20 LOC to setup scaleable asynchronous I/O w/ IOCP. It's been like that since 1996. So that's the benchmark as far as I'm concerned.
  - iforgotpassword 8y ago
    It really wasn't all that great during the first couple years. And it still has a few annoying caveats today. The advertised flexibility (port sharing etc) most probably wasn't the only reason for http.sys back in the day :) EDIT: that is not to say that it is as horrible to use as epoll (at least in multithreaded programs), but my favorite has to be kqueue.
- yxhuvud 8y ago
  You should wait with your complaining until the userspace convenience methods are in place. This looks amazingly powerful with lots of applications.
- unmole 8y ago
  > but very little cpu time is spent on kernel overhead for any heavy io application I would really like to see some backing for that claim. In I/O heavy applications that I've seen (Think performance enhancing proxies), syscalls easily account for over half the CPU cycles.
  - grandinj 8y ago
    most of that is spent in the network stack, not the userspace->kernel layer
alexeiz 8y ago
The act of passing data through the io_uring interface should be called "uring-ating."

125 points by jorangreef 8y ago

htfy96 8y ago

Introduction to io_uring on lwn: https://lwn.net/Articles/776703/ https://lwn.net/Articles/776703/

tick_tock_tick 8y ago

Interesting has anyone run any tests comparing this interface's performance to epoll?

htfy96 8y ago
> Axboe claims that it is far more efficient, but no benchmark results have been included yet to back up that claim. Among other things, this interface can do asynchronous buffered I/O without a context switch in cases where the desired data is in the page cache; buffered I/O has always been a bit of a sore spot for Linux AIO.
the8472 8y ago
This isn't just for sockets and pipes. You can use it for disk IO too. This is the One Ring.

mmastrac 8y ago

This will be great when you can add splice ops to the ring in addition to read/write/sync. Zero-copy between sockets with no syscall!

newnewpdro 8y ago

With spectre/meltdown mitigations increasing the cost of syscalls I imagine this becomes quite significant. Somewhat orthogonally, does anyone happen to know if the hardware fixes eliminating the need for software mitigations still increase syscall cost substantially? I imagine if you have to inhibit speculation across context boundaries the cost is still higher than a pre-spectre world, even if done in hardware...

nightfly 8y ago
Last time I looked, the hardware mitigations were almost exactly as expensive as the software ones.
- wtallis 8y ago
  I'm not sure if there are even any true hardware mitigations actually shipping yet. I think the closest we have are products that ship with stock microcode implementing the same changes that aftermarket microcode updates applied to existing products.
  - ac29 8y ago
    Some hardware fixes are already in place: https://www.anandtech.com/show/13301/spectre-and-meltdown-in-hardware-intel-clarifies-whiskey-lake-and-amber-lake https://www.anandtech.com/show/13301/spectre-and-meltdown-in...
- saagarjha 8y ago
  That's because they effectively ossified the software fix in hardware, right? At some point we might be able to come up with a real way to fix the issue.
- bonzini 8y ago
  For most of them (not Spectre v1) hardware fixes are easy, just adding some extra bits to cache or BTB tags. Problem is it's whack-a-mole.
caf 8y ago
The Meltdown workaround - separate kernel and userspace page tables with %cr3 switched on kernel entry and exit - is responsible for the bulk of the syscall cost increase. That one is entirely fixable in hardware without a significant cost penalty (after all, AMD wasn't affected by that one). The various kinds of Spectre are harder (and are wider than just the kernel, things like Javascript VMs are also affected), but the cost there is more diffuse.

grandinj 8y ago

This is impressive, but it adds yet another very complex io api to the Linux list. Compared to this Windows is looking more and more impressive and their api is decades old. I’m sure this is higher performing in some benchmark setup, but very little cpu time is spent on kernel overhead for any heavy io application. It would be nicer just to have a clean set of flexible asynchronous apis that all shared a relatively simpler model.

devbug 8y ago
It takes ~20 LOC to setup scaleable asynchronous I/O w/ IOCP. It's been like that since 1996. So that's the benchmark as far as I'm concerned.
- iforgotpassword 8y ago
  It really wasn't all that great during the first couple years. And it still has a few annoying caveats today. The advertised flexibility (port sharing etc) most probably wasn't the only reason for http.sys back in the day :) EDIT: that is not to say that it is as horrible to use as epoll (at least in multithreaded programs), but my favorite has to be kqueue.
yxhuvud 8y ago
You should wait with your complaining until the userspace convenience methods are in place. This looks amazingly powerful with lots of applications.
unmole 8y ago
> but very little cpu time is spent on kernel overhead for any heavy io application I would really like to see some backing for that claim. In I/O heavy applications that I've seen (Think performance enhancing proxies), syscalls easily account for over half the CPU cycles.
- grandinj 8y ago
  most of that is spent in the network stack, not the userspace->kernel layer

alexeiz 8y ago

The act of passing data through the io_uring interface should be called "uring-ating."