6 ms·

Lord of the Io_uring (2020)

173 points by LAC-Tech 2y ago

jauntywundrkind 2y ago
Definitely one of the best pieces of documentation out there for io_uring. But I'm not sure how much if at all it's been updated since 2020 & Linux 5.5. https://web.archive.org/web/20200527021134/https://unixism.net/loti/index.html https://web.archive.org/web/20200527021134/https://unixism.n...
- alecco 2y ago
  Yeah, it should have (2020) Previous discussion https://news.ycombinator.com/item?id=23132549 https://news.ycombinator.com/item?id=23132549
hinkley 2y ago
But they were all of them deceived?
- LAC-Tech 2y ago
  This made me laugh a lot. I can spot Tolkien's language from a mile off!
  - ratherbefuddled 2y ago
    Technically Peter Jackson's :)
- Cthulhu_ 2y ago
  And nine, nine async I/O programming APIs were gifted to the race of Linux users, who above all else desire power.
  - api 2y ago
    But this next API, we'll get it right. Let's call it io_uring2!
  - hinkley 2y ago
    I’m now imagining Torvalds riding a hell-hawk.
- friend_Fernando 2y ago
  One io_uring to root them all.
mgaunard 2y ago
a lot of the functionality was significantly improved in 6 and isn't reflected there. In practice io_uring can be used in many different ways, and it can be challenging to find the most efficient one.
- LAC-Tech 2y ago
  What are the big changes in 6? links welcome.
  - alecco 2y ago
    https://kernelnewbies.org/Linux_6.0#io_uring_features https://kernelnewbies.org/Linux_6.0#io_uring_features but only mentions zero copy and https://lwn.net/Articles/879724/ https://lwn.net/Articles/879724/ also https://www.phoronix.com/news/Linux-6.0-IO-Block-IO_uring https://www.phoronix.com/news/Linux-6.0-IO-Block-IO_uring
    - mgaunard 2y ago
      there were other changes to do with how register buffers work etc.
accelbred 2y ago
I'd like to use io_uring, but as long as it bypasses seccomp it should be disabled whenever seccomp is in use. As such, I use epoll, and find it annoying when kernel APIs like ublk require io_uring. The places I'd want to use ublk are inside sandboxes using seccomp. Given that container runtimes, hardened kernels, chromeos, etc., disable io_uring, using it means needing an epoll fallback anyways, so might as well just use epoll and not maintain two async backends for your application.
- poincaredisk 2y ago
  Is there a specific io_uring opcode you would like disabled in your sandboxes? It's not like io_uring is a complete seccomp bypass, just another syscall that provides an alternative way to do many things. I doubt you block "read" or "accept" in docker, for example. You can't execute a sysctl or mount a filesystem using io_uring, which are things that are actually blocked in Docker by default. edit: on the other hand, a good reason to disable uring in containers is that it's infested with vulnerabilities. It's new, complex, and does a whole lot of things - all of which make serious security bugs there quite common right now.
  - accelbred 2y ago
    Out of current ones, at a quick glance: connect, openat, openat2, renameat, mkdirat, and bind. More importantly, I'd like to block any opcode I haven't whitelisted, even when my software runs on future kernels with more opcodes available. Now that I think about it, how does io_uring interact with landlock?
  - ibotty 2y ago
    It's not only potentially infested with vulnerabilities. It's also not possible to filter io_uring using seccomp at all. So if you allow io_uring, you allow all that is possible with it.
  - JoshTriplett 2y ago
    > infested with vulnerabilities Current io_uring is not particularly prone to vulnerabilities. The original version of it had a design that often led to them (a kernel thread doing operations on behalf of the process and not always remembering to set the appropriate privileges), but it no longer uses that design, and the current design is much more resilient. Unfortunately, the original design led to a reputation that it's still trying to shake.
t00 2y ago
There are examples of cat and cp using io_uring. What are the chances of having io_uring utilised by standard commands to improve overall Linux performance? I presume GNU utils are not Linux specific hence such commands are programmed for a generic *nix. Another one is I could not find a benchmark with io_uring - this would confirm the benefit of going from epoll.
- fweimer 2y ago
  GNU coreutils already has tons of Linux-specific code. But it would be a bit of a kernel fail if io_uring were faster or other preferable to copy_file_range for cp (at least for files that do not have holes).
  - Sesse__ 2y ago
    Not at all; with io_uring, you can copy multiple files in parallel (and in fewer syscalls), which is a huge win for small files.
    - kragen 2y ago
      On a hard disk, copying multiple files in parallel is likely to make the copy run slower because it spends more time seeking back and forth between the files (except for small files). Perhaps that isn't a problem with SSDs? It seems like you'd still end up with the data from the different files interleaved in the erase blocks currently being written instead of contiguous, which seems like it would slow down all subsequent reads of those files (unless they're less than a page in size).
      - Sesse__ 2y ago
        > On a hard disk, copying multiple files in parallel is likely to make the copy run slower because it spends more time seeking back and forth between the files (except for small files). Certainly not; it's likely to make it run faster, since you can use the elevator algorithm more efficiently instead of seeking back and forth between the files. You can easily measure this yourself by using comparing wcp, which uses io_uring, and GNU cp (remember to empty the cache between each run).
twen_ty 2y ago
[flagged]
- JoshTriplett 2y ago
  Windows has copied io_uring from Linux.
  - cobbzilla 2y ago
    This is a troll, but NT did indeed support async IO via WaitForMultipleObjects in the late 90s, long before Linux had a good async IO story.
    - p_ing 2y ago
      I don't think it's a troll (though not a particularly useful comment); Linux has had no true async story thus far. poll, epoll, et. al. are all synchronous behind-the-scenes. What Linux still lacks is an OVERLAPPED data structure. NT has supported async I/O since it's inception. It was a design principle of the kernel -- all I/O operations in the kernel are async'ed.
      - JoshTriplett 2y ago
        I wasn't referring to async I/O. I'm talking about the ability to make system calls using a user/kernel shared memory buffer, without having to enter and exit the kernel. (This is particularly important with all the security mitigations that make kernel entry/exit more expensive.) https://windows-internals.com/ioring-vs-io_uring-a-comparison-of-windows-and-linux-implementations/ https://windows-internals.com/ioring-vs-io_uring-a-compariso...
        p_ing 2y ago
        Technically Windows has Registered I/O in Windows 8/Server 2012 for networking which provided this functionality. I/O Rings in Windows extended this to other types of I/O, but of course with a separate API. https://serverframework.com/asynchronousevents/2011/10/windows-8-registered-io-networking-extensions.html https://serverframework.com/asynchronousevents/2011/10/windo... So if we're talking about concepts... NT first, again ;-)
samsquire 2y ago
This document helped me learn the io_uring API. You can use io_uring with epoll to monitor eventfd to wake up your sleeping with io_uring wait for completions. I have implemented a barrier and thread safe techniques that I am trying to turn into a command line tool My goal is that thread safe performant servers are easy to write. I am using bloom filters for fast set intersection. I intend to use Simd instructions with the bloom hashes.
Thaxll 2y ago
Someone can comment on the security implications of sharing a buffer between user space and kernel space?
- fragmede 2y ago
  As you suspect, it's not awesome. https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring
- alexgartrell 2y ago
  Sharing a queue itself is not new https://www.kernel.org/doc/html/v5.8/networking/packet_mmap.html https://www.kernel.org/doc/html/v5.8/networking/packet_mmap.... and https://docs.kernel.org/next/userspace-api/perf_ring_buffer.html https://docs.kernel.org/next/userspace-api/perf_ring_buffer.... are two examples. Issues with io_uring security mostly stemmed from an old architecture and just the fact that there's a ton of surface area.
- quotemstr 2y ago
  binder shares a buffer between kernel and user space on billions of Android devices, and Android is by far the most secure Linux distribution. There's nothing wrong with the general concept.
saghm 2y ago
I'm realizing from the title of this that the intended pronunciation of "uring" is probably "yoo-ring"; for some reason I had mentally been reading it as "yurr-ring" all this time, and I guess I never heard anyone say it out loud before. In retrospect, I probably could have guessed that I might be missing something given that I had no clue what "uring" was supposed to mean.
dvektor 2y ago
I was actually glued to that page for a few days recently, it's a great write-up. io_uring is such a tremendous improvement over epoll, in both speed and user experience. With sqpoll, vectored ops and proper batching you can get some crazyy speed. I am definitely looking forward to seeing some of these seccomp and privilege issues getting fixed and getting container support in the future.

173 points by LAC-Tech 2y ago

Definitely one of the best pieces of documentation out there for io_uring. But I'm not sure how much if at all it's been updated since 2020 & Linux 5.5. https://web.archive.org/web/20200527021134/https://unixism.net/loti/index.html https://web.archive.org/web/20200527021134/https://unixism.n...

alecco 2y ago
Yeah, it should have (2020) Previous discussion https://news.ycombinator.com/item?id=23132549 https://news.ycombinator.com/item?id=23132549

hinkley 2y ago

But they were all of them deceived?

LAC-Tech 2y ago
This made me laugh a lot. I can spot Tolkien's language from a mile off!
- ratherbefuddled 2y ago
  Technically Peter Jackson's :)
Cthulhu_ 2y ago
And nine, nine async I/O programming APIs were gifted to the race of Linux users, who above all else desire power.
- api 2y ago
  But this next API, we'll get it right. Let's call it io_uring2!
- hinkley 2y ago
  I’m now imagining Torvalds riding a hell-hawk.
friend_Fernando 2y ago
One io_uring to root them all.

mgaunard 2y ago

a lot of the functionality was significantly improved in 6 and isn't reflected there. In practice io_uring can be used in many different ways, and it can be challenging to find the most efficient one.

LAC-Tech 2y ago
What are the big changes in 6? links welcome.
- alecco 2y ago
  https://kernelnewbies.org/Linux_6.0#io_uring_features https://kernelnewbies.org/Linux_6.0#io_uring_features but only mentions zero copy and https://lwn.net/Articles/879724/ https://lwn.net/Articles/879724/ also https://www.phoronix.com/news/Linux-6.0-IO-Block-IO_uring https://www.phoronix.com/news/Linux-6.0-IO-Block-IO_uring
  - mgaunard 2y ago
    there were other changes to do with how register buffers work etc.

accelbred 2y ago

I'd like to use io_uring, but as long as it bypasses seccomp it should be disabled whenever seccomp is in use. As such, I use epoll, and find it annoying when kernel APIs like ublk require io_uring. The places I'd want to use ublk are inside sandboxes using seccomp. Given that container runtimes, hardened kernels, chromeos, etc., disable io_uring, using it means needing an epoll fallback anyways, so might as well just use epoll and not maintain two async backends for your application.

poincaredisk 2y ago
Is there a specific io_uring opcode you would like disabled in your sandboxes? It's not like io_uring is a complete seccomp bypass, just another syscall that provides an alternative way to do many things. I doubt you block "read" or "accept" in docker, for example. You can't execute a sysctl or mount a filesystem using io_uring, which are things that are actually blocked in Docker by default. edit: on the other hand, a good reason to disable uring in containers is that it's infested with vulnerabilities. It's new, complex, and does a whole lot of things - all of which make serious security bugs there quite common right now.
- accelbred 2y ago
  Out of current ones, at a quick glance: connect, openat, openat2, renameat, mkdirat, and bind. More importantly, I'd like to block any opcode I haven't whitelisted, even when my software runs on future kernels with more opcodes available. Now that I think about it, how does io_uring interact with landlock?
- ibotty 2y ago
  It's not only potentially infested with vulnerabilities. It's also not possible to filter io_uring using seccomp at all. So if you allow io_uring, you allow all that is possible with it.
- JoshTriplett 2y ago
  > infested with vulnerabilities Current io_uring is not particularly prone to vulnerabilities. The original version of it had a design that often led to them (a kernel thread doing operations on behalf of the process and not always remembering to set the appropriate privileges), but it no longer uses that design, and the current design is much more resilient. Unfortunately, the original design led to a reputation that it's still trying to shake.

t00 2y ago

There are examples of cat and cp using io_uring. What are the chances of having io_uring utilised by standard commands to improve overall Linux performance? I presume GNU utils are not Linux specific hence such commands are programmed for a generic *nix. Another one is I could not find a benchmark with io_uring - this would confirm the benefit of going from epoll.

fweimer 2y ago
GNU coreutils already has tons of Linux-specific code. But it would be a bit of a kernel fail if io_uring were faster or other preferable to copy_file_range for cp (at least for files that do not have holes).
- Sesse__ 2y ago
  Not at all; with io_uring, you can copy multiple files in parallel (and in fewer syscalls), which is a huge win for small files.
  - kragen 2y ago
    On a hard disk, copying multiple files in parallel is likely to make the copy run slower because it spends more time seeking back and forth between the files (except for small files). Perhaps that isn't a problem with SSDs? It seems like you'd still end up with the data from the different files interleaved in the erase blocks currently being written instead of contiguous, which seems like it would slow down all subsequent reads of those files (unless they're less than a page in size).
    - Sesse__ 2y ago
      > On a hard disk, copying multiple files in parallel is likely to make the copy run slower because it spends more time seeking back and forth between the files (except for small files). Certainly not; it's likely to make it run faster, since you can use the elevator algorithm more efficiently instead of seeking back and forth between the files. You can easily measure this yourself by using comparing wcp, which uses io_uring, and GNU cp (remember to empty the cache between each run).

twen_ty 2y ago

[flagged]

JoshTriplett 2y ago
Windows has copied io_uring from Linux.
- cobbzilla 2y ago
  This is a troll, but NT did indeed support async IO via WaitForMultipleObjects in the late 90s, long before Linux had a good async IO story.
  - p_ing 2y ago
    I don't think it's a troll (though not a particularly useful comment); Linux has had no true async story thus far. poll, epoll, et. al. are all synchronous behind-the-scenes. What Linux still lacks is an OVERLAPPED data structure. NT has supported async I/O since it's inception. It was a design principle of the kernel -- all I/O operations in the kernel are async'ed.
    - JoshTriplett 2y ago
      I wasn't referring to async I/O. I'm talking about the ability to make system calls using a user/kernel shared memory buffer, without having to enter and exit the kernel. (This is particularly important with all the security mitigations that make kernel entry/exit more expensive.) https://windows-internals.com/ioring-vs-io_uring-a-comparison-of-windows-and-linux-implementations/ https://windows-internals.com/ioring-vs-io_uring-a-compariso...
      - p_ing 2y ago
        Technically Windows has Registered I/O in Windows 8/Server 2012 for networking which provided this functionality. I/O Rings in Windows extended this to other types of I/O, but of course with a separate API. https://serverframework.com/asynchronousevents/2011/10/windows-8-registered-io-networking-extensions.html https://serverframework.com/asynchronousevents/2011/10/windo... So if we're talking about concepts... NT first, again ;-)

samsquire 2y ago

This document helped me learn the io_uring API. You can use io_uring with epoll to monitor eventfd to wake up your sleeping with io_uring wait for completions. I have implemented a barrier and thread safe techniques that I am trying to turn into a command line tool My goal is that thread safe performant servers are easy to write. I am using bloom filters for fast set intersection. I intend to use Simd instructions with the bloom hashes.

Thaxll 2y ago

Someone can comment on the security implications of sharing a buffer between user space and kernel space?

fragmede 2y ago
As you suspect, it's not awesome. https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=io_uring
alexgartrell 2y ago
Sharing a queue itself is not new https://www.kernel.org/doc/html/v5.8/networking/packet_mmap.html https://www.kernel.org/doc/html/v5.8/networking/packet_mmap.... and https://docs.kernel.org/next/userspace-api/perf_ring_buffer.html https://docs.kernel.org/next/userspace-api/perf_ring_buffer.... are two examples. Issues with io_uring security mostly stemmed from an old architecture and just the fact that there's a ton of surface area.
quotemstr 2y ago
binder shares a buffer between kernel and user space on billions of Android devices, and Android is by far the most secure Linux distribution. There's nothing wrong with the general concept.

saghm 2y ago

I'm realizing from the title of this that the intended pronunciation of "uring" is probably "yoo-ring"; for some reason I had mentally been reading it as "yurr-ring" all this time, and I guess I never heard anyone say it out loud before. In retrospect, I probably could have guessed that I might be missing something given that I had no clue what "uring" was supposed to mean.

dvektor 2y ago

I was actually glued to that page for a few days recently, it's a great write-up. io_uring is such a tremendous improvement over epoll, in both speed and user experience. With sqpoll, vectored ops and proper batching you can get some crazyy speed. I am definitely looking forward to seeing some of these seccomp and privilege issues getting fixed and getting container support in the future.