9 ms·

Lord of the io_uring: io_uring tutorial, examples and reference

267 points by shuss 6y ago

tyingq 6y ago
There are some benchmarks that show io_uring as a significant boost over aio: https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.6-IO-uring-Tests https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.... I see that nginx accepted a pull request to use it, mid last year: https://github.com/hakasenyang/openssl-patch/issues/21 https://github.com/hakasenyang/openssl-patch/issues/21 Curious if it's also been adopted by other popular IO intensive software.
- shuss 6y ago
  Oh, yeah. QEMU 5.0 already uses io_uring. In fact, it uses liburing. Check out the changelog: https://wiki.qemu.org/ChangeLog/5.0 https://wiki.qemu.org/ChangeLog/5.0
  - Twirrim 6y ago
    To save people time, there's a single reference to it on the changelog: > The file-posix driver can now use the io_uring interface of Linux with aio=io_uring side note: I did note a change we built made it in to a released version of qemu: > qemu-img convert -n now understands a --target-is-zero option, which tells it that the target image is completely zero, so it does not need to be zeroed again. That's saving us so much time and I/O
- jandrewrogers 6y ago
  I have not adopted io_uring yet because it isn't clear that it will provide useful performance improvements over linux aio in cases where the disk I/O subsystem is already highly optimized. Where io_uring seems to show a benefit relative to linux aio is more naive software design, which adds a lot of value but is a somewhat different value proposition than has been expressed. For software that is already capable of driving storage hardware at its theoretical limit, the benefit is less immediate and offset by the requirement of having a very recent Linux kernel.
  - tyingq 6y ago
    If I understand the premise right, it should be fewer syscalls per IO. So even if it doesn't improve disk I/O, it might reduce CPU utilization.
    - matheusmoreira 6y ago
      It could also be nearly zero system calls per I/O operation. The kernel can poll the submission queue for new entries. This eliminates system call overhead at the cost of higher CPU utilization.
    - jandrewrogers 6y ago
      This is true, but for most intentionally optimized storage engines that syscall overhead is below the noise floor in practice, even on NVMe storage. A single core can easily drive gigabytes per second using the old Linux AIO interface. It appears to primarily be an optimization for storage that was not well-optimized to begin with. It is not obvious that it would make high-performance storage engines faster.
  - shuss 6y ago
    For regular files, aio works async only if they are opened in unbuffered mode. I think this is a huge limitation. io_uring on the other hand, can provide a uniform interface for all file descriptors whether they are sockets or regular files. This should be a decent win, IMO.
    - wbl 6y ago
      And works is used advisedly. Certain filesystem edge conditions particularly metadata changes due to block allocation can result in blocking behavior.
    - jandrewrogers 6y ago
      That was kind of my point. While all of this is true, these are not material limitations for the implementation of high-performance storage engines. For example, using unbuffered file descriptors is a standard design element of databases for performance reasons that remain true. Being able to drive networking over io_uring would be a big advantage but my understanding from people using it is that part is still a bit broken.
- jra_samba 6y ago
  Samba can optionally use it if you explicitly load the vfs_io_uring module, but it exposed a bug for us (see my comment above). We're fixing it right now.
- frevib 6y ago
  Echo server benchmarks, io_uring vs epoll: https://github.com/frevib/io_uring-echo-server/blob/io-uring-op-provide-buffers/benchmarks/benchmarks.md https://github.com/frevib/io_uring-echo-server/blob/io-uring...
  - tele_ski 6y ago
    Nice, reading through the epoll implementation shouldn't it re-register to make sure the send() call won't block? Looks like it only non-blocks on the accept socket and then a single read register
    - frevib 6y ago
      Could elaborate on “re-register”? It does not do short writes, if that’s what you mean.
      - tele_ski 6y ago
        Normally I'd expect an epoll implementation to epoll_ctl to make sure the socket can be written to without blocking. In this benchmark it probably makes no difference but I would think it would make the results a little more inline with a real applications usage of epoll.
        frevib 6y ago
        This a bare minimal echo server for educational purposes. It is not inline with a real world event loop.
        tele_ski 6y ago
        Yeah I understand that, but if you are going for identical performance characteristics of how epoll would normally be used then I would expect it to re-register. Thats all I was getting at.
eMSF 6y ago
A comment from the cat example: >/* For each block of the file we need to read, we allocate an iovec struct which is indexed into the iovecs array. This array is passed in as part of the submission. If you don't understand this, then you need to look up how the readv() and writev() system calls work. */ I have to say, I don't really understand why the author chose to individually allocate (up to millions of) single kilobyte buffers for each file. Perhaps there is a reason for it, but I think they should elaborate the choice. Anyway, I guess the first example is too simplified, which is why what follows after is not built on top of it in any way, hence they feel disjointed. The bigger problem here is that I don't know the author, or how talented they are. Choices like that, or writing non-async-signal-safe signal handlers don't help in estimating it, either. Is the rest of the advice sound?
- shuss 6y ago
  The author here: All examples in the guide are aimed at throwing light at the io_uring and liburing interfaces. They are not very useful or very real-worldish examples. The idea with this example in particular is to show the difference how readv/writev work synchronously vs how they would be "called" io_uring. May be I should call out the fact that these programs are more tuned towards explaining the io_uring interface a lot more in the text. Thanks for the feedback.
throw7 6y ago
The site pushes really hard that you shouldn't use the low-level system calls in your code and that you should (always?) be using a library (liburing). What exactly is liburing bringing to the table that I shouldn't be using the uring syscalls directly?
- matheusmoreira 6y ago
  You absolutely can use system calls in your code. The kernel has an awesome header that makes this easy and allows you to eliminate all dependencies: https://github.com/torvalds/linux/blob/master/tools/include/nolibc/nolibc.h https://github.com/torvalds/linux/blob/master/tools/include/... This system call avoidance dogma exists because libraries generally have more convenient interfaces and are therefore easier to use. They're not strictly necessary though. It should be noted that using certain system calls may cause problems with the libraries you're using. For example, glibc needs to maintain complete control over the threading model in order to implement thread-local storage. By issuing a clone system call directly, the glibc threading model is broken and even something simple like errno is likely to break. In my opinion, libraries shouldn't contain thread-local or global variables in the first place. Unfortunately, the C language is old and these problems will never be fixed. It's possible to create better libraries in freestanding C or even freestanding Rust but replacing what already exists is a lifetime of work. > What exactly is liburing bringing to the table that I shouldn't be using the uring syscalls directly? It's easier to use compared to the kernel interface. For example, it handles submission queue polling automatically without any extra code.
- shuss 6y ago
  The raw io_uring interface, once you ignore the boilerplate initialization code, is actually a super-simple interface to use. liburing is itself only a very thin wrapper on top of io_uring. I feel that if you ever used io_uring, after a while you'll end up with a bunch of convenience functions. liburing looks more like a collection of those functions to me today. One place where a slightly high-level interface is provided by liburing is in the function io_uring_submit(). It determines among other things if there is a need to call the io_uring_enter() system call depending on whether you are in polling mode or not, for example. You can read more about it here: https://unixism.net/loti/tutorial/sq_poll.html https://unixism.net/loti/tutorial/sq_poll.html Otherwise, at least at this time, liburing is a simple wrapper.
qubex 6y ago
Unfortunately I misread the title as “Lord of the Urine” and... was concerned.
matheusmoreira 6y ago
So awesome... The ring buffer is like a generic asynchronous system call submission mechanism. The set of supported operations is already a subset of available Linux system calls: https://github.com/torvalds/linux/blob/master/include/uapi/linux/io_uring.h https://github.com/torvalds/linux/blob/master/include/uapi/l... It almost gained support for ioctl: https://lwn.net/Articles/810414/ https://lwn.net/Articles/810414/ Wouldn't it be cool if it gained support for other types of system calls? Something this awesome shouldn't be restricted to I/O...
- diegocg 6y ago
  The author seems to be planning to expand it to be usable as a generic way of doing asynchronous syscalls
ignoramous 6y ago
Anyone familiar with the Infiniband's approach to exposing IO via rx/tx queues [0] comment whether it seems similar to io_uring's ring-buffers [1]? How do these contrast against each other? [0] https://www.cisco.com/c/en/us/td/docs/server_nw_virtual/2-10-0_release/element_manager/user_guide/appA.html#wp1007799 https://www.cisco.com/c/en/us/td/docs/server_nw_virtual/2-10... [1] https://news.ycombinator.com/item?id=19846261 https://news.ycombinator.com/item?id=19846261
- DmitryOlshansky 6y ago
  Very limited experience with Infiniband but it seems similar, a bit more flexible (esp recently with more syscalls supported). Also similar to but more general than RIO Sockets of Win8+: https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/hh997032(v=ws.11) https://docs.microsoft.com/en-us/previous-versions/windows/i...
rwmj 6y ago
By coincidence I asked a few questions on the mailing list about io_uring this morning: https://lore.kernel.org/io-uring/20200510080034.GI3888@redhat.com/T/#u https://lore.kernel.org/io-uring/20200510080034.GI3888@redha...
jra_samba 6y ago
io_uring still has its wrinkles. We are scrambling right now to fix a problem due to change in behavior exposed to user-space from the io_uring kernel module in later kernels. Turns out that in earlier kernels (Ubuntu 19.04 5.3.0-51-generic #44-Ubuntu SMP) io_uring will not return short reads/writes (that's where you ask for e.g. 8k, but there's only 4k in the buffer cache, so the call doesn't signal as complete and blocks until all 8k has been transferred). In later kernels (not sure when the behavior changed, but the one shipped with Fedora 32 has the new behavior) io_uring returns partial (short) reads to user space. e.g. You ask for 8k but there's only 4k in the buffer cache, so the call signals complete with a return of only 4k read, not the 8k you asked for. Userspace code now has to cope with this where it didn't before. You could argue (and kernel developers did :-) that this was always possible, so user code needs to be aware of this. But it didn't used to do that :-). Change for user space is bad, mkay :-).
- magicalhippo 6y ago
  I know nothing about io_uring but looking at the man page[1] of readv I see it returns number of bytes read. For me as a developer that's an unmistakable flag that partial reads is possible. Was readv changed? The man page also states that partial reads is possible, but I guess that might have been added later? If it always returned bytes read, it would hardly be the first case where the current behavior is mistaken for the specification. My fondest memory of that is all the OpenGL 1.x programs that broke when OpenGL 2.x was released. [1]: http://man7.org/linux/man-pages/man2/readv.2.html http://man7.org/linux/man-pages/man2/readv.2.html
  - jra_samba_org 6y ago
    Well pread/pwrite have the same return values, and historically for disk reads they block or return a device error. pread only returns a short value on EOF.
    - magicalhippo 6y ago
      Well, the man page does say that "The readv() system call works just like read(2) except that multiple buffers are filled". If we go to read(2) we find "It is not an error if [the return value] is smaller than the number of bytes requested; this may happen for example because fewer bytes are actually available right now [...], or because read() was interrupted by a signal." As an outsider, I'd never rely on this returning the requested number of bytes. If I required N bytes, I'd write use a read loop. But I do agree that the RWF_NOWAIT flag mentioned in your other comment doesn't help, as it suggests the default is to block.
beagle3 6y ago
Is there any intention to optimize work done, rather than just the calling interface? E.g., running an rsync if a 10m files hierarchy usually requires 10m synchronous stat calls. Using io-uring would make them asynchronous, but they could potentially be done more efficiently (e.g. convert file names to inodes in blocks of 20k, and then stat those 20k inodes in a batch). That would require e.g. the VFS layer to support batch operations. But the io-uring would actually allow that without a user space interface change.
geofft 6y ago
One thing this writeup made me realize is, if I have a misbehaving I/O system (NFS or remote block device over a flaky network, dying SSD, etc.), in the pre-io_uring world I'd probably see that via /proc/$pid/stack pretty clearly - I'd see a stack with the read syscall, then the particular I/O subsystem, then the physical implementation of that subsystem. Or if I looked at /proc/$pid/syscall I'd see a read call on a certain fd, and I could look in /proc/$pid/fd/ and see which fd it was and where it lived. However, in the post-io_uring world, I think I won't see that, right? If I understand right, I'll at most see a call to io_uring_enter, and maybe not even that. How do I tell what a stuck io_uring-using program is stuck on? Is there a way I can see all the pending I/Os and what's going on with them? How is this implemented internally - does it expand into one kernel thread per I/O, or something? (I guess, if you had a silly filesystem which spent 5 seconds in TASK_UNINTERRUPTIBLE on each read, and you used io_uring to submit 100 reads from it, what actually happens?)
- matheusmoreira 6y ago
  Good point. Would be great if the submission and completion ring buffers were accessible via procfs.
- Matthias247 6y ago
  I think that's a very reasonable concern. It however isn't really about io_uring - it applies to all "async" solutions. Even today if you are running async IO in userspace (e.g. using epoll), it's not very obvious where something went wrong, because no task is seemingly blocked. If you attach a debugger, you might most likely see something being blocked on epoll - but a callstack to the problematic application code is nowhere in sight. Even if pause execution while inside the application code there might not be a great stack which contains all relevant data. It will only contain the information since the last task resumption (e.g. through a callback). Depending on your solution (C callbacks, C++ closures, C# or Kotlin async/await, Rust async/await) the information will be between not very helpful and somewhat understandable, but never on par with a synchronous call.
  - Doxin 6y ago
jcoffland 6y ago
Maybe I just missed this but can anyone tell me what kernel versions support io_uring. I ran the following test program on 4.19.0 and it is not supported: #include <stdio.h> #include <stdlib.h> #include <sys/utsname.h> #include <liburing.h> #include <liburing/io_uring.h> static const char *op_strs[] = { "IORING_OP_NOP", "IORING_OP_READV", "IORING_OP_WRITEV", "IORING_OP_FSYNC", "IORING_OP_READ_FIXED", "IORING_OP_WRITE_FIXED", "IORING_OP_POLL_ADD", "IORING_OP_POLL_REMOVE", "IORING_OP_SYNC_FILE_RANGE", "IORING_OP_SENDMSG", "IORING_OP_RECVMSG", "IORING_OP_TIMEOUT", "IORING_OP_TIMEOUT_REMOVE", "IORING_OP_ACCEPT", "IORING_OP_ASYNC_CANCEL", "IORING_OP_LINK_TIMEOUT", "IORING_OP_CONNECT", "IORING_OP_FALLOCATE", "IORING_OP_OPENAT", "IORING_OP_CLOSE", "IORING_OP_FILES_UPDATE", "IORING_OP_STATX", "IORING_OP_READ", "IORING_OP_WRITE", "IORING_OP_FADVISE", "IORING_OP_MADVISE", "IORING_OP_SEND", "IORING_OP_RECV", "IORING_OP_OPENAT2", "IORING_OP_EPOLL_CTL", "IORING_OP_SPLICE", "IORING_OP_PROVIDE_BUFFERS", "IORING_OP_REMOVE_BUFFERS", }; int main() { struct utsname u; uname(&u); struct io_uring_probe *probe = io_uring_get_probe(); if (!probe) { printf("Kernel %s does not support io_uring.\n", u.release); return 0; } printf("List of kernel %s's supported io_uring operations:\n", u.release); for (int i = 0; i < IORING_OP_LAST; i++ ) { const char *answer = io_uring_opcode_supported(probe, i) ? "yes" : "no"; printf("%s: %s\n", op_strs[i], answer); } free(probe); return 0; }
- shuss 6y ago
  io_uring_get_probe() needs v5.6 at least.
- cesarb 6y ago
  If you have a clone of the Linux kernel source tree, you just have to look at the history of the include/uapi/linux/io_uring.h file. From a quick look here: everything up to IORING_OP_POLL_REMOVE came with Linux 5.1; IORING_OP_SYNC_FILE_RANGE was added in Linux 5.2; IORING_OP_SENDMSG and IORING_OP_RECVMSG came with Linux 5.3; IORING_OP_TIMEOUT with Linux 5.4; everything up to IORING_OP_CONNECT is in Linux 5.5; everything up to IORING_OP_EPOLL_CTL is in Linux 5.6; and the last three are going to be in Linux 5.7.
jmb001nyc 6y ago
Question: how does one detect socket push back using io_uring? For example, with libc "write/writev" for non-blocking socket would return less bytes than requested and allow code to poll for write readiness before writing more. This is quite useful to handle scenarios where there are impedance mismatches between processing speed and ability to send data over a network, e.g. processing needs to observe push back and handle it appropriately. Apologies: I posted this question to twitter before I read the redirect here.

267 points by shuss 6y ago

There are some benchmarks that show io_uring as a significant boost over aio: https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.6-IO-uring-Tests https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.... I see that nginx accepted a pull request to use it, mid last year: https://github.com/hakasenyang/openssl-patch/issues/21 https://github.com/hakasenyang/openssl-patch/issues/21 Curious if it's also been adopted by other popular IO intensive software.

shuss 6y ago
Oh, yeah. QEMU 5.0 already uses io_uring. In fact, it uses liburing. Check out the changelog: https://wiki.qemu.org/ChangeLog/5.0 https://wiki.qemu.org/ChangeLog/5.0
- Twirrim 6y ago
  To save people time, there's a single reference to it on the changelog: > The file-posix driver can now use the io_uring interface of Linux with aio=io_uring side note: I did note a change we built made it in to a released version of qemu: > qemu-img convert -n now understands a --target-is-zero option, which tells it that the target image is completely zero, so it does not need to be zeroed again. That's saving us so much time and I/O
jandrewrogers 6y ago
I have not adopted io_uring yet because it isn't clear that it will provide useful performance improvements over linux aio in cases where the disk I/O subsystem is already highly optimized. Where io_uring seems to show a benefit relative to linux aio is more naive software design, which adds a lot of value but is a somewhat different value proposition than has been expressed. For software that is already capable of driving storage hardware at its theoretical limit, the benefit is less immediate and offset by the requirement of having a very recent Linux kernel.
- tyingq 6y ago
  If I understand the premise right, it should be fewer syscalls per IO. So even if it doesn't improve disk I/O, it might reduce CPU utilization.
  - matheusmoreira 6y ago
    It could also be nearly zero system calls per I/O operation. The kernel can poll the submission queue for new entries. This eliminates system call overhead at the cost of higher CPU utilization.
  - jandrewrogers 6y ago
    This is true, but for most intentionally optimized storage engines that syscall overhead is below the noise floor in practice, even on NVMe storage. A single core can easily drive gigabytes per second using the old Linux AIO interface. It appears to primarily be an optimization for storage that was not well-optimized to begin with. It is not obvious that it would make high-performance storage engines faster.
- shuss 6y ago
  For regular files, aio works async only if they are opened in unbuffered mode. I think this is a huge limitation. io_uring on the other hand, can provide a uniform interface for all file descriptors whether they are sockets or regular files. This should be a decent win, IMO.
  - wbl 6y ago
    And works is used advisedly. Certain filesystem edge conditions particularly metadata changes due to block allocation can result in blocking behavior.
  - jandrewrogers 6y ago
    That was kind of my point. While all of this is true, these are not material limitations for the implementation of high-performance storage engines. For example, using unbuffered file descriptors is a standard design element of databases for performance reasons that remain true. Being able to drive networking over io_uring would be a big advantage but my understanding from people using it is that part is still a bit broken.
jra_samba 6y ago
Samba can optionally use it if you explicitly load the vfs_io_uring module, but it exposed a bug for us (see my comment above). We're fixing it right now.
frevib 6y ago
Echo server benchmarks, io_uring vs epoll: https://github.com/frevib/io_uring-echo-server/blob/io-uring-op-provide-buffers/benchmarks/benchmarks.md https://github.com/frevib/io_uring-echo-server/blob/io-uring...
- tele_ski 6y ago
  Nice, reading through the epoll implementation shouldn't it re-register to make sure the send() call won't block? Looks like it only non-blocks on the accept socket and then a single read register
  - frevib 6y ago
    Could elaborate on “re-register”? It does not do short writes, if that’s what you mean.
    - tele_ski 6y ago
      Normally I'd expect an epoll implementation to epoll_ctl to make sure the socket can be written to without blocking. In this benchmark it probably makes no difference but I would think it would make the results a little more inline with a real applications usage of epoll.
      - frevib 6y ago
        This a bare minimal echo server for educational purposes. It is not inline with a real world event loop.
        tele_ski 6y ago
        Yeah I understand that, but if you are going for identical performance characteristics of how epoll would normally be used then I would expect it to re-register. Thats all I was getting at.

eMSF 6y ago

A comment from the cat example: >/* For each block of the file we need to read, we allocate an iovec struct which is indexed into the iovecs array. This array is passed in as part of the submission. If you don't understand this, then you need to look up how the readv() and writev() system calls work. */ I have to say, I don't really understand why the author chose to individually allocate (up to millions of) single kilobyte buffers for each file. Perhaps there is a reason for it, but I think they should elaborate the choice. Anyway, I guess the first example is too simplified, which is why what follows after is not built on top of it in any way, hence they feel disjointed. The bigger problem here is that I don't know the author, or how talented they are. Choices like that, or writing non-async-signal-safe signal handlers don't help in estimating it, either. Is the rest of the advice sound?

shuss 6y ago
The author here: All examples in the guide are aimed at throwing light at the io_uring and liburing interfaces. They are not very useful or very real-worldish examples. The idea with this example in particular is to show the difference how readv/writev work synchronously vs how they would be "called" io_uring. May be I should call out the fact that these programs are more tuned towards explaining the io_uring interface a lot more in the text. Thanks for the feedback.

throw7 6y ago

The site pushes really hard that you shouldn't use the low-level system calls in your code and that you should (always?) be using a library (liburing). What exactly is liburing bringing to the table that I shouldn't be using the uring syscalls directly?

matheusmoreira 6y ago
You absolutely can use system calls in your code. The kernel has an awesome header that makes this easy and allows you to eliminate all dependencies: https://github.com/torvalds/linux/blob/master/tools/include/nolibc/nolibc.h https://github.com/torvalds/linux/blob/master/tools/include/... This system call avoidance dogma exists because libraries generally have more convenient interfaces and are therefore easier to use. They're not strictly necessary though. It should be noted that using certain system calls may cause problems with the libraries you're using. For example, glibc needs to maintain complete control over the threading model in order to implement thread-local storage. By issuing a clone system call directly, the glibc threading model is broken and even something simple like errno is likely to break. In my opinion, libraries shouldn't contain thread-local or global variables in the first place. Unfortunately, the C language is old and these problems will never be fixed. It's possible to create better libraries in freestanding C or even freestanding Rust but replacing what already exists is a lifetime of work. > What exactly is liburing bringing to the table that I shouldn't be using the uring syscalls directly? It's easier to use compared to the kernel interface. For example, it handles submission queue polling automatically without any extra code.
shuss 6y ago
The raw io_uring interface, once you ignore the boilerplate initialization code, is actually a super-simple interface to use. liburing is itself only a very thin wrapper on top of io_uring. I feel that if you ever used io_uring, after a while you'll end up with a bunch of convenience functions. liburing looks more like a collection of those functions to me today. One place where a slightly high-level interface is provided by liburing is in the function io_uring_submit(). It determines among other things if there is a need to call the io_uring_enter() system call depending on whether you are in polling mode or not, for example. You can read more about it here: https://unixism.net/loti/tutorial/sq_poll.html https://unixism.net/loti/tutorial/sq_poll.html Otherwise, at least at this time, liburing is a simple wrapper.

qubex 6y ago

Unfortunately I misread the title as “Lord of the Urine” and... was concerned.

matheusmoreira 6y ago

So awesome... The ring buffer is like a generic asynchronous system call submission mechanism. The set of supported operations is already a subset of available Linux system calls: https://github.com/torvalds/linux/blob/master/include/uapi/linux/io_uring.h https://github.com/torvalds/linux/blob/master/include/uapi/l... It almost gained support for ioctl: https://lwn.net/Articles/810414/ https://lwn.net/Articles/810414/ Wouldn't it be cool if it gained support for other types of system calls? Something this awesome shouldn't be restricted to I/O...

diegocg 6y ago
The author seems to be planning to expand it to be usable as a generic way of doing asynchronous syscalls

ignoramous 6y ago

Anyone familiar with the Infiniband's approach to exposing IO via rx/tx queues [0] comment whether it seems similar to io_uring's ring-buffers [1]? How do these contrast against each other? [0] https://www.cisco.com/c/en/us/td/docs/server_nw_virtual/2-10-0_release/element_manager/user_guide/appA.html#wp1007799 https://www.cisco.com/c/en/us/td/docs/server_nw_virtual/2-10... [1] https://news.ycombinator.com/item?id=19846261 https://news.ycombinator.com/item?id=19846261

DmitryOlshansky 6y ago
Very limited experience with Infiniband but it seems similar, a bit more flexible (esp recently with more syscalls supported). Also similar to but more general than RIO Sockets of Win8+: https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/hh997032(v=ws.11) https://docs.microsoft.com/en-us/previous-versions/windows/i...

rwmj 6y ago

By coincidence I asked a few questions on the mailing list about io_uring this morning: https://lore.kernel.org/io-uring/20200510080034.GI3888@redhat.com/T/#u https://lore.kernel.org/io-uring/20200510080034.GI3888@redha...

jra_samba 6y ago

io_uring still has its wrinkles. We are scrambling right now to fix a problem due to change in behavior exposed to user-space from the io_uring kernel module in later kernels. Turns out that in earlier kernels (Ubuntu 19.04 5.3.0-51-generic #44-Ubuntu SMP) io_uring will not return short reads/writes (that's where you ask for e.g. 8k, but there's only 4k in the buffer cache, so the call doesn't signal as complete and blocks until all 8k has been transferred). In later kernels (not sure when the behavior changed, but the one shipped with Fedora 32 has the new behavior) io_uring returns partial (short) reads to user space. e.g. You ask for 8k but there's only 4k in the buffer cache, so the call signals complete with a return of only 4k read, not the 8k you asked for. Userspace code now has to cope with this where it didn't before. You could argue (and kernel developers did :-) that this was always possible, so user code needs to be aware of this. But it didn't used to do that :-). Change for user space is bad, mkay :-).

magicalhippo 6y ago
I know nothing about io_uring but looking at the man page[1] of readv I see it returns number of bytes read. For me as a developer that's an unmistakable flag that partial reads is possible. Was readv changed? The man page also states that partial reads is possible, but I guess that might have been added later? If it always returned bytes read, it would hardly be the first case where the current behavior is mistaken for the specification. My fondest memory of that is all the OpenGL 1.x programs that broke when OpenGL 2.x was released. [1]: http://man7.org/linux/man-pages/man2/readv.2.html http://man7.org/linux/man-pages/man2/readv.2.html
- jra_samba_org 6y ago
  Well pread/pwrite have the same return values, and historically for disk reads they block or return a device error. pread only returns a short value on EOF.
  - magicalhippo 6y ago
    Well, the man page does say that "The readv() system call works just like read(2) except that multiple buffers are filled". If we go to read(2) we find "It is not an error if [the return value] is smaller than the number of bytes requested; this may happen for example because fewer bytes are actually available right now [...], or because read() was interrupted by a signal." As an outsider, I'd never rely on this returning the requested number of bytes. If I required N bytes, I'd write use a read loop. But I do agree that the RWF_NOWAIT flag mentioned in your other comment doesn't help, as it suggests the default is to block.

beagle3 6y ago

Is there any intention to optimize work done, rather than just the calling interface? E.g., running an rsync if a 10m files hierarchy usually requires 10m synchronous stat calls. Using io-uring would make them asynchronous, but they could potentially be done more efficiently (e.g. convert file names to inodes in blocks of 20k, and then stat those 20k inodes in a batch). That would require e.g. the VFS layer to support batch operations. But the io-uring would actually allow that without a user space interface change.

geofft 6y ago

One thing this writeup made me realize is, if I have a misbehaving I/O system (NFS or remote block device over a flaky network, dying SSD, etc.), in the pre-io_uring world I'd probably see that via /proc/$pid/stack pretty clearly - I'd see a stack with the read syscall, then the particular I/O subsystem, then the physical implementation of that subsystem. Or if I looked at /proc/$pid/syscall I'd see a read call on a certain fd, and I could look in /proc/$pid/fd/ and see which fd it was and where it lived. However, in the post-io_uring world, I think I won't see that, right? If I understand right, I'll at most see a call to io_uring_enter, and maybe not even that. How do I tell what a stuck io_uring-using program is stuck on? Is there a way I can see all the pending I/Os and what's going on with them? How is this implemented internally - does it expand into one kernel thread per I/O, or something? (I guess, if you had a silly filesystem which spent 5 seconds in TASK_UNINTERRUPTIBLE on each read, and you used io_uring to submit 100 reads from it, what actually happens?)

matheusmoreira 6y ago
Good point. Would be great if the submission and completion ring buffers were accessible via procfs.
Matthias247 6y ago
I think that's a very reasonable concern. It however isn't really about io_uring - it applies to all "async" solutions. Even today if you are running async IO in userspace (e.g. using epoll), it's not very obvious where something went wrong, because no task is seemingly blocked. If you attach a debugger, you might most likely see something being blocked on epoll - but a callstack to the problematic application code is nowhere in sight. Even if pause execution while inside the application code there might not be a great stack which contains all relevant data. It will only contain the information since the last task resumption (e.g. through a callback). Depending on your solution (C callbacks, C++ closures, C# or Kotlin async/await, Rust async/await) the information will be between not very helpful and somewhat understandable, but never on par with a synchronous call.
- Doxin 6y ago

jcoffland 6y ago

Maybe I just missed this but can anyone tell me what kernel versions support io_uring. I ran the following test program on 4.19.0 and it is not supported: #include <stdio.h> #include <stdlib.h> #include <sys/utsname.h> #include <liburing.h> #include <liburing/io_uring.h> static const char *op_strs[] = { "IORING_OP_NOP", "IORING_OP_READV", "IORING_OP_WRITEV", "IORING_OP_FSYNC", "IORING_OP_READ_FIXED", "IORING_OP_WRITE_FIXED", "IORING_OP_POLL_ADD", "IORING_OP_POLL_REMOVE", "IORING_OP_SYNC_FILE_RANGE", "IORING_OP_SENDMSG", "IORING_OP_RECVMSG", "IORING_OP_TIMEOUT", "IORING_OP_TIMEOUT_REMOVE", "IORING_OP_ACCEPT", "IORING_OP_ASYNC_CANCEL", "IORING_OP_LINK_TIMEOUT", "IORING_OP_CONNECT", "IORING_OP_FALLOCATE", "IORING_OP_OPENAT", "IORING_OP_CLOSE", "IORING_OP_FILES_UPDATE", "IORING_OP_STATX", "IORING_OP_READ", "IORING_OP_WRITE", "IORING_OP_FADVISE", "IORING_OP_MADVISE", "IORING_OP_SEND", "IORING_OP_RECV", "IORING_OP_OPENAT2", "IORING_OP_EPOLL_CTL", "IORING_OP_SPLICE", "IORING_OP_PROVIDE_BUFFERS", "IORING_OP_REMOVE_BUFFERS", }; int main() { struct utsname u; uname(&u); struct io_uring_probe *probe = io_uring_get_probe(); if (!probe) { printf("Kernel %s does not support io_uring.\n", u.release); return 0; } printf("List of kernel %s's supported io_uring operations:\n", u.release); for (int i = 0; i < IORING_OP_LAST; i++ ) { const char *answer = io_uring_opcode_supported(probe, i) ? "yes" : "no"; printf("%s: %s\n", op_strs[i], answer); } free(probe); return 0; }

shuss 6y ago
io_uring_get_probe() needs v5.6 at least.
cesarb 6y ago
If you have a clone of the Linux kernel source tree, you just have to look at the history of the include/uapi/linux/io_uring.h file. From a quick look here: everything up to IORING_OP_POLL_REMOVE came with Linux 5.1; IORING_OP_SYNC_FILE_RANGE was added in Linux 5.2; IORING_OP_SENDMSG and IORING_OP_RECVMSG came with Linux 5.3; IORING_OP_TIMEOUT with Linux 5.4; everything up to IORING_OP_CONNECT is in Linux 5.5; everything up to IORING_OP_EPOLL_CTL is in Linux 5.6; and the last three are going to be in Linux 5.7.

jmb001nyc 6y ago

Question: how does one detect socket push back using io_uring? For example, with libc "write/writev" for non-blocking socket would return less bytes than requested and allow code to poll for write readiness before writing more. This is quite useful to handle scenarios where there are impedance mismatches between processing speed and ability to send data over a network, e.g. processing needs to observe push back and handle it appropriately. Apologies: I posted this question to twitter before I read the redirect here.