6 ms·

Efficient IO in Linux with io_uring [pdf] (2019)

151 points by gurjeet 4y ago

acquacow 4y ago
Jens Axboe has been working on this for a long time. I'll stand behind anything he works on =)
ephaeton 4y ago
needs a [2019]
v3ss0n 4y ago
Needs asynchronous libs to adopt it
- yxhuvud 4y ago
  Which in turn need that Linuxes that they support also support io_uring. This world turns slowly.
  - v3ss0n 4y ago
    It's since very long ago right? Most major server distro should have it by now
    - yxhuvud 4y ago
      It is also improving a lot over time. What little there was support for back in Linux 5.3 is not sufficient.
- ioquatix 4y ago
  It's adopted by Ruby: <https://github.com/socketry/io-event https://github.com/socketry/io-event> which is used by <https://github.com/socketry/async https://github.com/socketry/async> which is part of the Ruby 3+ Fiber Scheduler for light weight concurrency. It shows promising performance.
  - v3ss0n 4y ago
    Interesting, but I read some issues with Benchmade showing it's not actually reproducible regarding performance improvement over epoll and epoll still faster.
- Zababa 4y ago
  It's used by Eio in OCaml for the Linux backend: https://github.com/ocaml-multicore/eio https://github.com/ocaml-multicore/eio.
  - v3ss0n 4y ago
    thanks , shame to see pull request waiting to be merged. Would be nice if python asyncio adopt it too.
- rektide 4y ago
  There was some very excellent work done in Node.js's libuv, but it never got across the line & has sat around getting more out of date for a while. https://github.com/libuv/libuv/issues/1947#issuecomment-485230126 https://github.com/libuv/libuv/issues/1947#issuecomment-4852...
  - v3ss0n 4y ago
    4 years already..
cash22 4y ago
On page 5: > To find the index of an event, the application must mask the current tail index with the size mask of the ring. This commonly looks something like the below: unsigned head; head = cqring→head; read_barrier(); if (head != cqring→tail) { struct io_uring_cqe *cqe; unsigned index; index = head & (cqring→mask); cqe = &cqring→cqes[index]; /* process completed cqe here */ ... /* we've now consumed this entry */ head++; } cqring→head = head; write_barrier(); Am I misunderstanding, or is "current tail index" supposed to be "current head index?"
kazinator 4y ago
I did this years before io_uring, circa 2006. Working on a Linux-based networking startup called Zeugma Systems, I implemented a kernel-based logging system for a multi-process, multi-node distributed application. I started on Linux 2.6.14. When that startup folded, I think we were on 2.6.27. The logging system was implemented in a module which was inserted into the kernel and then used the calling thread to run a service. The module provided a device and some ioctls. Processes used the ioctls to attach circular buffers to the device, which was mapped into kernel space using get_user_pages. Processes would just place messages into their circular buffers and update an index variable in the buffer header. The kernel would automatically pick up the message, without any system call. There was a wakeup ioctl to poke the kernel thread, which was used upon hitting a high water mark (buffer getting near full). This is the basic intuition behind io_uring. The kernel thread collected messages from multiple buffers and sent them into several destination (files and sockets). I do not have most of this code, but some of it survived, including a kernel mutex and condition variable library featuring a function that lets you give up a mutex to wait on a condition variable, while also polling any mixture of kernel file and socket handles, with a timeout. This function at the core of the kernel thread's loop. The nice thing was that when processes crashed, their buffer would not go away immediately. Of course their own address space would be gone, but the kernel's mapping of the shared buffer mapping would gracefully persist, until the thread emptied the buffer; everything put into the buffer before the crash was safe. Empty buffers belonging to processes that had died would then be cleaned away. I had a utility program that would list the buffers and the PIDs of their processes, and provide stats, like outstanding bytes, and is that process still alive. (The one inefficiency in logging was that log messages need time stamps. Depending on where you get a time stamp from, that requires a trip to the kernel. I can't remember what I did about that.) A bit of a difficulty in the whole approach is that I wasn't getting a linear mapping of the user pages in the kernel. So I wrote the grotty C code (kernel side) to pull the messages correctly from a scrambled buffer whose pages are out of order, without making an extra copy.
- moonchild 4y ago
  And I thought I was original for coming up with this scheme after I heard about vdsos, a year or two before io_uring became a thing :)
minraws 4y ago
For people who are curious before/after reading it, this is from 2019, there are updates the io_uring interface in Linux, quite a few actually, here's a changelog of liburing kind of the defacto library for io_uring. https://github.com/axboe/liburing/compare/liburing-0.4...liburing-2.2 https://github.com/axboe/liburing/compare/liburing-0.4...lib... Actually, io_uring has received around 500 commits just in 2022. Here's a list of interesting feature additions(this year), - https://www.phoronix.com/news/Linux-LPC2022-io_uring_spawn https://www.phoronix.com/news/Linux-LPC2022-io_uring_spawn - https://www.phoronix.com/news/Linux-520-XFS-uring-Async-Buff https://www.phoronix.com/news/Linux-520-XFS-uring-Async-Buff - https://www.phoronix.com/news/Linux-5.20-IO_uring-ZC-Send https://www.phoronix.com/news/Linux-5.20-IO_uring-ZC-Send - https://www.phoronix.com/news/Linux-5.19-IO_uring https://www.phoronix.com/news/Linux-5.19-IO_uring io_uring like API was also adopted by Windows recently. Lots of fun.
- dijit 4y ago
  That last line is interesting to me, one of the (admittedly few) things Windows had over Linux when it comes to being able to write extremely high performant software in an easy way was IOCP (IO completion ports), epoll on Linux was awkward to use and hard to get right. I’ve never managed to do a comparison of io_uring vs IOCP but I guess there was some benefit here, I left that job before io_uring was mature. Would anyone be willing to share a comparison of the two systems?
  - tripwanger 4y ago
    I think that highlights one of the issues with Linux being essentially just another clone of Unix, and thus being constrained by its limitations - both technically and in mindset of its developers. Whereas NT was a fresh new design, by seasoned operating systems designers and implementers, who had seen the shortcomings of the Unix model (and others) and not only knew what to do differently to improve on these, but actually wanted to and were permitted to as well.
    - speed_spread 4y ago
      Windows NT was not a fresh new design but a direct evolution of VMS, which had it's own history independent of Unix. That it steered clear of Unix's shortcomings is a result of a design process that's much less napkin-based and more "engineered" for better and worse.
deleted 4y ago
[deleted]
Sirened 4y ago
just need some conditionals and branch operations and then we'll be able to write full software in io_uring :P
espoal 4y ago
glad io_uring is gaining traction. If you plan on using it, or just want to learn more about it, check this out: https://github.com/espoal/awesome-iouring https://github.com/espoal/awesome-iouring
- russdill 4y ago
  Hopefully in a few years most people will be using it and have no idea due to framework adoption

151 points by gurjeet 4y ago

acquacow 4y ago

Jens Axboe has been working on this for a long time. I'll stand behind anything he works on =)

ephaeton 4y ago

needs a [2019]

v3ss0n 4y ago

Needs asynchronous libs to adopt it

yxhuvud 4y ago
Which in turn need that Linuxes that they support also support io_uring. This world turns slowly.
- v3ss0n 4y ago
  It's since very long ago right? Most major server distro should have it by now
  - yxhuvud 4y ago
    It is also improving a lot over time. What little there was support for back in Linux 5.3 is not sufficient.
ioquatix 4y ago
It's adopted by Ruby: <https://github.com/socketry/io-event https://github.com/socketry/io-event> which is used by <https://github.com/socketry/async https://github.com/socketry/async> which is part of the Ruby 3+ Fiber Scheduler for light weight concurrency. It shows promising performance.
- v3ss0n 4y ago
  Interesting, but I read some issues with Benchmade showing it's not actually reproducible regarding performance improvement over epoll and epoll still faster.
Zababa 4y ago
It's used by Eio in OCaml for the Linux backend: https://github.com/ocaml-multicore/eio https://github.com/ocaml-multicore/eio.
- v3ss0n 4y ago
  thanks , shame to see pull request waiting to be merged. Would be nice if python asyncio adopt it too.
rektide 4y ago
There was some very excellent work done in Node.js's libuv, but it never got across the line & has sat around getting more out of date for a while. https://github.com/libuv/libuv/issues/1947#issuecomment-485230126 https://github.com/libuv/libuv/issues/1947#issuecomment-4852...
- v3ss0n 4y ago
  4 years already..

cash22 4y ago

On page 5: > To find the index of an event, the application must mask the current tail index with the size mask of the ring. This commonly looks something like the below: unsigned head; head = cqring→head; read_barrier(); if (head != cqring→tail) { struct io_uring_cqe *cqe; unsigned index; index = head & (cqring→mask); cqe = &cqring→cqes[index]; /* process completed cqe here */ ... /* we've now consumed this entry */ head++; } cqring→head = head; write_barrier(); Am I misunderstanding, or is "current tail index" supposed to be "current head index?"

kazinator 4y ago

I did this years before io_uring, circa 2006. Working on a Linux-based networking startup called Zeugma Systems, I implemented a kernel-based logging system for a multi-process, multi-node distributed application. I started on Linux 2.6.14. When that startup folded, I think we were on 2.6.27. The logging system was implemented in a module which was inserted into the kernel and then used the calling thread to run a service. The module provided a device and some ioctls. Processes used the ioctls to attach circular buffers to the device, which was mapped into kernel space using get_user_pages. Processes would just place messages into their circular buffers and update an index variable in the buffer header. The kernel would automatically pick up the message, without any system call. There was a wakeup ioctl to poke the kernel thread, which was used upon hitting a high water mark (buffer getting near full). This is the basic intuition behind io_uring. The kernel thread collected messages from multiple buffers and sent them into several destination (files and sockets). I do not have most of this code, but some of it survived, including a kernel mutex and condition variable library featuring a function that lets you give up a mutex to wait on a condition variable, while also polling any mixture of kernel file and socket handles, with a timeout. This function at the core of the kernel thread's loop. The nice thing was that when processes crashed, their buffer would not go away immediately. Of course their own address space would be gone, but the kernel's mapping of the shared buffer mapping would gracefully persist, until the thread emptied the buffer; everything put into the buffer before the crash was safe. Empty buffers belonging to processes that had died would then be cleaned away. I had a utility program that would list the buffers and the PIDs of their processes, and provide stats, like outstanding bytes, and is that process still alive. (The one inefficiency in logging was that log messages need time stamps. Depending on where you get a time stamp from, that requires a trip to the kernel. I can't remember what I did about that.) A bit of a difficulty in the whole approach is that I wasn't getting a linear mapping of the user pages in the kernel. So I wrote the grotty C code (kernel side) to pull the messages correctly from a scrambled buffer whose pages are out of order, without making an extra copy.

moonchild 4y ago
And I thought I was original for coming up with this scheme after I heard about vdsos, a year or two before io_uring became a thing :)

minraws 4y ago

For people who are curious before/after reading it, this is from 2019, there are updates the io_uring interface in Linux, quite a few actually, here's a changelog of liburing kind of the defacto library for io_uring. https://github.com/axboe/liburing/compare/liburing-0.4...liburing-2.2 https://github.com/axboe/liburing/compare/liburing-0.4...lib... Actually, io_uring has received around 500 commits just in 2022. Here's a list of interesting feature additions(this year), - https://www.phoronix.com/news/Linux-LPC2022-io_uring_spawn https://www.phoronix.com/news/Linux-LPC2022-io_uring_spawn - https://www.phoronix.com/news/Linux-520-XFS-uring-Async-Buff https://www.phoronix.com/news/Linux-520-XFS-uring-Async-Buff - https://www.phoronix.com/news/Linux-5.20-IO_uring-ZC-Send https://www.phoronix.com/news/Linux-5.20-IO_uring-ZC-Send - https://www.phoronix.com/news/Linux-5.19-IO_uring https://www.phoronix.com/news/Linux-5.19-IO_uring io_uring like API was also adopted by Windows recently. Lots of fun.

dijit 4y ago
That last line is interesting to me, one of the (admittedly few) things Windows had over Linux when it comes to being able to write extremely high performant software in an easy way was IOCP (IO completion ports), epoll on Linux was awkward to use and hard to get right. I’ve never managed to do a comparison of io_uring vs IOCP but I guess there was some benefit here, I left that job before io_uring was mature. Would anyone be willing to share a comparison of the two systems?
- tripwanger 4y ago
  I think that highlights one of the issues with Linux being essentially just another clone of Unix, and thus being constrained by its limitations - both technically and in mindset of its developers. Whereas NT was a fresh new design, by seasoned operating systems designers and implementers, who had seen the shortcomings of the Unix model (and others) and not only knew what to do differently to improve on these, but actually wanted to and were permitted to as well.
  - speed_spread 4y ago
    Windows NT was not a fresh new design but a direct evolution of VMS, which had it's own history independent of Unix. That it steered clear of Unix's shortcomings is a result of a design process that's much less napkin-based and more "engineered" for better and worse.

deleted 4y ago

[deleted]

Sirened 4y ago

just need some conditionals and branch operations and then we'll be able to write full software in io_uring :P

espoal 4y ago

glad io_uring is gaining traction. If you plan on using it, or just want to learn more about it, check this out: https://github.com/espoal/awesome-iouring https://github.com/espoal/awesome-iouring

russdill 4y ago
Hopefully in a few years most people will be using it and have no idea due to framework adoption