4 ms·

Love it. I'm trying to understand why all command line tools don't use io_uring. As an example, all my nvme's on usb 3.2 gen 2 only reach 740MB/s peak. If I

Love it. I'm trying to understand why all command line tools don't use io_uring. As an example, all my nvme's on usb 3.2 gen 2 only reach 740MB/s peak. If I use tools with aio or io_uring I get 1005MB/s. I know I may not be copying many files simultaneously every time, but the queue length strategies and the fewer locks also help I guess.

superkuh 1y ago
One reason is so that they work in all linux environments rather than just bleeding edge installs from the last couple years.
tyingq 1y ago
Probably historical preference for portability without a bunch of #ifdef means platform+version-specific stuff is very late to get adopted. Though, at this point, the benefit of portability across various posixy platforms is much lower.
- Retr0id 1y ago
  Has anyone written an io_uring "polyfill" library with fallback to standard posix-y IO? It could presumably be done via background worker threads - at a perf cost.
  - vlovich123 1y ago
    Seems like a huge lift since io_uring is an ever growing set of interfaces that is encompassing more and more of the kernel surface area. Also, the problem tends to not necessarily be that the io_uring interface isn’t available at compile time but a) the version you distribute to has a kernel with it disabled or you don’t have permission to use it meaning you need to do LD_preload magic or use a framework b) the kernel you’re using supports some of the interfaces you’re trying to use but not all. Not sure how you solve that one without using a framework. But I agree. It would be cool if it was transparent, but this is actually what a bunch of io-uring runtimes do, using epoll as a fallback (eg in Rust monoio)
    - namibj 1y ago
      You can just ask io_uring what commands you have available to you. Though the way of the background thread should be readily available/usable by just indirectly calling the syscall (-helper) and replacing it with a futex-based handshake/wrapper. If you're not using the backend-ordering-imposing link bit, you could probably even use minor futex trickery to dispatch multiple background threads to snatch up from the submission queue in a "grab one at a time" fashion.
never_inline 1y ago
Poe's law hits again.
elcapitan 1y ago
iirc io_uring also had some pretty significant security issues early on (a couple of years ago). Those should be fixed by now, but that probably dampened adoption as well.
- jeffbee 1y ago
  Not years ago. io_uring has been a continuous parade of security problems, including a high severity one that wasn't fixed until a few months ago. Many large organizations have patched it out of their kernels on safety basis, which is one of the reasons it suffers from poor adoption.
- raesene9 1y ago
  Last I checked it's blocked by most container runtimes exactly because of the security problems, and Google blocked io_uring across all their services. I've not checked recently if that's still the case, but https://security.googleblog.com/2023/06/learnings-from-kctf-vrps-42-linux.html https://security.googleblog.com/2023/06/learnings-from-kctf-... has some background.
tln 1y ago
Thats a great speed boost. What tools are these?
Thaxll 1y ago
io_uring is a security nightmare.
- pjc50 1y ago
  How so?
  - Thaxll 1y ago
    This is a good read on the topic: https://chomp.ie/Blog+Posts/Put+an+io_uring+on+it+-+Exploiting+the+Linux+Kernel https://chomp.ie/Blog+Posts/Put+an+io_uring+on+it+-+Exploiti...
  - sim7c00 1y ago
    you give process direct access to a piece of kernel memory. its a reason why there is separation. thats all.
    - wtallis 1y ago
      Most of the security concerns with io_uring that I've seen aren't related to the shared buffers at all but simply stem from the fact that io_uring is a mechanism to instruct the kernel to do stuff without making system calls, so security measures that focus on what system calls a process is allowed to do are ineffective.
    - loeg 1y ago
      This isn't the issue; it's relatively easy to safely share some ring buffers. The issue was/is that io_uring is rapidly growing the equivalent of ~all historical Linux syscall interfaces and sometimes comparable security measures were missed on the new interfaces. (Also, stuff like seccomp filters on syscalls are kind of meaningless for io_uring.)
    - duped 1y ago
      ...don't you supply the memory in the submission queue? or do you mean the queues themselves?
fpoling 1y ago
io_uring is the asynchronous interface and that requires to use even-based architecture to use it effectively. But many command-line tools are still written is a straightforward sequential style. If C would have async or similar mechanism to pretend doing async programming sequentially, it would be easier to port. But without that a very significant refactoring is necessary. Besides, io_uring is not yet stable and who knows may be in 10 years it will be replaced by yet another mechanism to take advantage of even newer hardware. So simply waiting for io_uring prove it is here to stay is very viable strategy. Besides in 10 years we may have tools/AI that will do the rewrite automatically...
- mananaysiempre 1y ago
  > If C would have async or similar mechanism to pretend doing async programming sequentially, it would be easier to port. The *context() family of formerly-POSIX functions (clownishly deprecated as “use pthreads instead”) is essentially a full implementation of stackful coroutines. Even the arguable design botch of them preserving the signal mask (the reason why they aren’t the go-to option even on Linux) is theoretically fixable on the libc level without system calls, it’s just a lot of work and very few can be bothered to do signals well. As far as stackless coroutines, there’s a wide variety of libraries used in embedded systems and such (see the recent discussion[1] for some links), which are by necessity awkward enough that I don’t see any of them becoming broadly accepted. There were also a number of language extensions, among which I’d single out AC[2] (from the Barrelfish project) and CPC[3]. I’d love for, say, CPC to catch on, but it’s been over a decade now. [1] https://news.ycombinator.com/item?id=44546640 https://news.ycombinator.com/item?id=44546640 [2] https://users.soe.ucsc.edu/~abadi/Papers/acasync.pdf https://users.soe.ucsc.edu/~abadi/Papers/acasync.pdf [3] https://www.irif.fr/~jch/research/cpc-2012.pdf https://www.irif.fr/~jch/research/cpc-2012.pdf
- ThePallas 1y ago
  As an example of pretending to do exactly that, this project (author: me) combines io_uring and ucontext.. https://github.com/pallas/ioucontext/ https://github.com/pallas/ioucontext/
cesarb 1y ago
> I'm trying to understand why all command line tools don't use io_uring. Because it's fairly new. The coreutils package which contains the ls command (and the three earlier packages which were merged to create it) is decades old; io_uring appeared much later. It will take time for the "shared ring buffer" style of system call to win over traditional synchronous system calls.
Agingcoder 1y ago
Iouring is very recent

superkuh 1y ago

One reason is so that they work in all linux environments rather than just bleeding edge installs from the last couple years.

tyingq 1y ago

Probably historical preference for portability without a bunch of #ifdef means platform+version-specific stuff is very late to get adopted. Though, at this point, the benefit of portability across various posixy platforms is much lower.

Retr0id 1y ago
Has anyone written an io_uring "polyfill" library with fallback to standard posix-y IO? It could presumably be done via background worker threads - at a perf cost.
- vlovich123 1y ago
  Seems like a huge lift since io_uring is an ever growing set of interfaces that is encompassing more and more of the kernel surface area. Also, the problem tends to not necessarily be that the io_uring interface isn’t available at compile time but a) the version you distribute to has a kernel with it disabled or you don’t have permission to use it meaning you need to do LD_preload magic or use a framework b) the kernel you’re using supports some of the interfaces you’re trying to use but not all. Not sure how you solve that one without using a framework. But I agree. It would be cool if it was transparent, but this is actually what a bunch of io-uring runtimes do, using epoll as a fallback (eg in Rust monoio)
  - namibj 1y ago
    You can just ask io_uring what commands you have available to you. Though the way of the background thread should be readily available/usable by just indirectly calling the syscall (-helper) and replacing it with a futex-based handshake/wrapper. If you're not using the backend-ordering-imposing link bit, you could probably even use minor futex trickery to dispatch multiple background threads to snatch up from the submission queue in a "grab one at a time" fashion.

never_inline 1y ago

Poe's law hits again.

elcapitan 1y ago

iirc io_uring also had some pretty significant security issues early on (a couple of years ago). Those should be fixed by now, but that probably dampened adoption as well.

jeffbee 1y ago
Not years ago. io_uring has been a continuous parade of security problems, including a high severity one that wasn't fixed until a few months ago. Many large organizations have patched it out of their kernels on safety basis, which is one of the reasons it suffers from poor adoption.
raesene9 1y ago
Last I checked it's blocked by most container runtimes exactly because of the security problems, and Google blocked io_uring across all their services. I've not checked recently if that's still the case, but https://security.googleblog.com/2023/06/learnings-from-kctf-vrps-42-linux.html https://security.googleblog.com/2023/06/learnings-from-kctf-... has some background.

tln 1y ago

Thats a great speed boost. What tools are these?

Thaxll 1y ago

io_uring is a security nightmare.

pjc50 1y ago
How so?
- Thaxll 1y ago
  This is a good read on the topic: https://chomp.ie/Blog+Posts/Put+an+io_uring+on+it+-+Exploiting+the+Linux+Kernel https://chomp.ie/Blog+Posts/Put+an+io_uring+on+it+-+Exploiti...
- sim7c00 1y ago
  you give process direct access to a piece of kernel memory. its a reason why there is separation. thats all.
  - wtallis 1y ago
    Most of the security concerns with io_uring that I've seen aren't related to the shared buffers at all but simply stem from the fact that io_uring is a mechanism to instruct the kernel to do stuff without making system calls, so security measures that focus on what system calls a process is allowed to do are ineffective.
  - loeg 1y ago
    This isn't the issue; it's relatively easy to safely share some ring buffers. The issue was/is that io_uring is rapidly growing the equivalent of ~all historical Linux syscall interfaces and sometimes comparable security measures were missed on the new interfaces. (Also, stuff like seccomp filters on syscalls are kind of meaningless for io_uring.)
  - duped 1y ago
    ...don't you supply the memory in the submission queue? or do you mean the queues themselves?

fpoling 1y ago

io_uring is the asynchronous interface and that requires to use even-based architecture to use it effectively. But many command-line tools are still written is a straightforward sequential style. If C would have async or similar mechanism to pretend doing async programming sequentially, it would be easier to port. But without that a very significant refactoring is necessary. Besides, io_uring is not yet stable and who knows may be in 10 years it will be replaced by yet another mechanism to take advantage of even newer hardware. So simply waiting for io_uring prove it is here to stay is very viable strategy. Besides in 10 years we may have tools/AI that will do the rewrite automatically...

mananaysiempre 1y ago
> If C would have async or similar mechanism to pretend doing async programming sequentially, it would be easier to port. The *context() family of formerly-POSIX functions (clownishly deprecated as “use pthreads instead”) is essentially a full implementation of stackful coroutines. Even the arguable design botch of them preserving the signal mask (the reason why they aren’t the go-to option even on Linux) is theoretically fixable on the libc level without system calls, it’s just a lot of work and very few can be bothered to do signals well. As far as stackless coroutines, there’s a wide variety of libraries used in embedded systems and such (see the recent discussion[1] for some links), which are by necessity awkward enough that I don’t see any of them becoming broadly accepted. There were also a number of language extensions, among which I’d single out AC[2] (from the Barrelfish project) and CPC[3]. I’d love for, say, CPC to catch on, but it’s been over a decade now. [1] https://news.ycombinator.com/item?id=44546640 https://news.ycombinator.com/item?id=44546640 [2] https://users.soe.ucsc.edu/~abadi/Papers/acasync.pdf https://users.soe.ucsc.edu/~abadi/Papers/acasync.pdf [3] https://www.irif.fr/~jch/research/cpc-2012.pdf https://www.irif.fr/~jch/research/cpc-2012.pdf
ThePallas 1y ago
As an example of pretending to do exactly that, this project (author: me) combines io_uring and ucontext.. https://github.com/pallas/ioucontext/ https://github.com/pallas/ioucontext/

cesarb 1y ago

> I'm trying to understand why all command line tools don't use io_uring. Because it's fairly new. The coreutils package which contains the ls command (and the three earlier packages which were merged to create it) is decades old; io_uring appeared much later. It will take time for the "shared ring buffer" style of system call to win over traditional synchronous system calls.

Agingcoder 1y ago

Iouring is very recent