8 ms·

> I don't think io_uring is as complex as its reputation suggests. uring is extremely problematic to integrate into many common application / language runtimes

by raggi 2y ago

> I don't think io_uring is as complex as its reputation suggests. uring is extremely problematic to integrate into many common application / language runtimes and it has been demonstrably difficult to integrate into linux safely and correctly as well, with a continual stream of bugs, security and policy control issues. in principle a shared memory queue is a reasonable basis for improving the IO cost between applications and IO stacks such as the network or filesystem stacks, but this isn't easy to do well, cf. uring bugs and binder bugs.

JoshTriplett 2y ago
> with a continual stream of bugs, security and policy control issues This has not been true for a long time. There was an early design mistake that made it quite prone to these, but that mistake has been fixed. Unfortunately, the reputational damage will stick around for a while.
- raggi 2y ago
  13 CVEs so far this year afaik
  - bonzini 2y ago
    CVE numbers from the Linux CNA are bollocks.
    - JoshTriplett 2y ago
      This conversation would be a good one to point them to to show that their policy is not just harmless point-proving, but in fact does cause harm. For context, to the best of my knowledge the current approach of the Linux CNA is, in keeping with long-standing Linux security policy of "every single fix might be a security fix", to assign CVEs regardless of whether something has any security impact or not.
      - di4na 2y ago
        I would not call it harm. The use of uring in higher level languages is definitely prone to errors, bugs and security problems
        JoshTriplett 2y ago
        See the context I added to that comment; this is not about security issues, it's about the Linux CNA's absurd approach to CVE assignment for things that aren't CVEs.
        tialaramex 2y ago
        I don't agree that it's absurd. I would say it reflects a proper understanding of their situation. You've doubtless heard Tony Hoare's "There are two ways to write code: write code so simple there are obviously no bugs in it, or write code so complex that there are no obvious bugs in it.". Linux is definitely in the latter category, it's now such a sprawling system that determining whether a bug "really" has security implications is no long a reasonable task compared to just fixing the bug. The other reason is that Linux is so widely used that almost no assumption made to simplify that above task is definitely correct.
        JoshTriplett 2y ago
        That's fine, except that it is thus no longer meaningful to compare CVE count.
        hifromwork 2y ago
        I like CVEs, I think Linux approach to CVEs is stupid, but also it was never meaningful to compare CVE count. But I guess it's hard to make people stop doing that, and that's the reason Linux does the thing it does out of spite.
      - kuschku 2y ago
        CVE assignment != security issue CVE numbers are just a way to ensure everyone is talking about the same bug. Not every security issue has a CVE, not every CVE is a security issue. Often, a regular bug turns out years later to have been a security issue, or a security issue turns out to have no security impact at all. If you want a central authority to tell you what to think, just use CVSS instead of the binary "does it have a CVE" metric.
    - raggi 2y ago
      this is a bit of a distraction, sure the leaks and some of the deadlocks are fairly uninteresting, but the toctou, overflows, uid race/confusion and so on are real issues that shouldn't be dismissed as if they don't exist.
jeffparsons 2y ago
I find this surprising, given that my initial response to reading the iouring design was: 1. This is pretty clean and straightforward. 2. This is obviously what we need to decouple a bunch of things without the previous downsides. What has made it so hard to integrate it into common language runtimes? Do you have examples of where there's been an irreconcilable "impedance mismatch"?
- raggi 2y ago
  https://github.com/tailscale/tailscale/pull/2370 https://github.com/tailscale/tailscale/pull/2370 was a practical drive toward this, will not proceed on this path. much more approachable, boats has written about challenges integrating in rust: https://without.boats/tags/io-uring/ https://without.boats/tags/io-uring/ in the most general form: you need a fairly "loose" memory model to integrate the "best" (performance wise) parts, and the "best" (ease of use/forward looking safety) way to integrate requires C library linkage. This is troublesome in most GC languages, and many managed runtimes. There's also the issue that uring being non-portable means that the things it suggests you must do (such as say pinning a buffer pool and making APIs like read not immediate caller allocates) requires a substantially separate API for this platform than for others, or at least substantial reworks over all the existing POSIX modeled APIs - thus back to what I said originally, we need a replacement for POSIX & BSD here, broadly applied.
  - gpderetta 2y ago
    I can see how a zero-copy API would be hard to implement on some languages, but you could still implement something on top of io_uring with posix buffer copy semantics , while using batching to decrease syscall overhead. Zero-copy APIs will necessarily be tricky to implement and use, especially on memory safe languages.
    - gmokki 2y ago
      I think most GC languages support native/pinned me(at least Java and C# do memory to support talking to kernel or native libraries. The APIs are even quite nice.
arghwhat 2y ago
Two things: One, uring is not extremely problematic to integrate, as it can be chained into a conventional event loop if you want to, or can even be fit into a conventionally blocking design to get localized syscall benefits. That is, you do not need to convert to a fully uring event loop design, even if that would be superior - and it can usually be kept entirely within a (slightly modified) event loop abstraction. The reason it has not yet been implemented is just priority - most stuff isn't bottlenecked on IOPS. Two, yes you could have e middle-ground. I assume the syscall overhead you call out is the need to send UDP packets one at a time through sendmsg/sendto, rather than doing one big write for several packets worth of data on TCP. An API that allowed you to provide a chain of messages, like sendmsg takes an iovec for data, is possible. But it's also possible to do this already as a tiny blocking wrapper around io_uring, saving you new syscalls.
- Veserv 2y ago
  The system call to send multiple UDP packets in a single call has existed since Linux 3.0 over a decade ago[1]: sendmmsg(). [1] https://man7.org/linux/man-pages/man2/sendmmsg.2.html https://man7.org/linux/man-pages/man2/sendmmsg.2.html
  - arghwhat 2y ago
    Ah nice, in that case OP's point about syscall overhead is entirely moot. :) That should really be in the `SEE ALSO` of `man 3 sendmsg`...
    - evntdrvn 2y ago
      patches welcome :p
    - wtarreau 2y ago
      There's still the problem of sending to multiple destinations: OK sendmmsg() can send multiple datagrams, but for a given socket. When you have small windows (thank you cubic), you'll just send a few datagrams this way and don't save much.

by raggi 2y ago

JoshTriplett 2y ago

> with a continual stream of bugs, security and policy control issues This has not been true for a long time. There was an early design mistake that made it quite prone to these, but that mistake has been fixed. Unfortunately, the reputational damage will stick around for a while.

raggi 2y ago
13 CVEs so far this year afaik
- bonzini 2y ago
  CVE numbers from the Linux CNA are bollocks.
  - JoshTriplett 2y ago
    This conversation would be a good one to point them to to show that their policy is not just harmless point-proving, but in fact does cause harm. For context, to the best of my knowledge the current approach of the Linux CNA is, in keeping with long-standing Linux security policy of "every single fix might be a security fix", to assign CVEs regardless of whether something has any security impact or not.
    - di4na 2y ago
      I would not call it harm. The use of uring in higher level languages is definitely prone to errors, bugs and security problems
      - JoshTriplett 2y ago
        See the context I added to that comment; this is not about security issues, it's about the Linux CNA's absurd approach to CVE assignment for things that aren't CVEs.
        tialaramex 2y ago
        I don't agree that it's absurd. I would say it reflects a proper understanding of their situation. You've doubtless heard Tony Hoare's "There are two ways to write code: write code so simple there are obviously no bugs in it, or write code so complex that there are no obvious bugs in it.". Linux is definitely in the latter category, it's now such a sprawling system that determining whether a bug "really" has security implications is no long a reasonable task compared to just fixing the bug. The other reason is that Linux is so widely used that almost no assumption made to simplify that above task is definitely correct.
        JoshTriplett 2y ago
        That's fine, except that it is thus no longer meaningful to compare CVE count.
        hifromwork 2y ago
        I like CVEs, I think Linux approach to CVEs is stupid, but also it was never meaningful to compare CVE count. But I guess it's hard to make people stop doing that, and that's the reason Linux does the thing it does out of spite.
    - kuschku 2y ago
      CVE assignment != security issue CVE numbers are just a way to ensure everyone is talking about the same bug. Not every security issue has a CVE, not every CVE is a security issue. Often, a regular bug turns out years later to have been a security issue, or a security issue turns out to have no security impact at all. If you want a central authority to tell you what to think, just use CVSS instead of the binary "does it have a CVE" metric.
  - raggi 2y ago
    this is a bit of a distraction, sure the leaks and some of the deadlocks are fairly uninteresting, but the toctou, overflows, uid race/confusion and so on are real issues that shouldn't be dismissed as if they don't exist.

jeffparsons 2y ago

I find this surprising, given that my initial response to reading the iouring design was: 1. This is pretty clean and straightforward. 2. This is obviously what we need to decouple a bunch of things without the previous downsides. What has made it so hard to integrate it into common language runtimes? Do you have examples of where there's been an irreconcilable "impedance mismatch"?

raggi 2y ago
https://github.com/tailscale/tailscale/pull/2370 https://github.com/tailscale/tailscale/pull/2370 was a practical drive toward this, will not proceed on this path. much more approachable, boats has written about challenges integrating in rust: https://without.boats/tags/io-uring/ https://without.boats/tags/io-uring/ in the most general form: you need a fairly "loose" memory model to integrate the "best" (performance wise) parts, and the "best" (ease of use/forward looking safety) way to integrate requires C library linkage. This is troublesome in most GC languages, and many managed runtimes. There's also the issue that uring being non-portable means that the things it suggests you must do (such as say pinning a buffer pool and making APIs like read not immediate caller allocates) requires a substantially separate API for this platform than for others, or at least substantial reworks over all the existing POSIX modeled APIs - thus back to what I said originally, we need a replacement for POSIX & BSD here, broadly applied.
- gpderetta 2y ago
  I can see how a zero-copy API would be hard to implement on some languages, but you could still implement something on top of io_uring with posix buffer copy semantics , while using batching to decrease syscall overhead. Zero-copy APIs will necessarily be tricky to implement and use, especially on memory safe languages.
  - gmokki 2y ago
    I think most GC languages support native/pinned me(at least Java and C# do memory to support talking to kernel or native libraries. The APIs are even quite nice.

arghwhat 2y ago

Two things: One, uring is not extremely problematic to integrate, as it can be chained into a conventional event loop if you want to, or can even be fit into a conventionally blocking design to get localized syscall benefits. That is, you do not need to convert to a fully uring event loop design, even if that would be superior - and it can usually be kept entirely within a (slightly modified) event loop abstraction. The reason it has not yet been implemented is just priority - most stuff isn't bottlenecked on IOPS. Two, yes you could have e middle-ground. I assume the syscall overhead you call out is the need to send UDP packets one at a time through sendmsg/sendto, rather than doing one big write for several packets worth of data on TCP. An API that allowed you to provide a chain of messages, like sendmsg takes an iovec for data, is possible. But it's also possible to do this already as a tiny blocking wrapper around io_uring, saving you new syscalls.

Veserv 2y ago
The system call to send multiple UDP packets in a single call has existed since Linux 3.0 over a decade ago[1]: sendmmsg(). [1] https://man7.org/linux/man-pages/man2/sendmmsg.2.html https://man7.org/linux/man-pages/man2/sendmmsg.2.html
- arghwhat 2y ago
  Ah nice, in that case OP's point about syscall overhead is entirely moot. :) That should really be in the `SEE ALSO` of `man 3 sendmsg`...
  - evntdrvn 2y ago
    patches welcome :p
  - wtarreau 2y ago
    There's still the problem of sending to multiple destinations: OK sendmmsg() can send multiple datagrams, but for a given socket. When you have small windows (thank you cubic), you'll just send a few datagrams this way and don't save much.