7 ms·

> For example when submitting a write operation, the memory location of those bytes must not be deallocated or overwritten. > The io-uring crate doesn’t help m

> For example when submitting a write operation, the memory location of those bytes must not be deallocated or overwritten. > The io-uring crate doesn’t help much with this. The API doesn’t allow the borrow checker to protect you at compile time, and I don’t see it doing any runtime checks either. I've seen comments like this before[1], and I get the impression that building a a safe async Rust library around io_uring is actually quite difficult. Which is sort of a bummer. IIRC Alice from the tokio team also suggested there hasn't been much interest in pushing through these difficulties more recently, as the current performance is "good enough". [1] https://boats.gitlab.io/blog/post/io-uring/ https://boats.gitlab.io/blog/post/io-uring/

JoshTriplett 1y ago
I think the right way to build a safe interface around io_uring would be to use ring-owned buffers, ask the ring for a buffer when you want one, and give the buffer back to the ring when initiating a write.
- pingiun 1y ago
  This is something that Amos Wenger (fasterthanlime) has worked on: https://github.com/bearcove/loona/blob/main/crates/buffet/README.md https://github.com/bearcove/loona/blob/main/crates/buffet/RE...
- Tuna-Fish 1y ago
  This works perfectly well, and allows using the type system to handle safety. But it also really limits how you handle memory, and makes it impossible to do things like filling out parts of existing objects, so a lot of people are reluctant to take the plunge.
  - johncolanduoni 1y ago
    That’s annoying for people writing bespoke low-level networking code, but for a high-level HTTP library it’s a rounding error in the overall complexity on display. I think the bigger barrier for Tokio is that the interplay between having an epoll instance and a io_uring instance on the same pool is problematic and can erase performance gains. If done greenfield you could implement the “normal” APIs with ‘IORING_OP_POLL_ADD’, but not all of the exposed ‘mio’ surface area can work this way - only the oneshot API.
jcranmer 1y ago
There is, I think, an ownership model that Rust's borrow checker very poorly supports, and for lack of a better name, I've called it hot potato ownership. The basic idea is that you have a buffer which you can give out as ownership in the expectation that the person you gave it to will (eventually) give it back to you. It's a sort of non-lexical borrowing problem, and I very quickly discovered when trying to implement it myself in purely safe Rust that the "giving the buffer back" is just really gnarly to write.
- stouset 1y ago
  Maybe I’m misunderstanding, but why is that not possible with a Fn(_: T) -> T
  - iknowstuff 1y ago
    It totally is https://docs.rs/tokio-uring/latest/tokio_uring/fs/struct.File.html#method.read_at https://docs.rs/tokio-uring/latest/tokio_uring/fs/struct.Fil...
  - dwattttt 1y ago
    As sibling notes, it is. It's very rarely seen though. One place you might see something like it is if an API takes ownership, but returns it on error; you see the error side carry the resource you gave it, so you could try again.
  - IshKebab 1y ago
    How is that different to Fn(_: &mut T) ?
    - Soft 1y ago
      In the former the caller does not retain access to T until Fn returns.
      - andyferris 1y ago
        I think I'm lost. If I give a mutable reference to a function... I can't access it (even read it) until it returns, no? What is different?
ozgrakkurt 1y ago
You don’t have to represent everything with borrows. You can just use data structures like Slab to make it cancel safe. As an example this library I wrote before is cancel safe and doesn’t use lifetimes etc. for it. https://github.com/steelcake/io2 https://github.com/steelcake/io2
- ozgrakkurt 1y ago
  Just realised my code isn’t cancel safe either. It is invalid if the user just drops a read future and the buffer itself while the operation is in the kernel. It is just a PITA to get it fully right. Probably need the buffer to come from the async library so user allocates the buffers using the async library like a sibling comment says. It is just much easier to not use Rust and say futures should run fully always and can’t be just dropped and make some actual progress. So I’m just doing it in zig now
newpavlov 1y ago
This actually one of my many gripes about Rust async and why I consider it a bad addition to the language in the long term. The fundamental problem is that rust async was developed when epoll was dominant (and almost no one in the Rust circles cared about IOCP) and it has heavily influenced the async design (sometimes indirectly through other languages). Think about it for a second. Why do we not have this problem with "synchronous" syscalls? When you call `read` you also "pass mutable borrow" of the buffer to the kernel, but it maps well into the Rust ownership/borrow model since the syscall blocks execution of the thread and there are no ways to prevent it in user code. With poll-based async model you side-step this issues since you use the same "sync" syscalls, but which are guaranteed to return without blocking. For a completion-based IO to work properly with the ownership/borrow model we have to guarantee that the task code will not continue execution until it receives a completion event. You simply can not do it with state machines polled in user code. But the threading model fits here perfectly! If we are to replace threads with "green" threads, user Rust code will look indistinguishable from "synchronous" code. And no, the green threads model can work properly on embedded systems as demonstrated by many RTOSes. There are several ways of how we could've done it without making the async runtime mandatory for all targets (the main reason why green threads were removed from Rust 1.0). My personal favorite is introduction of separate "async" targets. Unfortunately, the Rust language developers made a bet on the unproved polling stackless model because of the promised efficiency and we are in the process of finding out whether the bet plays of or not.
- kibwen 1y ago
  > The fundamental problem is that rust async was developed when epoll was dominant (and almost no one in the Rust circles cared about IOCP) No, this is a mistaken retelling of history. The Rust developers were not ignorant of IOCP, nor were they zealous about any specific async model. They went looking for a model that fit with Rust's ethos, and completion didn't fit. Aaron Turon has an illuminating post from 2016 explaining their reasoning: https://aturon.github.io/tech/2016/09/07/futures-design/ https://aturon.github.io/tech/2016/09/07/futures-design/ See the section "Defining futures": There’s a very standard way to describe futures, which we found in every existing futures implementation we inspected: as a function that subscribes a callback for notification that the future is complete. Note: In the async I/O world, this kind of interface is sometimes referred to as completion-based, because events are signaled on completion of operations; Windows’s IOCP is based on this model. [...] Unfortunately, this approach nevertheless forces allocation at almost every point of future composition, and often imposes dynamic dispatch, despite our best efforts to avoid such overhead. [...] TL;DR, we were unable to make the “standard” future abstraction provide zero-cost composition of futures, and we know of no “standard” implementation that does so. [...] After much soul-searching, we arrived at a new “demand-driven” definition of futures. I'm not sure where this meme came from where people seem to think that the Rust devs rejected a completion-based scheme because of some emotional affinity for epoll. They spent a long time thinking about the problem, and came up with a solution that worked best for Rust's goals. The existence of a usable io_uring in 2016 wouldn't have changed the fundamental calculus.
aliceryhl 1y ago
> IIRC Alice from the tokio team also suggested there hasn't been much interest in pushing through these difficulties more recently, as the current performance is "good enough". Well, I think there is interest, but mostly for file IO. For file IO, the situation is pretty simple. We already have to implement that using spawn_blocking, and spawn_blocking has the exact same buffer challenges as io_uring does, so translating file IO to io_uring is not that tricky. On the other hand, I don't think tokio::net's existing APIs will support io_uring. Or at least they won't support the buffer-based io_uring APIs; there is no reason they can't register for readiness through io_uring.
- johncolanduoni 1y ago
  This covers probably 90% of the usefulness of io_uring for non-niche applications. Its original purpose was doing buffered async file IO without a bunch of caveats that make it effectively useless. The biggest speed up I’ve found with it is ‘stat’ing large sets of files in the VFS cache. It can literally be 50x faster at that, since you can do 1000 files with a single systemcall and the data you need from the kernel is all in memory. High throughput network usecases that don’t need/want AF_XDP or DPDK can get most of the speedup with ‘sendmmsg/recvmmsg’ and segmentation offload.
  - deleted 1y ago
    [deleted]
  - dpeckett 1y ago
    For TCP streams syscall overhead isn't a big issue really, you can easily transfer large chunks of data in each write(). If you have TCP segmentation offload available you'll have no serious issues pushing 100gbit/s. Also if you are sending static content don't forget sendfile(). UDP is a whole another kettle of fish, get's very complicated to go above 10gbit/s or so. This is a big part of why QUIC really struggles to scale well for fat pipes [1]. sendmmsg/recvmmsg + UDP GRO/GSO will probably get you to ~30gbit/s but beyond that is a real headache. The issue is that UDP is not stream focused so you're making a ton of little writes and the kernel networking stack as of today does a pretty bad job with these workloads. FWIW even the fastest QUIC implementations cap out at <10gbit/s today [2]. Had a good fight writing a ~20gbit userspace UDP VPN recently. Ended up having to bypass the kernels networking stack using AF_XDP [3]. I'm available for hire btw, if you've got an interesting networking project feel free to reach out. 1. https://arxiv.org/abs/2310.09423 https://arxiv.org/abs/2310.09423 2. https://microsoft.github.io/msquic/ https://microsoft.github.io/msquic/ 3. https://github.com/apoxy-dev/icx/blob/main/tunnel/tunnel.go https://github.com/apoxy-dev/icx/blob/main/tunnel/tunnel.go
johncolanduoni 1y ago
It’s annoying but possible to do this correctly and not have the API be too bad. The “happy path” of a clean success or error is fine if you accept that buffers can’t just be simple &[u8] slices. Cancellation can be handled safely with something like the following API contract: Have your function signature be async fn read(buffer: &mut Vec<u8>) -> Result<…>’ (you can use something more convenient like ‘&mut BytesMut’ too). If you run the future to completion (success or failure), the argument holds the same buffer passed in, with data filled in appropriately on success. If you cancel/drop the future, the buffer may point at an empty allocation instead (this is usually not an annoying constraint for most IO flows, and footgun potential is low). The way this works is that your library “takes” the underlying allocation before starting the operation out of the variable, replacing it with the default unallocated ‘Vec<u8>’. Once the buffer is no longer used by the IO system, it puts it back before returning. If you cancel, it manages the buffer in the background to release it when safe and the unallocated buffer is left in the passed variable.
- andyferris 1y ago
  It sounds like this would be better modelled by passing ownership of the buffer and expecting it to be returned on the success (ok) case. What you described doesn't seem compatible with what I would call a mutable borrow (mutate the contents of a Vec<u8>). Or maybe I've misunderstood?
  - johncolanduoni 1y ago
    It is compatible under Rust’s model (I’ve used it to implement safe io_uring interfaces specifically). ‘&mut Vec<u8>’ doesn’t just let you mutate contents or extend the allocation - you can call ‘mem::replace(…)’ and swap the allocation entirely. It’s morally equivalent to passing back and forth, and almost identical in the generated machine code (structure return values look a lot like mutable structure arguments at the register calling convention level). However it’s much less annoying to work with in practice - passing buffers back and forth and then reassigning them to the same variable name results in a lot of semantically irrelevant code to please the ownership model.
touisteur 1y ago
I wish I could have been paid to work on SPARK specification around io_uring so that one could have built on it. Or to work on SPARK-to-eBPF (there's already a llvm backend for gnat) and have some form of guarantees at the seams... alas.

JoshTriplett 1y ago

I think the right way to build a safe interface around io_uring would be to use ring-owned buffers, ask the ring for a buffer when you want one, and give the buffer back to the ring when initiating a write.

pingiun 1y ago
This is something that Amos Wenger (fasterthanlime) has worked on: https://github.com/bearcove/loona/blob/main/crates/buffet/README.md https://github.com/bearcove/loona/blob/main/crates/buffet/RE...
Tuna-Fish 1y ago
This works perfectly well, and allows using the type system to handle safety. But it also really limits how you handle memory, and makes it impossible to do things like filling out parts of existing objects, so a lot of people are reluctant to take the plunge.
- johncolanduoni 1y ago
  That’s annoying for people writing bespoke low-level networking code, but for a high-level HTTP library it’s a rounding error in the overall complexity on display. I think the bigger barrier for Tokio is that the interplay between having an epoll instance and a io_uring instance on the same pool is problematic and can erase performance gains. If done greenfield you could implement the “normal” APIs with ‘IORING_OP_POLL_ADD’, but not all of the exposed ‘mio’ surface area can work this way - only the oneshot API.

jcranmer 1y ago

There is, I think, an ownership model that Rust's borrow checker very poorly supports, and for lack of a better name, I've called it hot potato ownership. The basic idea is that you have a buffer which you can give out as ownership in the expectation that the person you gave it to will (eventually) give it back to you. It's a sort of non-lexical borrowing problem, and I very quickly discovered when trying to implement it myself in purely safe Rust that the "giving the buffer back" is just really gnarly to write.

stouset 1y ago
Maybe I’m misunderstanding, but why is that not possible with a Fn(_: T) -> T
- iknowstuff 1y ago
  It totally is https://docs.rs/tokio-uring/latest/tokio_uring/fs/struct.File.html#method.read_at https://docs.rs/tokio-uring/latest/tokio_uring/fs/struct.Fil...
- dwattttt 1y ago
  As sibling notes, it is. It's very rarely seen though. One place you might see something like it is if an API takes ownership, but returns it on error; you see the error side carry the resource you gave it, so you could try again.
- IshKebab 1y ago
  How is that different to Fn(_: &mut T) ?
  - Soft 1y ago
    In the former the caller does not retain access to T until Fn returns.
    - andyferris 1y ago
      I think I'm lost. If I give a mutable reference to a function... I can't access it (even read it) until it returns, no? What is different?

ozgrakkurt 1y ago

You don’t have to represent everything with borrows. You can just use data structures like Slab to make it cancel safe. As an example this library I wrote before is cancel safe and doesn’t use lifetimes etc. for it. https://github.com/steelcake/io2 https://github.com/steelcake/io2

ozgrakkurt 1y ago
Just realised my code isn’t cancel safe either. It is invalid if the user just drops a read future and the buffer itself while the operation is in the kernel. It is just a PITA to get it fully right. Probably need the buffer to come from the async library so user allocates the buffers using the async library like a sibling comment says. It is just much easier to not use Rust and say futures should run fully always and can’t be just dropped and make some actual progress. So I’m just doing it in zig now

newpavlov 1y ago

This actually one of my many gripes about Rust async and why I consider it a bad addition to the language in the long term. The fundamental problem is that rust async was developed when epoll was dominant (and almost no one in the Rust circles cared about IOCP) and it has heavily influenced the async design (sometimes indirectly through other languages). Think about it for a second. Why do we not have this problem with "synchronous" syscalls? When you call `read` you also "pass mutable borrow" of the buffer to the kernel, but it maps well into the Rust ownership/borrow model since the syscall blocks execution of the thread and there are no ways to prevent it in user code. With poll-based async model you side-step this issues since you use the same "sync" syscalls, but which are guaranteed to return without blocking. For a completion-based IO to work properly with the ownership/borrow model we have to guarantee that the task code will not continue execution until it receives a completion event. You simply can not do it with state machines polled in user code. But the threading model fits here perfectly! If we are to replace threads with "green" threads, user Rust code will look indistinguishable from "synchronous" code. And no, the green threads model can work properly on embedded systems as demonstrated by many RTOSes. There are several ways of how we could've done it without making the async runtime mandatory for all targets (the main reason why green threads were removed from Rust 1.0). My personal favorite is introduction of separate "async" targets. Unfortunately, the Rust language developers made a bet on the unproved polling stackless model because of the promised efficiency and we are in the process of finding out whether the bet plays of or not.

kibwen 1y ago
> The fundamental problem is that rust async was developed when epoll was dominant (and almost no one in the Rust circles cared about IOCP) No, this is a mistaken retelling of history. The Rust developers were not ignorant of IOCP, nor were they zealous about any specific async model. They went looking for a model that fit with Rust's ethos, and completion didn't fit. Aaron Turon has an illuminating post from 2016 explaining their reasoning: https://aturon.github.io/tech/2016/09/07/futures-design/ https://aturon.github.io/tech/2016/09/07/futures-design/ See the section "Defining futures": There’s a very standard way to describe futures, which we found in every existing futures implementation we inspected: as a function that subscribes a callback for notification that the future is complete. Note: In the async I/O world, this kind of interface is sometimes referred to as completion-based, because events are signaled on completion of operations; Windows’s IOCP is based on this model. [...] Unfortunately, this approach nevertheless forces allocation at almost every point of future composition, and often imposes dynamic dispatch, despite our best efforts to avoid such overhead. [...] TL;DR, we were unable to make the “standard” future abstraction provide zero-cost composition of futures, and we know of no “standard” implementation that does so. [...] After much soul-searching, we arrived at a new “demand-driven” definition of futures. I'm not sure where this meme came from where people seem to think that the Rust devs rejected a completion-based scheme because of some emotional affinity for epoll. They spent a long time thinking about the problem, and came up with a solution that worked best for Rust's goals. The existence of a usable io_uring in 2016 wouldn't have changed the fundamental calculus.

aliceryhl 1y ago

> IIRC Alice from the tokio team also suggested there hasn't been much interest in pushing through these difficulties more recently, as the current performance is "good enough". Well, I think there is interest, but mostly for file IO. For file IO, the situation is pretty simple. We already have to implement that using spawn_blocking, and spawn_blocking has the exact same buffer challenges as io_uring does, so translating file IO to io_uring is not that tricky. On the other hand, I don't think tokio::net's existing APIs will support io_uring. Or at least they won't support the buffer-based io_uring APIs; there is no reason they can't register for readiness through io_uring.

johncolanduoni 1y ago
This covers probably 90% of the usefulness of io_uring for non-niche applications. Its original purpose was doing buffered async file IO without a bunch of caveats that make it effectively useless. The biggest speed up I’ve found with it is ‘stat’ing large sets of files in the VFS cache. It can literally be 50x faster at that, since you can do 1000 files with a single systemcall and the data you need from the kernel is all in memory. High throughput network usecases that don’t need/want AF_XDP or DPDK can get most of the speedup with ‘sendmmsg/recvmmsg’ and segmentation offload.
- deleted 1y ago
  [deleted]
- dpeckett 1y ago
  For TCP streams syscall overhead isn't a big issue really, you can easily transfer large chunks of data in each write(). If you have TCP segmentation offload available you'll have no serious issues pushing 100gbit/s. Also if you are sending static content don't forget sendfile(). UDP is a whole another kettle of fish, get's very complicated to go above 10gbit/s or so. This is a big part of why QUIC really struggles to scale well for fat pipes [1]. sendmmsg/recvmmsg + UDP GRO/GSO will probably get you to ~30gbit/s but beyond that is a real headache. The issue is that UDP is not stream focused so you're making a ton of little writes and the kernel networking stack as of today does a pretty bad job with these workloads. FWIW even the fastest QUIC implementations cap out at <10gbit/s today [2]. Had a good fight writing a ~20gbit userspace UDP VPN recently. Ended up having to bypass the kernels networking stack using AF_XDP [3]. I'm available for hire btw, if you've got an interesting networking project feel free to reach out. 1. https://arxiv.org/abs/2310.09423 https://arxiv.org/abs/2310.09423 2. https://microsoft.github.io/msquic/ https://microsoft.github.io/msquic/ 3. https://github.com/apoxy-dev/icx/blob/main/tunnel/tunnel.go https://github.com/apoxy-dev/icx/blob/main/tunnel/tunnel.go

johncolanduoni 1y ago

It’s annoying but possible to do this correctly and not have the API be too bad. The “happy path” of a clean success or error is fine if you accept that buffers can’t just be simple &[u8] slices. Cancellation can be handled safely with something like the following API contract: Have your function signature be async fn read(buffer: &mut Vec<u8>) -> Result<…>’ (you can use something more convenient like ‘&mut BytesMut’ too). If you run the future to completion (success or failure), the argument holds the same buffer passed in, with data filled in appropriately on success. If you cancel/drop the future, the buffer may point at an empty allocation instead (this is usually not an annoying constraint for most IO flows, and footgun potential is low). The way this works is that your library “takes” the underlying allocation before starting the operation out of the variable, replacing it with the default unallocated ‘Vec<u8>’. Once the buffer is no longer used by the IO system, it puts it back before returning. If you cancel, it manages the buffer in the background to release it when safe and the unallocated buffer is left in the passed variable.

andyferris 1y ago
It sounds like this would be better modelled by passing ownership of the buffer and expecting it to be returned on the success (ok) case. What you described doesn't seem compatible with what I would call a mutable borrow (mutate the contents of a Vec<u8>). Or maybe I've misunderstood?
- johncolanduoni 1y ago
  It is compatible under Rust’s model (I’ve used it to implement safe io_uring interfaces specifically). ‘&mut Vec<u8>’ doesn’t just let you mutate contents or extend the allocation - you can call ‘mem::replace(…)’ and swap the allocation entirely. It’s morally equivalent to passing back and forth, and almost identical in the generated machine code (structure return values look a lot like mutable structure arguments at the register calling convention level). However it’s much less annoying to work with in practice - passing buffers back and forth and then reassigning them to the same variable name results in a lot of semantically irrelevant code to please the ownership model.

touisteur 1y ago

I wish I could have been paid to work on SPARK specification around io_uring so that one could have built on it. Or to work on SPARK-to-eBPF (there's already a llvm backend for gnat) and have some form of guarantees at the seams... alas.