4 ms·

io_uring sounds promising. It'll be interesting to see how long it takes for Tokio to add support for it in the Rust community, and how long it takes for Go to

io_uring sounds promising. It'll be interesting to see how long it takes for Tokio to add support for it in the Rust community, and how long it takes for Go to transparently support this instead of a pool of file I/O workers.

jcranmer 7y ago
The bug is here, if you want to follow it: https://github.com/tokio-rs/mio/issues/923 https://github.com/tokio-rs/mio/issues/923
Matthias247 7y ago
It will be interesting to see whether Rusts async ecosystem gets support for it. However there is a little bit of a challenge in the form of an impedance mismatch: Rust Futures and thereby also async functions require that an operation is cancellable at any point of time (by dropping the respective Future). IO completion based systems (like io_uring and IOCP) typically don't support this. If the operation in the Rust userspace would be cancelled while the Kernel proceeds with the operation the Kernel would act on invalid memory. There are some workarounds for this, like wrapping the APIs in ones that only used owned buffers that are passed for the whole duration of the operation and reference-counting the IO objects (file descriptors). But the behavior is pretty tricky. [1] describes the problem. [1] https://github.com/rust-lang-nursery/futures-rs/issues/1278 https://github.com/rust-lang-nursery/futures-rs/issues/1278
- coder543 7y ago
  That's really interesting. I think some kind of reference counting and a shadow event loop is going to be necessary, if that's the case, so that I/O futures can always be driven to completion even if the developer-facing one is dropped.
  - Matthias247 7y ago
    Yes, reference-counting of outstanding operations and the used buffers is a suitable workaround. It comes with some downsides though too: The IO operations then only accept certain specialized buffer types and nothing generic like pure byte slices anymore, which might make those things less interoperable. The other issue is that while using refcounting leaves at least the memory in a non-corrupted state the IO source will be in a somewhat undefined state after the cancellation. E.g. if a write operation had been issued on the socket, the API user can not make any assumptions whether or not any byte had been written, and which impact a subsequent write might have. The only reasonable thing to do is to close the IO source (where the close might also need to wait until the still pending background task has completed).
    - coder543 7y ago
      If the future has been dropped, I agree that the source/sink should be closed as soon as safely possible. Usually the resource has to be moved into a future for it to be used, and thus it's logical that no one could continue using it after the future is dropped.
- GolDDranks 7y ago
  I wonder if it would be possible to pass the ownership of a structure that contains the buffer to kernel, passing a pointer in the user_data. The application could then `std::mem::forget` the buffer, and re-construct it from the raw pointer when the completion event arrives. The Future that originated the IO job could then be dropped independently from the buffer.
- the8472 7y ago
  io_uring definitely wants to own the buffers, either for its entire lifetime (with preregistered memory mappings) or at least until completion. if possible the future should take ownership of the buffer and have it returned on completion.
stbuehler 7y ago
https://github.com/stbuehler/rust-io-uring https://github.com/stbuehler/rust-io-uring Note that io_uring in linux 5.1 has various bugs; especially READ* and WRITE* on sockets/pipe don't work very well (afaik they either block submission or return -EAGAIN with O_NONBLOCK), so the example code in my repo isn't that useful for linux 5.1 (POLL_ADD works fine though). For proper support in tokio this would have to be done in mio afaict (or tokio would have to replace mio with something else).
- coder543 7y ago
  do those issues only apply to networking? the big benefit I see for io_uring is that it provides a truly async interface for file I/O for the first time ever in Linux... as far as I can tell.
  - stbuehler 7y ago
    I'm not sure about the details for file I/O; files with FMODE_NOWAIT should be fine (block devices, ext4, btrfs, ...). The idea is that during submission an operation is started with IOCB_NOWAIT; an operation shouldn't block with that. If it returns EAGAIN it will be retried in another kernel thread without IOCB_NOWAIT. Afaik IOCB_NOWAIT support should be announced through FMODE_NOWAIT. Right now operations that don't support IOCB_NOWAIT ("magic check" in io_file_supports_async [1]) are still run during submission but without the IOCB_NOWAIT flag. Sockets/pipes don't handle IOCB_NOWAIT, they only care about O_NONBLOCK right now (which is a bug imho). [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/io_uring.c?h=v5.1#n823 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

jcranmer 7y ago

The bug is here, if you want to follow it: https://github.com/tokio-rs/mio/issues/923 https://github.com/tokio-rs/mio/issues/923

Matthias247 7y ago

It will be interesting to see whether Rusts async ecosystem gets support for it. However there is a little bit of a challenge in the form of an impedance mismatch: Rust Futures and thereby also async functions require that an operation is cancellable at any point of time (by dropping the respective Future). IO completion based systems (like io_uring and IOCP) typically don't support this. If the operation in the Rust userspace would be cancelled while the Kernel proceeds with the operation the Kernel would act on invalid memory. There are some workarounds for this, like wrapping the APIs in ones that only used owned buffers that are passed for the whole duration of the operation and reference-counting the IO objects (file descriptors). But the behavior is pretty tricky. [1] describes the problem. [1] https://github.com/rust-lang-nursery/futures-rs/issues/1278 https://github.com/rust-lang-nursery/futures-rs/issues/1278

coder543 7y ago
That's really interesting. I think some kind of reference counting and a shadow event loop is going to be necessary, if that's the case, so that I/O futures can always be driven to completion even if the developer-facing one is dropped.
- Matthias247 7y ago
  Yes, reference-counting of outstanding operations and the used buffers is a suitable workaround. It comes with some downsides though too: The IO operations then only accept certain specialized buffer types and nothing generic like pure byte slices anymore, which might make those things less interoperable. The other issue is that while using refcounting leaves at least the memory in a non-corrupted state the IO source will be in a somewhat undefined state after the cancellation. E.g. if a write operation had been issued on the socket, the API user can not make any assumptions whether or not any byte had been written, and which impact a subsequent write might have. The only reasonable thing to do is to close the IO source (where the close might also need to wait until the still pending background task has completed).
  - coder543 7y ago
    If the future has been dropped, I agree that the source/sink should be closed as soon as safely possible. Usually the resource has to be moved into a future for it to be used, and thus it's logical that no one could continue using it after the future is dropped.
GolDDranks 7y ago
I wonder if it would be possible to pass the ownership of a structure that contains the buffer to kernel, passing a pointer in the user_data. The application could then `std::mem::forget` the buffer, and re-construct it from the raw pointer when the completion event arrives. The Future that originated the IO job could then be dropped independently from the buffer.
the8472 7y ago
io_uring definitely wants to own the buffers, either for its entire lifetime (with preregistered memory mappings) or at least until completion. if possible the future should take ownership of the buffer and have it returned on completion.

stbuehler 7y ago

https://github.com/stbuehler/rust-io-uring https://github.com/stbuehler/rust-io-uring Note that io_uring in linux 5.1 has various bugs; especially READ* and WRITE* on sockets/pipe don't work very well (afaik they either block submission or return -EAGAIN with O_NONBLOCK), so the example code in my repo isn't that useful for linux 5.1 (POLL_ADD works fine though). For proper support in tokio this would have to be done in mio afaict (or tokio would have to replace mio with something else).

coder543 7y ago
do those issues only apply to networking? the big benefit I see for io_uring is that it provides a truly async interface for file I/O for the first time ever in Linux... as far as I can tell.
- stbuehler 7y ago
  I'm not sure about the details for file I/O; files with FMODE_NOWAIT should be fine (block devices, ext4, btrfs, ...). The idea is that during submission an operation is started with IOCB_NOWAIT; an operation shouldn't block with that. If it returns EAGAIN it will be retried in another kernel thread without IOCB_NOWAIT. Afaik IOCB_NOWAIT support should be announced through FMODE_NOWAIT. Right now operations that don't support IOCB_NOWAIT ("magic check" in io_file_supports_async [1]) are still run during submission but without the IOCB_NOWAIT flag. Sockets/pipes don't handle IOCB_NOWAIT, they only care about O_NONBLOCK right now (which is a bug imho). [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/io_uring.c?h=v5.1#n823 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...