3 ms·

Language syntax and futures is only a half of the problem. IMO actual async IO implementation is harder. Too many incompatible platform-specific APIs: epoll an

Language syntax and futures is only a half of the problem. IMO actual async IO implementation is harder. Too many incompatible platform-specific APIs: epoll and AIO on Linux, kqueue on BSD and OSX, IOCP and new thread pool API on Windows. I’m pretty sure I forgot couple others, and each of them have non-trivial amount of quirks, bugs, and performance-ruining edge cases. Also these APIs are not directly compatible to each other. To be usable, async IO needs to be integrated into the rest of IO. You can’t just place that on top, e.g. Java did that, IMO didn’t work particularly well. The combination of the above makes creating good cross-platform abstraction for async IO challenging. Not saying impossible, but it’s very hard to do. I’ve tried once in C++, for that project I needed epoll and iocp, but I wasn’t able within reasonable time. Ended up swapping relatively large IO-related parts of my app depending on platform.

bvinc 8y ago
Most of the work you're talking about has been completed for a long time in the crate "mio". It abstracts out all of the platform dependent async io operations into a single consistent interface.
- Const-me 8y ago
  Are you sure about that? I've looked at the library, it doesn't support anything besides epoll and IOCP, i.e. no BSD or OSX support, no AIO on Linux (the kernel one, not POSIX). You probably don't want to consume IOCP API on Windows anymore, Vista introduced higher level and easier to use StartThreadpoolIo and friends. Also on Windows, even with IOCP, you don't want to split async IO from thread pool. The OS kernel manages them both at the same time. Work stealing or other custom scheduling is usually slower than what kernel does.
  - bvinc 8y ago
    It says that it is also backed by kqueue and supports FreeBSD , NetBSD, and OS X. I found this issue where they're discussing rewriting the Windows implementation using the library `wepoll`. https://github.com/tokio-rs/gsoc/issues/3 https://github.com/tokio-rs/gsoc/issues/3 What would be the advantages of supporting AIO on Linux?
    - Const-me 8y ago
      > rewriting the Windows implementation using the library `wepoll`. Interesting idea but I don’t like it too much. While removing the huge complexity of manually managing IOCP and required resources, It’s an undocumented API. New Vista+ threadpool-based IO also removes that complexity, removes complexity of implementing thread pools in the higher level in Tokyo. It’s documented and supported, and Rust has issues with WinXP support anyway. > What would be the advantages of supporting AIO on Linux? Faster async disk IO for the apps which are OK with the limitations, i.e. which read/write complete blocks of O_DIRECT files. Databases come to mind. BTW, io_uring feature coming into Linux kernel removes most limitations of AIO while also improving performance.

bvinc 8y ago

Most of the work you're talking about has been completed for a long time in the crate "mio". It abstracts out all of the platform dependent async io operations into a single consistent interface.

Const-me 8y ago
Are you sure about that? I've looked at the library, it doesn't support anything besides epoll and IOCP, i.e. no BSD or OSX support, no AIO on Linux (the kernel one, not POSIX). You probably don't want to consume IOCP API on Windows anymore, Vista introduced higher level and easier to use StartThreadpoolIo and friends. Also on Windows, even with IOCP, you don't want to split async IO from thread pool. The OS kernel manages them both at the same time. Work stealing or other custom scheduling is usually slower than what kernel does.
- bvinc 8y ago
  It says that it is also backed by kqueue and supports FreeBSD , NetBSD, and OS X. I found this issue where they're discussing rewriting the Windows implementation using the library `wepoll`. https://github.com/tokio-rs/gsoc/issues/3 https://github.com/tokio-rs/gsoc/issues/3 What would be the advantages of supporting AIO on Linux?
  - Const-me 8y ago
    > rewriting the Windows implementation using the library `wepoll`. Interesting idea but I don’t like it too much. While removing the huge complexity of manually managing IOCP and required resources, It’s an undocumented API. New Vista+ threadpool-based IO also removes that complexity, removes complexity of implementing thread pools in the higher level in Tokyo. It’s documented and supported, and Rust has issues with WinXP support anyway. > What would be the advantages of supporting AIO on Linux? Faster async disk IO for the apps which are OK with the limitations, i.e. which read/write complete blocks of O_DIRECT files. Databases come to mind. BTW, io_uring feature coming into Linux kernel removes most limitations of AIO while also improving performance.