5 ms·

After I've read the article which kind of praises AIO, I have impression that AIO is misdesigned, have misleading name and overall quite terrible. I better stay

by adontz 8y ago

After I've read the article which kind of praises AIO, I have impression that AIO is misdesigned, have misleading name and overall quite terrible. I better stay away from it. Also, epoll is good enough for me, my tasks. So no actual need to jump on the code to replace my epoll event loops.

ahartmetz 8y ago
Apart from the ugly API (including no glibc support), the most ridiculous part is that it only works asynchronously with XFS and without caching! Without caching! In an API that is supposed to improve performance! I think we just need I/O readiness like epoll for disk I/O. AIO for disk seems to use the completion model ("do it and tell me when you're done" like Windows), which has higher maximum performance, but is more difficult to use / integrate. So you can now do both disk and network I/O with AIO... but it's going to be readiness based for network and completion based for disk. What the actual fuck. Edits: Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much (not necessary for network sockets), which is new API that you have to use. Reading must be done when the disk is ready and can't wait for you, so it needs another buffer and memcpy. Readiness based disk writing also kind of works and it's easier - the system tells you when the write buffer has room again (threshold selectable or fixed), but submitted data is not going to be written right away. This might be the semi-reasonable explanation for the strange O_DIRECT limitation of AIO - if you need async buffered writes, just let the write cache do its work (and, you know, randomly block sometimes - never mind reading). So readiness-based disk I/O could work with reasonably user-friendly new API for reading and not much worse performance loss than readiness-based network I/O. No idea about how much work it would take in the kernel. It would also be nice to have completion-based network I/O with AIO for symmetry reasons and in case you really need the best performance. Though I have never worked on something where completion-based network I/O would have made a meaningful performance difference.
- phs2501 8y ago
  The problem with readiness-based IO for disk is that without starting the request it doesn't know where it needs to read on the disk to fetch, so it can't possibly tell you if the data's ready. If you were to limit file IO to streaming rather than random-access it could potentially work, but that's not the POSIX interface. Since pipes and sockets are streams they don't have this issue. I'm not defending the suckiness of the AIO situation on Linux, but I understand why it is to some extent.
  - ahartmetz 8y ago
    You're right, I was working on an edit to address the problem while you wrote your comment :)
  - loeg 8y ago
    At a minimum it could tell you if the first byte at the fd offset is in cache, which it does not today. That'd at least get you a filesystem block (or more likely, 4k page). I agree this is less useful than, say, the full generality of NT IO completion.
    - the8472 8y ago
      Linux has fadvise to request an async read and preadv2(..., RWF_NOWAIT) to try to get things from the page cache. All that's missing now is a notification.
      - loeg 8y ago
        That's the key distinction between polling and select-like waiting.
- quotemstr 8y ago
  In the intended use case, the lack of kernel-level caching can be a performance win. A database can do a better job of caching than the kernel can because it has a clearer picture of the expected access patterns. O_DIRECT in this kind of application makes a lot of sense.
- jandrewrogers 8y ago
  A major reason to use Linux AIO is because it has no caching. That's a feature! A well-designed database engine can have a disk cache that outperforms the kernel-provided disk cache by a very large margin. Same story with I/O scheduling, which Linux AIO also lets you replace. Almost all the performance benefits that accrue from Linux AIO for disk access is that it allows you to replace the operating system cache and scheduler with something that is better designed for your application. And large gains are demonstrably possible. If you just wanted asynchronous disk calls, there are other APIs that use thread pools for that.
  - loeg 8y ago
    A good model would have a flag that allowed databases to disable caching and everyone else to enable it, because not everything is a database engine. The absence of choice is not a feature.
    - ahartmetz 8y ago
      At least if both choices make sense, which they do.
      - loeg 8y ago
        Sure, I wasn't clear enough — I was implying that.
        ahartmetz 8y ago
        You were clear enough, I was being redundant. Let's stop this here ;)
- phs2501 8y ago
  > Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much [...], which is new API that you have to use. Reading must be done when the disk is ready and can't wait for you, so it needs another buffer and memcpy. > So readiness-based disk I/O could work with reasonably user-friendly new API for reading and not much worse performance loss than readiness-based network I/O. Off the cuff that new API sounds exactly like a completion-signaled read. You tell the system up from where to read and how much (the async read call), it reads when the disk is ready and buffers for you (your passed buffer and/or the cache if you're clever), and then it signals "readiness" which is the same as completion in this case. Do you see this as distinct in any other way?
  - ahartmetz 8y ago
    At their core, they are distinct only regarding when data is copied from kernel to user space. Readiness based copies when the application does the read() after being notified of "readiness", completion based copies (or DMAs directly from disk controller and maps into user space) before the application is notified of completion. The former is easier to use (no buffer ownership headaches), the latter is faster. Though there may be buffer management headaches in the kernel with readiness - what to do about ready reads the aren't collected? (Probably just throw them away if uncollected at the next AIO poll...)
- loeg 8y ago
  > Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much > It would also be nice to have completion-based network I/O with AIO for symmetry reasons I.e., you want something like NT IO completion ports, introduced in 1994.
  - ahartmetz 8y ago
    > I.e., you want something like NT IO completion ports, introduced in 1994. Yes, that's why I mentioned Windows.
jandrewrogers 8y ago
The challenge most people have with using Linux AIO is that it is an "all or nothing" proposition, you can't sprinkle a little of it onto some existing code and get a good result. Using Linux AIO well entails writing a complete disk I/O implementation stack including your own disk cache infrastructure, I/O scheduler, etc with a high degree of technical competency and APIs to match. If you actually do all of this, Linux AIO is powerful and pleasant to use for high-performance disk access, if somewhat poorly documented. It works even better with raw block devices instead of files in filesystems -- you can pin some of its quirks on annoying filesystem behavior.
jpgvm 8y ago
Worth mentioning you can use AIO with epoll by using io_set_eventfd.

by adontz 8y ago

ahartmetz 8y ago

Apart from the ugly API (including no glibc support), the most ridiculous part is that it only works asynchronously with XFS and without caching! Without caching! In an API that is supposed to improve performance! I think we just need I/O readiness like epoll for disk I/O. AIO for disk seems to use the completion model ("do it and tell me when you're done" like Windows), which has higher maximum performance, but is more difficult to use / integrate. So you can now do both disk and network I/O with AIO... but it's going to be readiness based for network and completion based for disk. What the actual fuck. Edits: Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much (not necessary for network sockets), which is new API that you have to use. Reading must be done when the disk is ready and can't wait for you, so it needs another buffer and memcpy. Readiness based disk writing also kind of works and it's easier - the system tells you when the write buffer has room again (threshold selectable or fixed), but submitted data is not going to be written right away. This might be the semi-reasonable explanation for the strange O_DIRECT limitation of AIO - if you need async buffered writes, just let the write cache do its work (and, you know, randomly block sometimes - never mind reading). So readiness-based disk I/O could work with reasonably user-friendly new API for reading and not much worse performance loss than readiness-based network I/O. No idea about how much work it would take in the kernel. It would also be nice to have completion-based network I/O with AIO for symmetry reasons and in case you really need the best performance. Though I have never worked on something where completion-based network I/O would have made a meaningful performance difference.

phs2501 8y ago
The problem with readiness-based IO for disk is that without starting the request it doesn't know where it needs to read on the disk to fetch, so it can't possibly tell you if the data's ready. If you were to limit file IO to streaming rather than random-access it could potentially work, but that's not the POSIX interface. Since pipes and sockets are streams they don't have this issue. I'm not defending the suckiness of the AIO situation on Linux, but I understand why it is to some extent.
- ahartmetz 8y ago
  You're right, I was working on an edit to address the problem while you wrote your comment :)
- loeg 8y ago
  At a minimum it could tell you if the first byte at the fd offset is in cache, which it does not today. That'd at least get you a filesystem block (or more likely, 4k page). I agree this is less useful than, say, the full generality of NT IO completion.
  - the8472 8y ago
    Linux has fadvise to request an async read and preadv2(..., RWF_NOWAIT) to try to get things from the page cache. All that's missing now is a notification.
    - loeg 8y ago
      That's the key distinction between polling and select-like waiting.
quotemstr 8y ago
In the intended use case, the lack of kernel-level caching can be a performance win. A database can do a better job of caching than the kernel can because it has a clearer picture of the expected access patterns. O_DIRECT in this kind of application makes a lot of sense.
jandrewrogers 8y ago
A major reason to use Linux AIO is because it has no caching. That's a feature! A well-designed database engine can have a disk cache that outperforms the kernel-provided disk cache by a very large margin. Same story with I/O scheduling, which Linux AIO also lets you replace. Almost all the performance benefits that accrue from Linux AIO for disk access is that it allows you to replace the operating system cache and scheduler with something that is better designed for your application. And large gains are demonstrably possible. If you just wanted asynchronous disk calls, there are other APIs that use thread pools for that.
- loeg 8y ago
  A good model would have a flag that allowed databases to disable caching and everyone else to enable it, because not everything is a database engine. The absence of choice is not a feature.
  - ahartmetz 8y ago
    At least if both choices make sense, which they do.
    - loeg 8y ago
      Sure, I wasn't clear enough — I was implying that.
      - ahartmetz 8y ago
        You were clear enough, I was being redundant. Let's stop this here ;)
phs2501 8y ago
> Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much [...], which is new API that you have to use. Reading must be done when the disk is ready and can't wait for you, so it needs another buffer and memcpy. > So readiness-based disk I/O could work with reasonably user-friendly new API for reading and not much worse performance loss than readiness-based network I/O. Off the cuff that new API sounds exactly like a completion-signaled read. You tell the system up from where to read and how much (the async read call), it reads when the disk is ready and buffers for you (your passed buffer and/or the cache if you're clever), and then it signals "readiness" which is the same as completion in this case. Do you see this as distinct in any other way?
- ahartmetz 8y ago
  At their core, they are distinct only regarding when data is copied from kernel to user space. Readiness based copies when the application does the read() after being notified of "readiness", completion based copies (or DMAs directly from disk controller and maps into user space) before the application is notified of completion. The former is easier to use (no buffer ownership headaches), the latter is faster. Though there may be buffer management headaches in the kernel with readiness - what to do about ready reads the aren't collected? (Probably just throw them away if uncollected at the next AIO poll...)
loeg 8y ago
> Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much > It would also be nice to have completion-based network I/O with AIO for symmetry reasons I.e., you want something like NT IO completion ports, introduced in 1994.
- ahartmetz 8y ago
  > I.e., you want something like NT IO completion ports, introduced in 1994. Yes, that's why I mentioned Windows.

jandrewrogers 8y ago

The challenge most people have with using Linux AIO is that it is an "all or nothing" proposition, you can't sprinkle a little of it onto some existing code and get a good result. Using Linux AIO well entails writing a complete disk I/O implementation stack including your own disk cache infrastructure, I/O scheduler, etc with a high degree of technical competency and APIs to match. If you actually do all of this, Linux AIO is powerful and pleasant to use for high-performance disk access, if somewhat poorly documented. It works even better with raw block devices instead of files in filesystems -- you can pin some of its quirks on annoying filesystem behavior.

jpgvm 8y ago

Worth mentioning you can use AIO with epoll by using io_set_eventfd.