9 ms·

Io_submit and Linux AIO – An epoll alternative

121 points by majke 8y ago

majke 8y ago
I’m trying to convince Antirez to use this technique in redis: https://twitter.com/antirez/status/1081197002573139968 https://twitter.com/antirez/status/1081197002573139968 > There was a problem with that… I don’t remember what exactly, but it was like, the structures you fill did not match how Redis stored the data or something. I need to try again soon or later because the speedup in Redis would be huge.
- hoytech 8y ago
  Hi Marek, great article as usual. > (2) Pre-warm the disk cache with posix_fadvise(2) and hope for the best. Just as an aside, there is also the option of mlock(2) to improve this technique as implemented in the flash webserver (https://www.usenix.org/legacy/events/usenix99/full_papers/pai/pai.pdf https://www.usenix.org/legacy/events/usenix99/full_papers/pa...). Also some people use vmtouch (https://hoytech.com/vmtouch/ https://hoytech.com/vmtouch/) for similar things. Regarding this new interface, the LWN article says: > AIO poll operations always operate in the "one-shot" mode; once a poll notification has been generated, a new IOCB_CMD_POLL IOCB must be submitted for that file descriptor if further notifications are needed. Does this mean that in the (I think fairly common) case of wanting to persist a poll indefinitely until (for example) a socket is closed, with this new interface you will need to re-issue a poll after every IO operation? If so would that mitigate some/any of the performance benefit you described? Lastly just wanted to point out the following research in case you haven't seen it before (and the various other designs it cites): https://www.usenix.org/legacy/event/atc11/tech/final_files/Soares.pdf https://www.usenix.org/legacy/event/atc11/tech/final_files/S... Cheers!
- drewg123 8y ago
  I wonder if Linux has considered the technique that FreeBSD uses for async sendfile? The FreeBSD sendfile() syscall is async (mostly; there is a potential to block reading metadata in VOP_BMAP). It launches a request to read disk blocks into pages, and attaches the pages to the socket buffer via mbufs, but marks them M_NOTREADY so that the protocol layer cannot send them. When the disk I/O completes, the mbufs are marked ready in the context of the disk interrupt thread, and then the proto output routine (eg, tcp_output()) is called. This is more-or-less fire and forget for nginx. (yes, it does check the status via kqueue, but it does not block) This was a huge performance win for us at Netflix, and the not-ready concept is the basis for our kernel-tls layer (mbufs are left not-ready, and encrypted after the disk read, and then the crypto thread calls the proto output).
  - khc 8y ago
    at cloudflare, we mostly don't stream large files so this is probably not a big win in our case
  - loeg 8y ago
    async sendfile is highly specialized for ... sendfile. It'd be nice to have a more general approach in FreeBSD as well as Linux. Honestly, NT has a great model here we could adopt.
adontz 8y ago
After I've read the article which kind of praises AIO, I have impression that AIO is misdesigned, have misleading name and overall quite terrible. I better stay away from it. Also, epoll is good enough for me, my tasks. So no actual need to jump on the code to replace my epoll event loops.
- ahartmetz 8y ago
  Apart from the ugly API (including no glibc support), the most ridiculous part is that it only works asynchronously with XFS and without caching! Without caching! In an API that is supposed to improve performance! I think we just need I/O readiness like epoll for disk I/O. AIO for disk seems to use the completion model ("do it and tell me when you're done" like Windows), which has higher maximum performance, but is more difficult to use / integrate. So you can now do both disk and network I/O with AIO... but it's going to be readiness based for network and completion based for disk. What the actual fuck. Edits: Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much (not necessary for network sockets), which is new API that you have to use. Reading must be done when the disk is ready and can't wait for you, so it needs another buffer and memcpy. Readiness based disk writing also kind of works and it's easier - the system tells you when the write buffer has room again (threshold selectable or fixed), but submitted data is not going to be written right away. This might be the semi-reasonable explanation for the strange O_DIRECT limitation of AIO - if you need async buffered writes, just let the write cache do its work (and, you know, randomly block sometimes - never mind reading). So readiness-based disk I/O could work with reasonably user-friendly new API for reading and not much worse performance loss than readiness-based network I/O. No idea about how much work it would take in the kernel. It would also be nice to have completion-based network I/O with AIO for symmetry reasons and in case you really need the best performance. Though I have never worked on something where completion-based network I/O would have made a meaningful performance difference.
  - phs2501 8y ago
    The problem with readiness-based IO for disk is that without starting the request it doesn't know where it needs to read on the disk to fetch, so it can't possibly tell you if the data's ready. If you were to limit file IO to streaming rather than random-access it could potentially work, but that's not the POSIX interface. Since pipes and sockets are streams they don't have this issue. I'm not defending the suckiness of the AIO situation on Linux, but I understand why it is to some extent.
quotemstr 8y ago
The lack of clear vision in Linux system call design is a source of continual frustration. I feel like any time I or anyone else proposes generic ways to solve certain longstanding problems, the response is a hostile sort of "no" based on, ostensibly, lack of demonstrated use cases for genericity, but in actuality, I think, simple change aversion. The result is an accumulation of a pile of ad-hoc kludges and crud that do one thing or not the other (e.g., the batch-but-not-async and async-but-not-batch distinction the article mentions). There needs to be more openness to providing general interfaces that solve whole classes of interface problems all at once, and without the technical conservatism that confronts people trying to contribute new interfaces.
- cyphar 8y ago
  I think this is based on a fear of having an API that wasn't needed and needs to be maintained forever. I bet that Linux's syscall design would be much nicer if iterating on APIs was possible (as we see in other Unix-likes). My personally least favourite syscall design misstep is how VFS flags work. AT_ flags were supposed to be generic but we now have AT_STATX_, O_ flags are overused, and some at(2) syscalls like renameat2(2) have their own brand of flags (RENAME_). Internally VFS has uniform flags for lookup and other various things, so it's a bit silly that these flags are all basically incompatible. (Also, the fact that O_RDWR != O_RDONLY|O_WRONLY is just silly. It takes up just as many bytes but makes life needlessly difficult when designing new extensions to openat(2).)
  - quotemstr 8y ago
    If I had my way, the application stable ABI would be libc, not SYSENTER. Windows gets this right and is able to iterate faster on kernel interfaces as a result. A while ago, I proposed a kernel supplied libc add-on that would provide the syscall wrappers that glibc won't. Maybe we could declare that all applications must make new system calls through this interface and maintain syscall compatibility only for the current legacy set of system calls.
    - jcranmer 8y ago
polskibus 8y ago
Can anyone compare this to IO Completion Ports on Windows?
- loeg 8y ago
  tl;dr worse in every way
  - loeg 8y ago
    Looks like 3-4 Linux-cannot-be-worse-than-NT-in-any-way apologists here to deny facts. The NT IO completion port model has handle asynchronous IO, to disk and network, since NT 3.5 in the mid-90s. It has none of the problems of (1) accidental fd reuse due to compact range; (2) synchronous-by-default IO APIs like read() and write(); (3) inelegant AIO models that require disabling caching. There are definitely areas Linux wins at (path lookup; metadata operations; cost, license, and open source nature) — but AIO to files is not one of them. I'm an open source user and developer, but we could stand to learn some lessons from NT rather than shove our fingers in our collective ears, as it seems some people are doing.
    - Arnavion 8y ago
      >Looks like 3-4 Linux-cannot-be-worse-than-NT-in-any-way apologists here to deny facts. Your first comment would've benefitted from containing the information from your second comment, rather than a content-free dismissal.
    - rossy 8y ago
      I agree in principle. NT IO is way more elegant than Linux IO, especially when it comes to AIO, and especially AIO to disk files. I don't agree with your examples though. I think NT has those flaws as well. Accidental handle reuse in NT seems just as easy as fd reuse in Linux. Internally, NT handles have the first two bits cleared, and the rest are an index into a handle table that is allocated sequentially (handles look like 0x4, 0x8, 0xc, 0x10, etc.) When allocating a new handle, NT doesn't strictly reuse the lowest unused value like Linux/Unix, but it seems to prefer to reuse handles rather than expanding the handle table, so handle values are reused often. And yeah, Linux file descriptions are synchronous by default, but so are NT file objects, and unlike Linux, an NT file object that was opened in synchronous mode can't be changed to asynchronous mode. This can make it annoying to write programs and libraries that work with arbitrary handles, since even if all your handles are asynchronous, handles created out of your control might not be. stdin, stdout and inherited CRT fds are all synchronous by convention. This combined with the fact that asynchronous handles can act like synchronous handles for various reasons (eg. NTFS compression/encryption, resource contention in the cache manager,) means you have to write asynchronous code with the assumption that every IO operation on a particular handle will always block. I've heard this is why libuv chooses to use synchronous IO with file handles on Windows, even though asynchronous IO is possible. So we could stand to learn some lessons from NT, but Linux IO isn't "worse in every way." I think an operating system with a good IO/AIO system would have to leapfrog both NT and Linux and aim for something better.

121 points by majke 8y ago

I’m trying to convince Antirez to use this technique in redis: https://twitter.com/antirez/status/1081197002573139968 https://twitter.com/antirez/status/1081197002573139968 > There was a problem with that… I don’t remember what exactly, but it was like, the structures you fill did not match how Redis stored the data or something. I need to try again soon or later because the speedup in Redis would be huge.

hoytech 8y ago
Hi Marek, great article as usual. > (2) Pre-warm the disk cache with posix_fadvise(2) and hope for the best. Just as an aside, there is also the option of mlock(2) to improve this technique as implemented in the flash webserver (https://www.usenix.org/legacy/events/usenix99/full_papers/pai/pai.pdf https://www.usenix.org/legacy/events/usenix99/full_papers/pa...). Also some people use vmtouch (https://hoytech.com/vmtouch/ https://hoytech.com/vmtouch/) for similar things. Regarding this new interface, the LWN article says: > AIO poll operations always operate in the "one-shot" mode; once a poll notification has been generated, a new IOCB_CMD_POLL IOCB must be submitted for that file descriptor if further notifications are needed. Does this mean that in the (I think fairly common) case of wanting to persist a poll indefinitely until (for example) a socket is closed, with this new interface you will need to re-issue a poll after every IO operation? If so would that mitigate some/any of the performance benefit you described? Lastly just wanted to point out the following research in case you haven't seen it before (and the various other designs it cites): https://www.usenix.org/legacy/event/atc11/tech/final_files/Soares.pdf https://www.usenix.org/legacy/event/atc11/tech/final_files/S... Cheers!
drewg123 8y ago
I wonder if Linux has considered the technique that FreeBSD uses for async sendfile? The FreeBSD sendfile() syscall is async (mostly; there is a potential to block reading metadata in VOP_BMAP). It launches a request to read disk blocks into pages, and attaches the pages to the socket buffer via mbufs, but marks them M_NOTREADY so that the protocol layer cannot send them. When the disk I/O completes, the mbufs are marked ready in the context of the disk interrupt thread, and then the proto output routine (eg, tcp_output()) is called. This is more-or-less fire and forget for nginx. (yes, it does check the status via kqueue, but it does not block) This was a huge performance win for us at Netflix, and the not-ready concept is the basis for our kernel-tls layer (mbufs are left not-ready, and encrypted after the disk read, and then the crypto thread calls the proto output).
- khc 8y ago
  at cloudflare, we mostly don't stream large files so this is probably not a big win in our case
- loeg 8y ago
  async sendfile is highly specialized for ... sendfile. It'd be nice to have a more general approach in FreeBSD as well as Linux. Honestly, NT has a great model here we could adopt.

adontz 8y ago

After I've read the article which kind of praises AIO, I have impression that AIO is misdesigned, have misleading name and overall quite terrible. I better stay away from it. Also, epoll is good enough for me, my tasks. So no actual need to jump on the code to replace my epoll event loops.

ahartmetz 8y ago
Apart from the ugly API (including no glibc support), the most ridiculous part is that it only works asynchronously with XFS and without caching! Without caching! In an API that is supposed to improve performance! I think we just need I/O readiness like epoll for disk I/O. AIO for disk seems to use the completion model ("do it and tell me when you're done" like Windows), which has higher maximum performance, but is more difficult to use / integrate. So you can now do both disk and network I/O with AIO... but it's going to be readiness based for network and completion based for disk. What the actual fuck. Edits: Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much (not necessary for network sockets), which is new API that you have to use. Reading must be done when the disk is ready and can't wait for you, so it needs another buffer and memcpy. Readiness based disk writing also kind of works and it's easier - the system tells you when the write buffer has room again (threshold selectable or fixed), but submitted data is not going to be written right away. This might be the semi-reasonable explanation for the strange O_DIRECT limitation of AIO - if you need async buffered writes, just let the write cache do its work (and, you know, randomly block sometimes - never mind reading). So readiness-based disk I/O could work with reasonably user-friendly new API for reading and not much worse performance loss than readiness-based network I/O. No idea about how much work it would take in the kernel. It would also be nice to have completion-based network I/O with AIO for symmetry reasons and in case you really need the best performance. Though I have never worked on something where completion-based network I/O would have made a meaningful performance difference.
- phs2501 8y ago
  The problem with readiness-based IO for disk is that without starting the request it doesn't know where it needs to read on the disk to fetch, so it can't possibly tell you if the data's ready. If you were to limit file IO to streaming rather than random-access it could potentially work, but that's not the POSIX interface. Since pipes and sockets are streams they don't have this issue. I'm not defending the suckiness of the AIO situation on Linux, but I understand why it is to some extent.

quotemstr 8y ago

The lack of clear vision in Linux system call design is a source of continual frustration. I feel like any time I or anyone else proposes generic ways to solve certain longstanding problems, the response is a hostile sort of "no" based on, ostensibly, lack of demonstrated use cases for genericity, but in actuality, I think, simple change aversion. The result is an accumulation of a pile of ad-hoc kludges and crud that do one thing or not the other (e.g., the batch-but-not-async and async-but-not-batch distinction the article mentions). There needs to be more openness to providing general interfaces that solve whole classes of interface problems all at once, and without the technical conservatism that confronts people trying to contribute new interfaces.

cyphar 8y ago
I think this is based on a fear of having an API that wasn't needed and needs to be maintained forever. I bet that Linux's syscall design would be much nicer if iterating on APIs was possible (as we see in other Unix-likes). My personally least favourite syscall design misstep is how VFS flags work. AT_ flags were supposed to be generic but we now have AT_STATX_, O_ flags are overused, and some at(2) syscalls like renameat2(2) have their own brand of flags (RENAME_). Internally VFS has uniform flags for lookup and other various things, so it's a bit silly that these flags are all basically incompatible. (Also, the fact that O_RDWR != O_RDONLY|O_WRONLY is just silly. It takes up just as many bytes but makes life needlessly difficult when designing new extensions to openat(2).)
- quotemstr 8y ago
  If I had my way, the application stable ABI would be libc, not SYSENTER. Windows gets this right and is able to iterate faster on kernel interfaces as a result. A while ago, I proposed a kernel supplied libc add-on that would provide the syscall wrappers that glibc won't. Maybe we could declare that all applications must make new system calls through this interface and maintain syscall compatibility only for the current legacy set of system calls.
  - jcranmer 8y ago

polskibus 8y ago

Can anyone compare this to IO Completion Ports on Windows?

loeg 8y ago
tl;dr worse in every way
- loeg 8y ago
  Looks like 3-4 Linux-cannot-be-worse-than-NT-in-any-way apologists here to deny facts. The NT IO completion port model has handle asynchronous IO, to disk and network, since NT 3.5 in the mid-90s. It has none of the problems of (1) accidental fd reuse due to compact range; (2) synchronous-by-default IO APIs like read() and write(); (3) inelegant AIO models that require disabling caching. There are definitely areas Linux wins at (path lookup; metadata operations; cost, license, and open source nature) — but AIO to files is not one of them. I'm an open source user and developer, but we could stand to learn some lessons from NT rather than shove our fingers in our collective ears, as it seems some people are doing.
  - Arnavion 8y ago
    >Looks like 3-4 Linux-cannot-be-worse-than-NT-in-any-way apologists here to deny facts. Your first comment would've benefitted from containing the information from your second comment, rather than a content-free dismissal.
  - rossy 8y ago
    I agree in principle. NT IO is way more elegant than Linux IO, especially when it comes to AIO, and especially AIO to disk files. I don't agree with your examples though. I think NT has those flaws as well. Accidental handle reuse in NT seems just as easy as fd reuse in Linux. Internally, NT handles have the first two bits cleared, and the rest are an index into a handle table that is allocated sequentially (handles look like 0x4, 0x8, 0xc, 0x10, etc.) When allocating a new handle, NT doesn't strictly reuse the lowest unused value like Linux/Unix, but it seems to prefer to reuse handles rather than expanding the handle table, so handle values are reused often. And yeah, Linux file descriptions are synchronous by default, but so are NT file objects, and unlike Linux, an NT file object that was opened in synchronous mode can't be changed to asynchronous mode. This can make it annoying to write programs and libraries that work with arbitrary handles, since even if all your handles are asynchronous, handles created out of your control might not be. stdin, stdout and inherited CRT fds are all synchronous by convention. This combined with the fact that asynchronous handles can act like synchronous handles for various reasons (eg. NTFS compression/encryption, resource contention in the cache manager,) means you have to write asynchronous code with the assumption that every IO operation on a particular handle will always block. I've heard this is why libuv chooses to use synchronous IO with file handles on Windows, even though asynchronous IO is possible. So we could stand to learn some lessons from NT, but Linux IO isn't "worse in every way." I think an operating system with a good IO/AIO system would have to leapfrog both NT and Linux and aim for something better.