4 ms·

Atomic I/O operation support can be taken a step further, though, by adding asynchronous I/O (AIO) support. The nice thing about the Linux AIO interface (which

Atomic I/O operation support can be taken a step further, though, by adding asynchronous I/O (AIO) support. The nice thing about the Linux AIO interface (which is not generally acknowledged to have many nice aspects) is that it allows an application to enqueue multiple I/O operations with a single system call. I'm not familiar with the AIO interface, but it sounds fairly handy. Could someone comment on what he means by it not having 'many nice aspects'?

benjarrell 13y ago
A good overview is here: https://code.google.com/p/kernel/wiki/AIOUserGuide https://code.google.com/p/kernel/wiki/AIOUserGuide Unless a program is designed to use it, it won't be. It doesn't work on sockets or pipes and filesystem support is limited. The only things I have seen that can use it are MySQL, DB2, and Oracle.
paraboul 13y ago
AIO still "sux" (e.g. doesn't support async open()'ing). What we need is non-blocking flag similar to what BSD socket offers and then use whatever I/O notification API you want (epoll).
wmf 13y ago
The biggest issue with AIO is that file reads aren't actually async unless you use O_DIRECT which has performance implications (sometimes it's faster but sometimes it's slower). O_DIRECT also requires alignment which makes it harder to use.
jandrewrogers 13y ago
The Linux AIO interface is excellent for high-performance disk I/O but using it requires a much bigger code investment than is immediately implied by the AIO API by itself. The complexity of using it is that for it to work properly, you need to bypass most blocking and buffering operations in the kernel related to file system I/O. Specifically: - You should always use O_DIRECT or it defeats the purpose. Consequently, the caching and buffering you normally get for "free" using more conventional approaches is not available. For good performance you have to write your own disk buffer cache implementation which is not trivial if you have never done it before. This buffer code will exceed the size of the AIO code if you are doing it well. On the upside, for many purposes you can design a buffer cache that greatly outperforms the standard kernel implementation. - You need to minimize the number of file system metadata operations that can block and/or bypass the file system altogether. I recommend the former. In practical systems, you end up doing very coarse file system operations, like allocating a huge chunks of disk, and then building your own lightweight disk management system on top of that. Again, this is a lot of extra work for the sake of using AIO effectively. - You need to figure out the best way to schedule all of the disk I/O operations you are now responsible for. Background writing, prefetching, etc that are used to optimize disk performance are now part of your implementation. Linux AIO is simple and pretty nice in the abstract. Using it well requires some complex domain expertise around scheduling, file organization, and cache replacement algorithms and techniques. There are no libraries that provide quality implementations of the supporting infrastructure that I am aware of so you have to roll your own. You can't do just a little bit of AIO and see much benefit in most cases. There are big benefits though which is why advanced database engines are implemented this way. If you build out an entire userspace disk I/O subsystem on top Linux AIO you can easily double your disk I/O performance or more relative to using the conventional POSIX, mmap(), etc interfaces. Furthermore, it often can be designed to degrade performance more gracefully during periods of high I/O pressure.

benjarrell 13y ago

A good overview is here: https://code.google.com/p/kernel/wiki/AIOUserGuide https://code.google.com/p/kernel/wiki/AIOUserGuide Unless a program is designed to use it, it won't be. It doesn't work on sockets or pipes and filesystem support is limited. The only things I have seen that can use it are MySQL, DB2, and Oracle.

paraboul 13y ago

AIO still "sux" (e.g. doesn't support async open()'ing). What we need is non-blocking flag similar to what BSD socket offers and then use whatever I/O notification API you want (epoll).

wmf 13y ago

The biggest issue with AIO is that file reads aren't actually async unless you use O_DIRECT which has performance implications (sometimes it's faster but sometimes it's slower). O_DIRECT also requires alignment which makes it harder to use.

jandrewrogers 13y ago

The Linux AIO interface is excellent for high-performance disk I/O but using it requires a much bigger code investment than is immediately implied by the AIO API by itself. The complexity of using it is that for it to work properly, you need to bypass most blocking and buffering operations in the kernel related to file system I/O. Specifically: - You should always use O_DIRECT or it defeats the purpose. Consequently, the caching and buffering you normally get for "free" using more conventional approaches is not available. For good performance you have to write your own disk buffer cache implementation which is not trivial if you have never done it before. This buffer code will exceed the size of the AIO code if you are doing it well. On the upside, for many purposes you can design a buffer cache that greatly outperforms the standard kernel implementation. - You need to minimize the number of file system metadata operations that can block and/or bypass the file system altogether. I recommend the former. In practical systems, you end up doing very coarse file system operations, like allocating a huge chunks of disk, and then building your own lightweight disk management system on top of that. Again, this is a lot of extra work for the sake of using AIO effectively. - You need to figure out the best way to schedule all of the disk I/O operations you are now responsible for. Background writing, prefetching, etc that are used to optimize disk performance are now part of your implementation. Linux AIO is simple and pretty nice in the abstract. Using it well requires some complex domain expertise around scheduling, file organization, and cache replacement algorithms and techniques. There are no libraries that provide quality implementations of the supporting infrastructure that I am aware of so you have to roll your own. You can't do just a little bit of AIO and see much benefit in most cases. There are big benefits though which is why advanced database engines are implemented this way. If you build out an entire userspace disk I/O subsystem on top Linux AIO you can easily double your disk I/O performance or more relative to using the conventional POSIX, mmap(), etc interfaces. Furthermore, it often can be designed to degrade performance more gracefully during periods of high I/O pressure.