3 ms·

Kernel AIO is virtually unusable for anything other than databases because of all the restrictions you mention, and it can still block in many cases (some of wh

by gue5t 10y ago

Kernel AIO is virtually unusable for anything other than databases because of all the restrictions you mention, and it can still block in many cases (some of which are runtime conditions like "too many outstanding requests"--no, it doesn't give EAGAIN or such for that; it just blocks). And it doesn't do filesystem operations, only file content operations. POSIX AIO is backed by a thread pool in glibc.

daurnimator 10y ago
yes... but it does possibly direct us to where the improvements need to happen.
zvrba 10y ago
The nout about aio in glibc documentation says the following: "The current Linux POSIX AIO implementation is provided in user space by glibc. This has a number of limitations, most notably that maintaining multiple threads to perform I/O operations is expensive and scales poorly." Why does it scale poorly? It shouldn't given a decent thread-pool implementation and lock-free queueing.
- mtanski 10y ago
  It scales poorly because the underlying queue is not lock-free. It was designed in a much simpler time or slower drives (spinning disks). Even if you make it lock-free you still end up paying the communication cost (in/notification). Any time you move data to a different thread that's going to introduce latency (which matters on modern fast drives). Most of these thread pool implementations also suffer head of line blocking... eg. small and fast (page cache) requests get stuck behind long and slow requests. I know because, I've been fighting this stuff for all of my career. I've recently (in the last two years) tried to push upstream to Linux an a non-blocking page cache read. Thus you have a short circuit case for skipping the queue. In my tests it made samba file sharing ~ 23% faster. https://lwn.net/Articles/612483/ https://lwn.net/Articles/612483/ https://lwn.net/Articles/636967/ https://lwn.net/Articles/636967/ https://lwn.net/Articles/670231/ https://lwn.net/Articles/670231/ The preadv2 patches stalled... but Christoph revived them and now they are in the kernel. So what's need now is RWF_NONBLOCK.

by gue5t 10y ago

daurnimator 10y ago

yes... but it does possibly direct us to where the improvements need to happen.

zvrba 10y ago

The nout about aio in glibc documentation says the following: "The current Linux POSIX AIO implementation is provided in user space by glibc. This has a number of limitations, most notably that maintaining multiple threads to perform I/O operations is expensive and scales poorly." Why does it scale poorly? It shouldn't given a decent thread-pool implementation and lock-free queueing.

mtanski 10y ago
It scales poorly because the underlying queue is not lock-free. It was designed in a much simpler time or slower drives (spinning disks). Even if you make it lock-free you still end up paying the communication cost (in/notification). Any time you move data to a different thread that's going to introduce latency (which matters on modern fast drives). Most of these thread pool implementations also suffer head of line blocking... eg. small and fast (page cache) requests get stuck behind long and slow requests. I know because, I've been fighting this stuff for all of my career. I've recently (in the last two years) tried to push upstream to Linux an a non-blocking page cache read. Thus you have a short circuit case for skipping the queue. In my tests it made samba file sharing ~ 23% faster. https://lwn.net/Articles/612483/ https://lwn.net/Articles/612483/ https://lwn.net/Articles/636967/ https://lwn.net/Articles/636967/ https://lwn.net/Articles/670231/ https://lwn.net/Articles/670231/ The preadv2 patches stalled... but Christoph revived them and now they are in the kernel. So what's need now is RWF_NONBLOCK.