3 ms·
Why not just a fd flag that makes it so `write` doesn't return until the write has hit the disk? Oh wait... isn't that O_SYNC/O_DSYNC? In which case, the probl
by daurnimator 11y ago
Why not just a fd flag that makes it so `write` doesn't return until the write has hit the disk?
Oh wait... isn't that O_SYNC/O_DSYNC? In which case, the problem is really using O_SYNC file descriptors asynchronously.
Idea:
Return EAGAIN if you try to write and the last write hasn't finished yet. + the file descriptor returns POLLOUT only when no writes are in flight.
This unfortunately means you can't have multiple writes in flight.
-----------------------
Wait, this is all too familiar....
I think this is solved by the linux kernel's existing async IO system.
Relatively unknown is that you can pass an eventfd file descriptor to async io operations (see https://gist.github.com/daurnimator/778c36c2de32f8882d0d https://gist.github.com/daurnimator/778c36c2de32f8882d0d for an example).
With IOCB_CMD_FSYNC + IOCB_CMD_FDSYNC we should have everything we need?
I wonder if O_SYNC works with IOCB_CMD_PWRITE?
The kernel aio interface is not wrapped by libc, only works on some file systems, has some odd-er alignment+size requirements and only supported by linux... though posix aio may be similar enough, I'd need to check.
------------------------
I did just find http://lkml.iu.edu/hypermail/linux/kernel/1601.1/02634.html http://lkml.iu.edu/hypermail/linux/kernel/1601.1/02634.html which seems like an interesting development in this area.
- gue5t 11y agoKernel AIO is virtually unusable for anything other than databases because of all the restrictions you mention, and it can still block in many cases (some of which are runtime conditions like "too many outstanding requests"--no, it doesn't give EAGAIN or such for that; it just blocks). And it doesn't do filesystem operations, only file content operations. POSIX AIO is backed by a thread pool in glibc.
- daurnimator 11y agoyes... but it does possibly direct us to where the improvements need to happen.
- zvrba 11y agoThe nout about aio in glibc documentation says the following: "The current Linux POSIX AIO implementation is provided in user space by glibc. This has a number of limitations, most notably that maintaining multiple threads to perform I/O operations is expensive and scales poorly." Why does it scale poorly? It shouldn't given a decent thread-pool implementation and lock-free queueing.
- mtanski 11y agoIt scales poorly because the underlying queue is not lock-free. It was designed in a much simpler time or slower drives (spinning disks). Even if you make it lock-free you still end up paying the communication cost (in/notification). Any time you move data to a different thread that's going to introduce latency (which matters on modern fast drives). Most of these thread pool implementations also suffer head of line blocking... eg. small and fast (page cache) requests get stuck behind long and slow requests. I know because, I've been fighting this stuff for all of my career. I've recently (in the last two years) tried to push upstream to Linux an a non-blocking page cache read. Thus you have a short circuit case for skipping the queue. In my tests it made samba file sharing ~ 23% faster. https://lwn.net/Articles/612483/ https://lwn.net/Articles/612483/ https://lwn.net/Articles/636967/ https://lwn.net/Articles/636967/ https://lwn.net/Articles/670231/ https://lwn.net/Articles/670231/ The preadv2 patches stalled... but Christoph revived them and now they are in the kernel. So what's need now is RWF_NONBLOCK.