4 ms·

Okay, here's a stab at it: • SIGIOFIN works like SIGIO except it fires when the IO operation completes (is on the disk; has left the network card, etc). I sho

by geocar 11y ago

Okay, here's a stab at it: • SIGIOFIN works like SIGIO except it fires when the IO operation completes (is on the disk; has left the network card, etc). I should be able to use signalfd() to collect these. Let me use TCP_DEFER_ACCEPT on regular file descriptors -- basically let me make sure there are n bytes before being woken up on a fifo, and n events in my signalfd() before waking up. Here's another idea: • all() returns a file descriptor that is initially readable. • on(POLLIN,f,d) while poll(f,POLLIN) would block, d is unreadable • POLLSYNC added to poll(), etc which is flagged when all previous writes to the file descriptor have reached the disk (has left the network card, etc). To get multiple checkpoints use dup(). I'm not sure which one I'd like better. If we didn't have to make it "primitive", we could simply copy what windows does[1], since they figured this out years ago. What do you think? [1]: https://msdn.microsoft.com/en-gb/library/windows/desktop/aa365198(v=vs.85).aspx https://msdn.microsoft.com/en-gb/library/windows/desktop/aa3...

gue5t 11y ago
I advise skipping the signal nonsense and making the interface purely fd-based. The only "advantage" of signals over fd readiness is that signals can interrupt syscalls that don't expect them, and this can be emulated without undue performance loss using asynchronous requests and epoll-style architectures. Signals cause a lot of pain in concurrent (e.g. coroutines and reentrancy) and multithreaded contexts. Plus, including data about which IO request finished and whether it succeeded or failed is more complicated with signals than simply including it in the event structs read from an fd. Another problem is that this doesn't really address the synchronous assumptions of filesystem (rather than file content) operations: applications expect that they can "mkdir(...); chdir(...); open(...);" and the filesystem doesn't know if the application would work properly with those operations reordered (it may or may not depending on the values of their arguments). Applications need to be more explicit with the transactional nature of what they're requesting from filesystems. Right now the filesystem tries to guess at the transaction structure based on fsync/fdatasync hints, and can't do as good a job scheduling as a result.
- skissane 11y ago
  > Applications need to be more explicit with the transactional nature of what they're requesting from filesystems. Right now the filesystem tries to guess at the transaction structure based on fsync/fdatasync hints, and can't do as good a job scheduling as a result. How about a transactional filesystem API, where applications can call an API saying "start transaction on current thread", do some IO operations, and then call one saying "end transaction on current thread" (or abort/rollback)? Windows has something like it: https://msdn.microsoft.com/en-us/library/windows/desktop/bb968806(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/bb9... Albeit, it is rarely used, and Microsoft suggests one shouldn't use it, and that maybe a future version of Windows might remove it. I think part of the reason it is rarely used is that it is a whole new API–for many of the existing filesystem APIs calls, they have new versions with "Transacted" appended to their names, and an additional transaction handle argument. If they'd made the current transaction implicit (i.e. thread-local) in the standard filesystem API, it might have seen more adoption.
zAy0LfpBZLC8mAC 11y ago
(1) This doesn't address ordering guarantees independently from durability guarantees (there are cases where you need to make sure writes don't get reordered, but durability right now is not required). (2) Signals are just a terrible interface, especially because they are process (or at least thread) global, so they are incredibly hard to use in libraries.
daurnimator 11y ago
Why not just a fd flag that makes it so `write` doesn't return until the write has hit the disk? Oh wait... isn't that O_SYNC/O_DSYNC? In which case, the problem is really using O_SYNC file descriptors asynchronously. Idea: Return EAGAIN if you try to write and the last write hasn't finished yet. + the file descriptor returns POLLOUT only when no writes are in flight. This unfortunately means you can't have multiple writes in flight. ----------------------- Wait, this is all too familiar.... I think this is solved by the linux kernel's existing async IO system. Relatively unknown is that you can pass an eventfd file descriptor to async io operations (see https://gist.github.com/daurnimator/778c36c2de32f8882d0d https://gist.github.com/daurnimator/778c36c2de32f8882d0d for an example). With IOCB_CMD_FSYNC + IOCB_CMD_FDSYNC we should have everything we need? I wonder if O_SYNC works with IOCB_CMD_PWRITE? The kernel aio interface is not wrapped by libc, only works on some file systems, has some odd-er alignment+size requirements and only supported by linux... though posix aio may be similar enough, I'd need to check. ------------------------ I did just find http://lkml.iu.edu/hypermail/linux/kernel/1601.1/02634.html http://lkml.iu.edu/hypermail/linux/kernel/1601.1/02634.html which seems like an interesting development in this area.
- gue5t 11y ago
  Kernel AIO is virtually unusable for anything other than databases because of all the restrictions you mention, and it can still block in many cases (some of which are runtime conditions like "too many outstanding requests"--no, it doesn't give EAGAIN or such for that; it just blocks). And it doesn't do filesystem operations, only file content operations. POSIX AIO is backed by a thread pool in glibc.
  - daurnimator 11y ago
    yes... but it does possibly direct us to where the improvements need to happen.
  - zvrba 11y ago
    The nout about aio in glibc documentation says the following: "The current Linux POSIX AIO implementation is provided in user space by glibc. This has a number of limitations, most notably that maintaining multiple threads to perform I/O operations is expensive and scales poorly." Why does it scale poorly? It shouldn't given a decent thread-pool implementation and lock-free queueing.

by geocar 11y ago

gue5t 11y ago

I advise skipping the signal nonsense and making the interface purely fd-based. The only "advantage" of signals over fd readiness is that signals can interrupt syscalls that don't expect them, and this can be emulated without undue performance loss using asynchronous requests and epoll-style architectures. Signals cause a lot of pain in concurrent (e.g. coroutines and reentrancy) and multithreaded contexts. Plus, including data about which IO request finished and whether it succeeded or failed is more complicated with signals than simply including it in the event structs read from an fd. Another problem is that this doesn't really address the synchronous assumptions of filesystem (rather than file content) operations: applications expect that they can "mkdir(...); chdir(...); open(...);" and the filesystem doesn't know if the application would work properly with those operations reordered (it may or may not depending on the values of their arguments). Applications need to be more explicit with the transactional nature of what they're requesting from filesystems. Right now the filesystem tries to guess at the transaction structure based on fsync/fdatasync hints, and can't do as good a job scheduling as a result.

skissane 11y ago
> Applications need to be more explicit with the transactional nature of what they're requesting from filesystems. Right now the filesystem tries to guess at the transaction structure based on fsync/fdatasync hints, and can't do as good a job scheduling as a result. How about a transactional filesystem API, where applications can call an API saying "start transaction on current thread", do some IO operations, and then call one saying "end transaction on current thread" (or abort/rollback)? Windows has something like it: https://msdn.microsoft.com/en-us/library/windows/desktop/bb968806(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/bb9... Albeit, it is rarely used, and Microsoft suggests one shouldn't use it, and that maybe a future version of Windows might remove it. I think part of the reason it is rarely used is that it is a whole new API–for many of the existing filesystem APIs calls, they have new versions with "Transacted" appended to their names, and an additional transaction handle argument. If they'd made the current transaction implicit (i.e. thread-local) in the standard filesystem API, it might have seen more adoption.

zAy0LfpBZLC8mAC 11y ago

(1) This doesn't address ordering guarantees independently from durability guarantees (there are cases where you need to make sure writes don't get reordered, but durability right now is not required). (2) Signals are just a terrible interface, especially because they are process (or at least thread) global, so they are incredibly hard to use in libraries.

daurnimator 11y ago

Why not just a fd flag that makes it so `write` doesn't return until the write has hit the disk? Oh wait... isn't that O_SYNC/O_DSYNC? In which case, the problem is really using O_SYNC file descriptors asynchronously. Idea: Return EAGAIN if you try to write and the last write hasn't finished yet. + the file descriptor returns POLLOUT only when no writes are in flight. This unfortunately means you can't have multiple writes in flight. ----------------------- Wait, this is all too familiar.... I think this is solved by the linux kernel's existing async IO system. Relatively unknown is that you can pass an eventfd file descriptor to async io operations (see https://gist.github.com/daurnimator/778c36c2de32f8882d0d https://gist.github.com/daurnimator/778c36c2de32f8882d0d for an example). With IOCB_CMD_FSYNC + IOCB_CMD_FDSYNC we should have everything we need? I wonder if O_SYNC works with IOCB_CMD_PWRITE? The kernel aio interface is not wrapped by libc, only works on some file systems, has some odd-er alignment+size requirements and only supported by linux... though posix aio may be similar enough, I'd need to check. ------------------------ I did just find http://lkml.iu.edu/hypermail/linux/kernel/1601.1/02634.html http://lkml.iu.edu/hypermail/linux/kernel/1601.1/02634.html which seems like an interesting development in this area.

gue5t 11y ago
Kernel AIO is virtually unusable for anything other than databases because of all the restrictions you mention, and it can still block in many cases (some of which are runtime conditions like "too many outstanding requests"--no, it doesn't give EAGAIN or such for that; it just blocks). And it doesn't do filesystem operations, only file content operations. POSIX AIO is backed by a thread pool in glibc.
- daurnimator 11y ago
  yes... but it does possibly direct us to where the improvements need to happen.
- zvrba 11y ago
  The nout about aio in glibc documentation says the following: "The current Linux POSIX AIO implementation is provided in user space by glibc. This has a number of limitations, most notably that maintaining multiple threads to perform I/O operations is expensive and scales poorly." Why does it scale poorly? It shouldn't given a decent thread-pool implementation and lock-free queueing.