The things that want sync don't usually want a promise for future time, they want positive ack the data is on the storage. So a deadline approach is pointless.

The things that want sync don't usually want a promise for future time, they want positive ack the data is on the storage. So a deadline approach is pointless. Async w/O_DIRECT is the correct approach, despite the complexity. The problem is handling table scans that exploit system buffer cache for read-ahead. The answer to that is to issue more i/os to bigger buffers, or to a set of buffers. Scatter-gather disk i/o would help with that. It might be particularly sweet if well-behaved applications could rely on mixing O_DIRECT with bufferd i/o on the same files. That is, all writes done O_DIRECT on pages read with O_DIRECT, and reads might be done against page-cached buffered files.