The problem with O_DIRECT is it pretty much puts each app in the business of doing its own buffer cache, which bypasses the ability of the kernel to look at the

The problem with O_DIRECT is it pretty much puts each app in the business of doing its own buffer cache, which bypasses the ability of the kernel to look at the system holistically and make decisions about how to buffer data. Compound that with fairly inconsistent contracts around the interface and it's semi-synchronous behaviour... ick. As to improve fadvise()? I'd like to see FADV_SEQUENTIAL (or perhaps a variant) not just double the read ahead buffer, but also dump pages immediately after they've been read unless there is another FD open somewhere else (you can keep calling fadvise with FADV_DONTNEED, but that's lame on several levels). I'd like to see semantics that make it clear to the kernel that data you are writing to a file (particularly if it is in append mode) likely won't be read for a very long time, so it can minimize polluting the buffer cache with freshly written data. I'd like to see fadvise() calls that specify a portion of a file only effect the portion of the file. What'd be REALLY nice would be a way to express "buffer part X and Y of the file, but if you are under pressure, dump Y before you dump X".