4 ms·

You can use io_uring with O_DIRECT, with the usual alignment restrictions. io_uring itself is carefully designed to eliminate copying even request/completion ev

You can use io_uring with O_DIRECT, with the usual alignment restrictions. io_uring itself is carefully designed to eliminate copying even request/completion event structures.

nextaccountic 5y ago
Thank you for the pointer, this seems very interesting. do you know of an example of open source code using io_uring and O_DIRECT to bypass the fs cache? What about posix_fadvise? I'm reading here https://unix.stackexchange.com/questions/6467/use-of-o-direct-on-linux https://unix.stackexchange.com/questions/6467/use-of-o-direc... that some people don't think it's a good idea to bypass the fs cache like that. Perhaps mmap could have been salvageable if it had a better error handling API.
- jasonwatkinspdx 5y ago
  I don't know an example of that sadly. O_DIRECT is definitely doing things the hard way for sure. There's a very real risk you won't be as smart as the kernel, even knowing your app's patterns well. Linus famously has ranted against this for a long time. But like mentioned above, with databases, taking on the complexity is often warranted. I've been tinkering off and on with a database design, and one of the things I want to explore is a hybrid approach where I still have an on heap buffer cache, but misses to that cache get filled out of mmap. The buffer cache is a lock free hash table, greatly simplified by using Copy on Write pages as a fundamental part of the design (don't have to worry about racing updates or duplication races during resizing other than as performance degradations). It's still not finished and totally untested, but the way I imagine it working is the buffer cache is smaller than what you'd typically reserve for a database, and the OS can manage its page cache as a sort of L2 cache.
  - nextaccountic 5y ago
    Thanks for sharing those ideas. This sounds interesting. Is it on github?
    - jasonwatkinspdx 5y ago
      Nope but I'm trying to herd my own cats to get a prototype shipped and up. It'll get shared here when I do for sure.
      - nextaccountic 5y ago
        Nice to hear, I hope it does well!

You can use io_uring with O_DIRECT, with the usual alignment restrictions. io_uring itself is carefully designed to eliminate copying even request/completion event structures.

nextaccountic 5y ago

Thank you for the pointer, this seems very interesting. do you know of an example of open source code using io_uring and O_DIRECT to bypass the fs cache? What about posix_fadvise? I'm reading here https://unix.stackexchange.com/questions/6467/use-of-o-direct-on-linux https://unix.stackexchange.com/questions/6467/use-of-o-direc... that some people don't think it's a good idea to bypass the fs cache like that. Perhaps mmap could have been salvageable if it had a better error handling API.

jasonwatkinspdx 5y ago
I don't know an example of that sadly. O_DIRECT is definitely doing things the hard way for sure. There's a very real risk you won't be as smart as the kernel, even knowing your app's patterns well. Linus famously has ranted against this for a long time. But like mentioned above, with databases, taking on the complexity is often warranted. I've been tinkering off and on with a database design, and one of the things I want to explore is a hybrid approach where I still have an on heap buffer cache, but misses to that cache get filled out of mmap. The buffer cache is a lock free hash table, greatly simplified by using Copy on Write pages as a fundamental part of the design (don't have to worry about racing updates or duplication races during resizing other than as performance degradations). It's still not finished and totally untested, but the way I imagine it working is the buffer cache is smaller than what you'd typically reserve for a database, and the OS can manage its page cache as a sort of L2 cache.
- nextaccountic 5y ago
  Thanks for sharing those ideas. This sounds interesting. Is it on github?
  - jasonwatkinspdx 5y ago
    Nope but I'm trying to herd my own cats to get a prototype shipped and up. It'll get shared here when I do for sure.
    - nextaccountic 5y ago
      Nice to hear, I hope it does well!