2 ms·

O_DiRECT is insufficient because you have no way to coordinate with the kernel to drop io buffers when there’s memory pressure

ericbarrett 3y ago
This is commonly avoided by not putting memory pressure on your O_DIRECT application, which is probably an RDBMS on a dedicated single-purpose host. Not saying more work in the space wouldn't be welcome and interesting, just that it's generally the assumption whenever I've seen O_DIRECT (since the Solaris days and through today) that what's using it is the only thing of substance on the box.
- vlovich123 3y ago
  I think that’s a flawed assumption. Databases show up in many spots besides dedicated RDBMS (eg SQLite and RocksDB power all sorts of things and likely coexist with compute nodes because even compute nodes can need some amount of local storage).
inkyoto 3y ago
I did not imply that O_DIRECT was a full equivalent of raw devices, it was rather a compromise as Linus would not concede with the idea of raw devices making it into Linux. Besides, O_DIRECT can be used at the file level (and the file system level as well via chattr – if the file system supports it) whereas the raw devices are, well, device level only.
- vlovich123 3y ago
  I’m pretty sure you can open disks raw in Linux. But then the disk is owned by the RDBMS. Ceph can do this in its BlueStore engine. But yeah, I understand you weren’t talking about buffers, I was just highlighting that O_DIRECT and raw devices are totally separate problems from having a user space cache that can be evicted by the kernel memory and reconstructed by the application lazily.
  - hyc_symas 3y ago
    Yes, you can use raw block devices in Linux. LMDB works on raw block devices as well. You get write-through cache behavior; writes are immediate but everything still goes into the page cache. I've not yet bothered to check how things work with raw devices in Windows. I believe only Admins can even operate on them so it would be kind of limited usefulness.
    - vlovich123 3y ago
      I don’t believe the page cache is accessed if you open the raw block device via o_direct.

ericbarrett 3y ago

This is commonly avoided by not putting memory pressure on your O_DIRECT application, which is probably an RDBMS on a dedicated single-purpose host. Not saying more work in the space wouldn't be welcome and interesting, just that it's generally the assumption whenever I've seen O_DIRECT (since the Solaris days and through today) that what's using it is the only thing of substance on the box.

vlovich123 3y ago
I think that’s a flawed assumption. Databases show up in many spots besides dedicated RDBMS (eg SQLite and RocksDB power all sorts of things and likely coexist with compute nodes because even compute nodes can need some amount of local storage).

inkyoto 3y ago

I did not imply that O_DIRECT was a full equivalent of raw devices, it was rather a compromise as Linus would not concede with the idea of raw devices making it into Linux. Besides, O_DIRECT can be used at the file level (and the file system level as well via chattr – if the file system supports it) whereas the raw devices are, well, device level only.

vlovich123 3y ago
I’m pretty sure you can open disks raw in Linux. But then the disk is owned by the RDBMS. Ceph can do this in its BlueStore engine. But yeah, I understand you weren’t talking about buffers, I was just highlighting that O_DIRECT and raw devices are totally separate problems from having a user space cache that can be evicted by the kernel memory and reconstructed by the application lazily.
- hyc_symas 3y ago
  Yes, you can use raw block devices in Linux. LMDB works on raw block devices as well. You get write-through cache behavior; writes are immediate but everything still goes into the page cache. I've not yet bothered to check how things work with raw devices in Windows. I believe only Admins can even operate on them so it would be kind of limited usefulness.
  - vlovich123 3y ago
    I don’t believe the page cache is accessed if you open the raw block device via o_direct.