4 ms·
> There is a trade off obviously where the kernel can evict I/O buffers more efficiently. But to me all that says is that the interface for managing I/O buffers
by inkyoto 3y ago
> There is a trade off obviously where the kernel can evict I/O buffers more efficiently. But to me all that says is that the interface for managing I/O buffers in user space is really immature and needs some better performing API so that we have something in between mmap and buffer management.
This was essentially the argument against adding the raw device support to Linux, which delayed the Oracle database Linux port for years, with Linus himself being adamant that operating system's own VMM and page cache management was superior to that of the database (except it was not), which was proven to be a fallacy as the transaction and error handling database requires completely different guarantees, and the VMM pursuing different design objectives that most of the time are incompatible with the database expectations.
The raw devices have been used precisely for that reason: «let's bypass the VMM and the page cache for this device, and we will take it from there». That is what the Oracle database did on Solaris – it wanted an entire raw device that it would claim for the table space. Granted, Oracle has had its own VMM and page cache (so to speak) implementation within the DB, an OS mini-kernel if you like.
The argument was eventually settled with adding the O_DIRECT flag to open(2) accomplishing the same outcome, and the raw devices have never made into Linux.
- vlovich123 3y agoO_DiRECT is insufficient because you have no way to coordinate with the kernel to drop io buffers when there’s memory pressure
- ericbarrett 3y agoThis is commonly avoided by not putting memory pressure on your O_DIRECT application, which is probably an RDBMS on a dedicated single-purpose host. Not saying more work in the space wouldn't be welcome and interesting, just that it's generally the assumption whenever I've seen O_DIRECT (since the Solaris days and through today) that what's using it is the only thing of substance on the box.
- vlovich123 3y agoI think that’s a flawed assumption. Databases show up in many spots besides dedicated RDBMS (eg SQLite and RocksDB power all sorts of things and likely coexist with compute nodes because even compute nodes can need some amount of local storage).
- inkyoto 3y agoI did not imply that O_DIRECT was a full equivalent of raw devices, it was rather a compromise as Linus would not concede with the idea of raw devices making it into Linux. Besides, O_DIRECT can be used at the file level (and the file system level as well via chattr – if the file system supports it) whereas the raw devices are, well, device level only.
- vlovich123 3y agoI’m pretty sure you can open disks raw in Linux. But then the disk is owned by the RDBMS. Ceph can do this in its BlueStore engine. But yeah, I understand you weren’t talking about buffers, I was just highlighting that O_DIRECT and raw devices are totally separate problems from having a user space cache that can be evicted by the kernel memory and reconstructed by the application lazily.
- hyc_symas 3y agoYes, you can use raw block devices in Linux. LMDB works on raw block devices as well. You get write-through cache behavior; writes are immediate but everything still goes into the page cache. I've not yet bothered to check how things work with raw devices in Windows. I believe only Admins can even operate on them so it would be kind of limited usefulness.
- vlovich123 3y agoI don’t believe the page cache is accessed if you open the raw block device via o_direct.