8 ms·
File system semantics need a rethink. Here's what I've proposed. This is not original with me; these are all things implemented in some non-UNIX-based systems.
by Animats 4y ago
File system semantics need a rethink. Here's what I've proposed. This is not original with me; these are all things implemented in some non-UNIX-based systems.
Files should come in several kinds:
- Unit files. They're immutable, but replaceable. This is the normal case.
You create them with "creat",
write them, and when closed, they replace any previous version with the same name. Any program which opens one for reading sees a complete file. On a crash before close, the new file is dropped and the old version remains. If a program aborts, that also happens. An explicit close commits the file update.
- Log files. These are append only. On a crash, you are guaranteed that the file will be intact to some point ending at a write boundary. The last transactions may be lost, but the file must not tail off into junk. For journals, log files, etc.
- Temp files. These disappear on a crash.
- Managed files. These are for databases. They can do everything files can do.
Along with this, you need async I/O. This is primarily for log files and managed files. When you do a write, you get two callbacks, one when the buffer has been copied, and one when the data has been committed to permanent storage and will survive a crash. It is the job of the OS, drivers, and devices to not send the data-committed completion until the data is safe.
Only a few programs need that, but they're the important ones - database managers.
This turns the fsync problem around. Instead of doing an fsync to force write-out, databases wait for a completion before considering the data committed. It's not a flush-everything; it's a commit-complete for a single write. So if you have multiple I/O operations pending on a single file, a normal event in database land, there's no big stall for a cache flush.
There's also no incentive for drive manufacturers to send commit-completes early. The only people who use them are the ones who will notice if someone cheats. Most bulk I/O is on unit files, where something is being copied.
(There are mainframe systems that had this sort of thing half a century ago.)
- sokoloff 4y agoWouldn’t early commits be a way to game synthetic benchmarks? (And thus sell more hardware if your product benchmarks well.)
- tarruda 4y ago> File system semantics need a rethink Normally when writing software that needs fopen/fsync, I would just use SQLite instead
- josephg 4y agoThere's a host of reasons this isn't good enough: - SQLite itself would be simpler and probably faster if it could run on top of a better OS-level API - It doesn't make sense to build every other program on top of SQLite. Eg, other databases (redis, postgres), anything performance sensitive, etc. The filesystem is the system interface. Not sqlite. Aesthetically, its also pretty gross to make a mess in POSIX, then solve it in userland (via sqlite). Its much better to just have the OS expose an API that is actually nice to work with and avoid all the slowdown, complexity and bugs which can arise from the nasty fsync API. As an example, redis runs a whole extra userland thread dedicated to issuing fsync commands to the operating system. There's a whole signalling system needed to manage that thread. With an API like the GP proposed, none of that code would be needed.
- 1over137 4y ago>It doesn't make sense to build every other program on top of SQLite I dunno, it's not a crazy as building everything on top of Electron. :)
- josephg 4y agoTrue - but that's not exactly a high bar for success. I don't want every program I run to be an electron style hodgepodge mess which takes 1800 dependencies to copy an iso to a USB stick (or whatever).
- mgaunard 4y agoMany databases (including SQLite) don't use the filesystem correctly. The fault doesn't lie with the filesystem but with the database.
- josephg 4y agoCan you give some examples of ways databases misuse the filesystem?
- 4y ago
- mikepurvis 4y agoHow much are current database implementations hamstrung by the standard file semantics? If it's a lot, I'm surprised there aren't more databases that just mount a bare disk or partition and do whatever they want with it. Or ship a special kernel module they can talk to that implements a better interface to the underlying block device? Or does none of this really matter in a world of virtualized-everything where your "block device" is still ultimately hitting multiple layers of actually-just-being-a-file on its way to hardware?
- hyc_symas 4y agoLMDB performance suffers when used on a journaling filesystem. Consequently, LMDB supports using a plain block device, to eliminate all FS overheads. And yes, all of the perf benefits go out the window if you're running on several layers of virtualization anyway.
- meangrape 4y agoThirty years ago it was standard practice to mount raw disks for Oracle to use.
- icedchai 4y ago~15 years ago, the last time I used Oracle in production, that was still standard practice. I'd be interested to hear from someone who's used it recently.
- needusername 4y agoThat or ASM devices. To my knowledge io_submit and io_getevents only really work with raw partitions anyway.
- FridgeSeal 4y agoI was under the impression that MSSQL basically runs almost an entire FS/OS kernel inside itself to manage its own IO? Scylla DB also goes to great lengths to optimise IO paths and talks directly to storage devices via DirectIO IIUC.
- jmspring 4y agoFiles work fine and there are abstraction layers on top of the file system that handle more of it.
- josephg 4y agoThey don't work fine for everyone. I'm building a custom storage engine for a CRDT at the moment and I don't want my data corrupted under any circumstances. Writing code to do this reliably is painful. Even figuring out the rules is difficult - is fsync enough? fdatasync? Will it still corrupt on power loss events in modern consumer SSDs? How do I test / simulate my software under these failure conditions? If I get an error when I fsync, what assumptions can I make about the state of my data? Some atomicity rules seem to be page-size specific, but I can't find any way to query the OS for the kernel's page size, or the page size of the underlying device. The status quo is horrid.
- mangamadaiyan 4y agoWhat operating system are you on?
- josephg 4y agoI'm developing on linux, but my code should work on any modern platform. (Windows, MacOS, iOS, Android, Linux). On Apple's hardware, fsync (the "no really" version) actually does durably flush data to disk when you ask it to. And darwin has a write barrier command, which is great because I don't need to fsync at all in order to append to my transaction log. But linux is missing a barrier command, and I've heard most consumer grade SSDs lie when you ask them to fsync. Generally, linux doesn't really seems designed to preserve your data when the system crashes or experiences a power loss. The single API it gives you to help (fsync) is badly designed, coarse grained, slow, badly documented and its hard to use correctly. And worst of all, it often doesn't work anyway thanks to crap SSDs. There's lots of better APIs out there. IO completion ports on windows are a better design. Likewise, adding write barriers (like macos has done) would help a lot. The filesystem APIs on linux are long in need of being overhauled. Its sorely lacking if you want your software to be reliable.
- josephg 4y agoYep this would be fantastic. The ironic thing about it is that a fine-grained write completion callback like you mention would actually improve performance compared to the coarse fsync we have today. Another simpler way to improve the filesystem API would be to add a write barrier system call. If I call write(file, a); barrier(file); write(file, b); then the OS guarantees that the second write can't start before the first write is fully committed to disk. This is enough to implement most journaling systems, databases and log files. This would perform much better than the current POSIX API because you don't need to "stop the world" with an expensive fsync before issuing subsequent write commands. The kernel can buffer chunks of writes separated by barriers, and issue them to the underlying device as needed. The completion API is nicer though, because you can (for example), not respond to a network request to update the database until the changes are flushed to disk.
- hyc_symas 4y agoWrite barriers have been discussed for years and years. Personally I prefer grouped writes. https://www.spinics.net/lists/linux-fsdevel/msg70047.html https://www.spinics.net/lists/linux-fsdevel/msg70047.html The underlying SCSI and SATA protocols would support this with command queueing. I would have envisioned using an fcntl() to set the current group ID on an fd.
- josephg 4y agoGrouped writes seem more or less equivalent to write barriers in terms of their semantics, but probably with 1 less syscall - which is nice. But I'd take any of these approaches over years of discussion & no solutions. I wonder what it would take to actually implement this stuff in linux. A lot of work, sure - but far from impossible. Thanks for lmdb btw! The design is lovely. I've used it in a couple projects, and I had a read through your code for inspiration while I was designing out a little storage engine for something I've been working on. Its delightful.
- hyc_symas 4y ago> Thanks for lmdb btw! Thanks, that's great feedback! Happy to hear that you're both using it successfully and that you actually read the code and learned from it.
- sayrer 4y agoHave you read "Designing Data-Intensive Applications" by Martin Kleppmann? I'm sure most of it will be old news to you, but it does add multiple computers to what you've described here. You will have to read like 80% of the book to get to that part though.
- mananaysiempre 4y ago> Along with this, you need async I/O. This is primarily for log files and managed files. When you do a write, you get two callbacks, one when the buffer has been copied, and one when the data has been committed to permanent storage and will survive a crash. This is what you want if you need to communicate the durability guarantee to the outside world, but if you’re just trying to enforce some consistency guarantees for internal purposes (e.g. transaction journal write goes out before data write), callbacks impose an unnecessary CPU-disk-CPU roundtrip. That’s what Featherstitch[1] was trying to avoid. Or is that not worth it? More generally, how does this compare to other throughput-oriented queue-adjacent things, such as GPU commands buffers and fences, or Linux io_uring and Windows RIO? (A comparison to NIC queues is probably less interesting because NICs don’t reorder things—or do they? What about radios?) Do all of these need to be separate mechanisms? Sudden thought: Featherstitch is to completion callbacks as promise pipelining[2] is to traditional RPC. [1] https://lwn.net/Articles/354861/ https://lwn.net/Articles/354861/ [2] https://capnproto.org/rpc.html https://capnproto.org/rpc.html
- ggm 4y agoVax/VMS filesystem's different flavours of file wasn't that dissimilar to this. Additionally it had "mailboxes" which were basically named pipes.
- ritcgab 4y agoThe semantics of a bare file system cannot be (easily) changed, I think. What would be nice is having a thin shim layer on top of their existing semantics to handle more advanced file abstraction. Applications nowadays need to repeatedly do something that file system doesn't provide, but sometimes their solution partially overlaps with the fs. A middleware that provides clearer separation would be nice.
- needusername 4y ago> - Temp files. O_TMPFILE should do this. > - Unit files. O_TMPFILE together with renameat2 and RENAME_EXCHANGE should do this.
- CJefferson 4y agoLast time I looked, about three years ago, one still couldn’t assume O_TMPFILE would be supported, too many file systems and old kernels didn’t have it. It is a nice flag, so hopefully it’s more likely to be supported now.
- FridgeSeal 4y agoThis is a cool idea, I like this. Forgive my lack of knowledge, but wouldn’t something like io_uring satisfy the need of an async API, and also obviates the need for the first callback? You write into the ring (which the kernel does not have to do a secondary copy from) and it writes that onto your disk and tells you when it’s done? In fact, IIUC, direct IO goes a step further and let’s you write directly to the storage-and in return you give up some niceties like OS-managed caching and prefetching because you have to do them yourself. In which case, a step towards this better FS API you’ve proposed would be some higher-level API’s around io_uring/directIO to make it better to use?