5 ms·
That's progress. I've previously suggested that operating systems should have stronger file integrity guarantees. "Unit" files (rewriting replaces the whole fi
by Animats 6y ago
That's progress.
I've previously suggested that operating systems should have stronger file integrity guarantees. "Unit" files (rewriting replaces the whole file atomically, no reader ever sees a partially written file). That's the default. "Log" files (always end at a clean end point, don't tail off into junk). "Temp" files (disappear on reboot). And, for databases, "Managed" files.
Managed files have more I/O functions. In particular, you get two completion events on writes - "copy complete" (the caller can reuse the buffer) and "safely stored" (the data has reached its final resting place, all links are complete, etc.). Programs like databases would use that. Those are the semantics databases want, and struggle to get by flushing, waiting, and various workarounds.
When I mention this, what usually happens is that people get lost in complicated workarounds for simulating unit files. Different approaches are needed for Linux, Windows, NTFS, and various VM systems. This should Just Work.
This isn't my invention; it's from Popek's kernel in 1985 at UCLA, later seen as UCLA-Locus and as an IBM product. They had explicit commit and revert functions for file systems. I'd suggest having the default be commit on normal close or normal program exit, but if the program aborts or crashes or is killed, unit files don't commit and remain unchanged.
- rodgerd 6y agoMeanwhile ext4 is plowing even further down the path of "we get great benchmarks if we make data integrity a userspace problem LOL"
- Dylan16807 6y agoHow so? They had that recent batch of safe optimizations, and trying to make fsync act closer to what it actually says is a pretty good thing.
- wmf 6y agoThe fact that write(), close(), rename() isn't safe unless you fsync is pretty annoying.
- Dylan16807 6y agoIt is, but also isn't new.
- the8472 6y agoa) posix doesn't guarantee or require it b) ext4 actually provides a workaround for that kind of broken software with its auto_da_alloc behavior, which is on by default.
- Dylan16807 6y agoWhat posix actually requires is an awful mess and just as broken if not moreso. Oh, you just said to sync the file, not the directory? Too bad, data's gone.
- a1369209993 6y agoPosix doesn't guarantee or require that you can reliably close a file descriptor. Modus tollens and all, or as the saying goes, 98% confidence has been refuted at general relativity.
- jcranmer 6y agoPOSIX filesystem semantics are one of the things where POSIX just plain got stuff wrong. And it isn't helped by the fact that a lot of people want to game benchmarks by slightly lying about their durability compliance, so you really have to jump through hoops to make sure you actually achieve durability. I do agree with you that we need a better way to interact with the filesystem with regards to integrity and durability, although I'm not sure we entirely agree on how that would look. The idea of multiple modes makes a lot of sense: * File-atomic mode. This is I believe trivial to implement in the filesystem layer for all filesystems, and the basic idea of this mode has existed for decades. When a reader opens a file for reading, it will never see any other writes to the file. A writer will only update the file when it closes the file [1], at which point any new reader will see only the new file created. The code is intrinsically safe in the face of multiple processes interacting with the file, and is probably the semantics most people would prefer in that situation. * Append-only transactional files. Here, you can't random-access write into the file (but you can random-access read), only write at the end or truncate the file. A writer designates the text to append to the file as atomic blocks: the reader will only atomically see or not see the block [2]. If the file is truncated, all readers see the original contents of the file until they close. * Raw files. Don't pretend that a file is a stream of bytes. Instead, expose it as a set of blocks that can be atomically updated (including atomically adding or removing blocks from the file at different places). I don't know filesystem semantics to give any good details here, but my understanding is that databases basically try to get these semantics today, and that getting good guarantees on fully random-access read/write semantics is effectively impossible anyways. There does feel to me to be a bit of a hole here, where you basically get no multiprocess interactions via files unless you completely change how your code works, but I'm not sure it's entirely feasible to have a middle ground here. You can probably get close enough for most needs with a way to be notified and reopen the file in file-atomic mode, and anything where that's not sufficient probably needs you to go to raw files to really get the guarantees you want. In addition to the basic file I/O issues, there also needs to be a way to be more transactional with directories, I think. Using paths as the basis for filesystem issues is already opening up programmers to time-of-check-time-of-use attacks today, and moving to a file descriptor-based approach for directory manipulation would solve that while opening up the possibility for better transactional support on the directory level. The other issue is durability. Most applications in the first two modes would probably be fine with an optional durability: the result of an unexpected power outage would be a file that is out of date, but not corrupt. The filesystem could provide an optional callback on commit that returns when it is durability committed, which would handle those cases where you do actually need to make sure that the data will be committed on unexpected power outage. And the simple semantics of the first two modes means that providing durability reliably should be easy for filesystems. [1] This also suggests that there should be a way to abort the write. [2] You can also see how a file-atomic reader can interact with an append-only writer: the filesystem layer needs to remember the size the reader first saw and pretend that's the EOF, but otherwise there's no issue. And append-only readers will act as a file-atomic reader with respect to a file-atomic writer. The interactions make sense, that's a good sign for the model!
- josephg 6y agoI agree - although I see all of those cases as special cases of the same general action: filesystem operations are state machine transitions. I’ve been playing with the idea lately that the OS could support general transformation functions (eg “insert an ‘A’ character at this location”, “append this log entry”, “overwrite this byte range”, etc. Those transformation functions could be generic or written in wasm/BPF. The filesystem can process operations like this efficiently by storing them to a log and periodically flushing. (Or whatever makes sense for the operations). Having a completion API that separates “buffer can be reused” and “persistently flushed” is great, and should be available for all filesystem operations. Not just for databases! If an application wants to use traditional posix semantics, write() calls can just be one of the supported operation types. And as a bonus, filesystem watching can become highly granular - you could subscribe to the stream of semantic changes to a file!
- KMag 6y agoAs we continue to increase throughput by increasing the number of cores, high-throughput local filesystems are going to start looking more like distributed filesystems. (A multicore system really is a distributed system with a low-latency interconnect and some hardware to give the illusion of flat access to main memory.) Having done a fair amount of work on long-term persistence of data from a Kafka queue to a distributed filesystem, I wish filesystems provided Lamport timestamp for the current time and a Lamport timestamp of the earliest uncommitted write. The distributed filesystem itself would internally keep track of the earliest uncommitted write using a vector clock, though only the minimum component of that vector would be externally visible. When I perform a write, I want to send an opaque 64/128-bit ID of my choosing, the write location, and the data. I want to asynchronously get back a message with either an error message and the write ID, or else [the write ID, the current Lamport timestamp, and the minimum component of the uncommitted timestamp vector] (as known by the node where I just wrote). By keeping a mapping of these Lamport timestamps to my Kafka partion read offsets, this would reduce the number of network round-trips necessary to commit my Kafka partition read offsets back to Kafka. As it stands, I periodically need some extra network round-trips to query commit status of my async writes before committing read offset changes for my consumer group back to Kafka. With the distributed filesestem keeping a vector cloock, and each async write getting back a message containing the current Lamport timestamp and the minimum uncommitted timestamp, then I could put that current time in my map of timestamps to Kafka read offsets, and then commit back to Kafka all read offsets corresponding to fs writes that are older than the earliest uncommitted write. Presumably, the Lamport timestamp and vector clock maintenance would be piggy-backed on the internal messages necessary for the distributed filesystem to persist the writes to disk on multiple nodes. That is, I write to node A, which tells me "that's a write at time 9", and node A sends messages saying "here's a write at time 9, and my earliest uncommitted write is at time 4" to nodes B and G. The next time that node G needs to forward a write to node A, then node A will update its current time and its G component of the uncommitted timestamp vector clock. You'd also want each node to guarantee a maximum amount of time between its forwarding writes to any given node, forwarding an empty write for clock update purposes if it has been too long.
- skissane 6y ago> This isn't my invention; it's from Popek's kernel in 1985 at UCLA, later seen as UCLA-Locus and as an IBM product Which IBM product? I am guessing AIX PS/2 and AIX/370, since those are the two operating systems Locus Computing developed for IBM.
- KMag 6y agoFor the temp file issue, there's the common pattern of creating a file and unlinking it right away while keeping an open file handle. It's unfortunate that there's not a version of creat() that takes a path to a directory (for determining on which fs to create) but doesn't link the file, plus a version of link() that takes a source file descriptor instead of a source file. If extended to support non-file fds, the link function might also do some interesting things, like have TCP connections show up in the virtual file system, similar to Unix pipes.