3 ms·
POSIX filesystem semantics are one of the things where POSIX just plain got stuff wrong. And it isn't helped by the fact that a lot of people want to game bench
by jcranmer 6y ago
POSIX filesystem semantics are one of the things where POSIX just plain got stuff wrong. And it isn't helped by the fact that a lot of people want to game benchmarks by slightly lying about their durability compliance, so you really have to jump through hoops to make sure you actually achieve durability.
I do agree with you that we need a better way to interact with the filesystem with regards to integrity and durability, although I'm not sure we entirely agree on how that would look. The idea of multiple modes makes a lot of sense:
* File-atomic mode. This is I believe trivial to implement in the filesystem layer for all filesystems, and the basic idea of this mode has existed for decades. When a reader opens a file for reading, it will never see any other writes to the file. A writer will only update the file when it closes the file [1], at which point any new reader will see only the new file created. The code is intrinsically safe in the face of multiple processes interacting with the file, and is probably the semantics most people would prefer in that situation.
* Append-only transactional files. Here, you can't random-access write into the file (but you can random-access read), only write at the end or truncate the file. A writer designates the text to append to the file as atomic blocks: the reader will only atomically see or not see the block [2]. If the file is truncated, all readers see the original contents of the file until they close.
* Raw files. Don't pretend that a file is a stream of bytes. Instead, expose it as a set of blocks that can be atomically updated (including atomically adding or removing blocks from the file at different places). I don't know filesystem semantics to give any good details here, but my understanding is that databases basically try to get these semantics today, and that getting good guarantees on fully random-access read/write semantics is effectively impossible anyways.
There does feel to me to be a bit of a hole here, where you basically get no multiprocess interactions via files unless you completely change how your code works, but I'm not sure it's entirely feasible to have a middle ground here. You can probably get close enough for most needs with a way to be notified and reopen the file in file-atomic mode, and anything where that's not sufficient probably needs you to go to raw files to really get the guarantees you want.
In addition to the basic file I/O issues, there also needs to be a way to be more transactional with directories, I think. Using paths as the basis for filesystem issues is already opening up programmers to time-of-check-time-of-use attacks today, and moving to a file descriptor-based approach for directory manipulation would solve that while opening up the possibility for better transactional support on the directory level.
The other issue is durability. Most applications in the first two modes would probably be fine with an optional durability: the result of an unexpected power outage would be a file that is out of date, but not corrupt. The filesystem could provide an optional callback on commit that returns when it is durability committed, which would handle those cases where you do actually need to make sure that the data will be committed on unexpected power outage. And the simple semantics of the first two modes means that providing durability reliably should be easy for filesystems.
[1] This also suggests that there should be a way to abort the write.
[2] You can also see how a file-atomic reader can interact with an append-only writer: the filesystem layer needs to remember the size the reader first saw and pretend that's the EOF, but otherwise there's no issue. And append-only readers will act as a file-atomic reader with respect to a file-atomic writer. The interactions make sense, that's a good sign for the model!
- bcrl 6y agoI wouldn't exactly call POSIX file i/o semantics wrong: POSIX came after the systems that it was trying to document, so it wasn't in a position to propose new semantics. That has always been the problem with advanced filesystem APIs. Without new advanced APIs being available on all major platforms, developers fall back to the lowest common denominator across the platforms they are trying to support. And it's difficult to try to change these semantics once systems are in place. It is also important to note that complex semantics make building high performance systems much more difficult. Nobody in their right mind would use POSIX locks in a high performance application. More complex APIs often have implicit locking requirements which defeat the efforts made to improve scalability on multi-core systems over the past decades of system development. Systems development is all about trade-offs, and complexity constraints choices in often unexpected ways.