4 ms·
Just wondering, but how is an OS supposed to achieve this without fsync()ing all writes to memory and being slow as dog$h!t? Separate from the persistent stora
by Animats 5y ago
Just wondering, but how is an OS supposed to achieve this without fsync()ing all writes to memory and being slow as dog$h!t?
Separate from the persistent storage thing, I've wanted traditional file systems that work like this:
* Unit files. The unit of file consistency is the file. The default case. Open, create, close, no other open can read it before close, on close it replaces the old file for all later opens. On crashes, you always get a consistent file, the old version if the close hadn't committed yet.
* Log files. The unit of file consistency is one write. Append only. On crashes, you get a consistent file up to the end of a previous write. No trailing off into junk.
* Temp files. There is no file consistency guaranteed. Random access. On crashes, deleted.
* Managed files. A more complex API is available. Random access. On a write, you get two completion events. The first one is "buffer has been copied". The second is "data is safely in permanent storage".
This is for databases, which really care about when the data is safely stored. The second full completion event might also be offered for other file types, primarily log files.
That covers the standard use cases and isn't too hard to do.
- zozbot234 5y ago> The unit of file consistency is the file. The default case. Open, create, close, no other open can read it before close "$PROGRAM cannot open document $FILENAME because it's already open in $PROGRAM. Try closing any open applications and open $FILENAME again." Who thinks this is actually sane behavior? Unix does it right.
- kragen 5y agoInstead of giving you a mutual exclusion error, a newly created file written this way might simply be invisible until it's complete, at which point it atomically replaces the old version of the file (which might still be accessible through mechanisms like VMS version numbers, WAFL snapshots, Linux reflinks, or simply having opened it before the replacement). This is probably what a transactional filesystem would give you: the creation, filling, and naming of the file would normally be part of the same transaction. But, with optimistic synchronization, transactions that were reading an old version of the file might require redoing from the start when a new version was written.
- skissane 5y ago> "$PROGRAM cannot open document $FILENAME because it's already open in $PROGRAM. Try closing any open applications and open $FILENAME again." Who thinks this is actually sane behavior? Unix does it right. Wouldn't a better approach be: "$FILENAME is already open in $PROGRAM. Would you like to: (1) Switch to $PROGRAM and continue editing it there; (2) Ask $PROGRAM to close it; (3) Edit it anyway (DANGEROUS!); (4) Abort". (For non-interactive processes, you wouldn't want to ask, you'd probably just want to always abort–which causes problems for some situations like "I want to install an upgrade to this shared library but $PROGRAM is executing using it", but there are ways to solve those problems.) Kind of like swap files in vim – although vim doesn't have the "switch to $PROGRAM" option, because the underlying operating systems don't have good support for it – but that's an application-level feature which only works in one app. It would be better if it was an OS-level feature which worked across all apps.
- kragen 5y agoI agree that this would be useful. Have you ever written a prototype of it?
- jandrewrogers 5y agoAs additional support for this thesis, the interface to the logical (sometimes physical) file system inside many database kernels is almost exactly this. All four file types have a concrete use case that requires their existence. One of the things that makes them easier to use in a database kernel context is that even though they are all built on top of the same low-level storage primitives, each file type usually has its own purpose-built API instead of a generic overloaded file API, which would get messy very quickly. A file's type never changes and the set of types is enumerable, so this is clean. The only thing keeping this from being a user space thing is that there is no practical way of making this POSIX compliant. POSIX has all but permanently frozen what is feasible in a file system. Many database kernels either work off raw block devices or emulated raw blocks device on top of a POSIX file system, on which they build a file system, but it is never exposed to the world which eliminates any requirement for compliance with external standards. Imagine how much easier it would be to build things like databases if this was a first-class API.
- rurban 5y agonot being POSIX compliant is actually needed for a modern OS. with POSIX you'll never get to safe concurrency, so you'll hardly can make use of all the CPU kernels. it must be a microkernel, supporting only safe languages. certainly not rust. Genode is close, pony was close, Fuchsia is getting there, singularity was there already. concurrent pascal also.
- kragen 5y agoHmm, are you saying POSIX compliance makes scalability impossible? Because it makes it possible to run software written in unsafe languages? I'm pretty sure Fuchsia will always be able to run software written in C.
- rurban 5y agonot impossible but hard. but more importantly not safe. Fuchsia supports C++, sure. Fuchsia is not a safe OS, just like Rust a way to better safety. Dart for flutter is their safe language there.