3 ms·
I've proposed this informally before, along these lines: - Unit files. You create, you write, you close, and then others can read the new version. Until the or
by Animats 3y ago
I've proposed this informally before, along these lines:
- Unit files. You create, you write, you close, and then others can read the new version. Until the original writer closes and gets a good close status, nobody else can read it. If the program aborts or the system crashes before closing cleanly, the file reverts. All readers see a fully written file. This is the default. It's what most programs need. Replacing a file by creating a new one on top of it is both permitted and an atomic operation. UCLA-LOCUS and some IBM systems that followed worked that way.
- Temporary files. Disappear on system crash.
- Log files. Append-only. All readers are guaranteed to see an end of file position that corresponds to the end of a previous write. Usually the most recent write, but buffering may make log file reading run a little behind.
- Managed files. Read, write, share. Write operations return two completions, probably via some async mechanism. The first completion means "buffer contents taken". The second completion means "committed to storage that will survive a crash". Database systems would use this, but few other programs would bother.
This tells the database what it really needs to know - when is the data safe? That tells the database when it can commit a transaction. The database can do other things, including more I/O, while waiting for commitment.
"fsync" is really a clunky way of getting that second completion.
- the8472 3y agoI think most of those things are implementable in userspace with the right combination of O_TMPFILE, rename, fsync, fsync(dir), sync_file_range and io_uring. So all that's needed is encapsulating it in a library that provides those different abstractions. Well ok, we could also use an additional flag for linkat() that allows atomically replacing the target to close that tiny window where a temporary file might get left behind after a crash.
- vlovich123 3y agoActually, a good chunk of that is available via memfd + linkat. - Unit files: memfd, fdatasync, linkat, fsync(dir) - Temporary files. memfd, preferably within /tmp to avoid any disk I/O unless you have a particularly large file to buffer - Log files. This is trickier. Probably requires running on a fs that supports reflinks and creating a new memfd that’s a true clone of the original file, appending, & then replacing it. The one piece that is incompatible with this is that other file descriptors are stale and you have to reopen to see new data but that’s at odds with APIs available today - either readers can see partial writes or you have to have a new file. There have been proposals to allow buffering writes but delaying publishing the metadata acking the data but that hasn’t gone anywhere unfortunately. - Managed files. I’m not sure that completions are the right contract here because technically the kernel could defer that completion indefinitely whereas databases typical need some kind of latency bounds of “OK - I really want this to commit to disk nowish”.
- nh2 3y ago"memfd + linkat"? How shall that work? memfd is for files in RAM. linkat cannot get them onto disk. What you seem to have in mind is what the grandparent post already described: "O_TMPFILE + linkat".
- vlovich123 3y agoSorry that’s right. O_TMPFILE is indeed what I meant
- throwaway09223 3y agoIt's not, though, because fsync has no relation whatsoever to what other processes see. Durability is totally different from concurrency or multiprocess consistency. There are already guarantees about how we can order reads and writes in a shared file. Very well defined guarantees -- ones we've built entire RDBMS systems on top of. Guaranteeing durability (persistence across a crash) is a totally separate problem from understanding write visibility between threads.
- Animats 3y ago> It's not, though, because fsync has no relation whatsoever to what other processes see. Ah, the SQLite people. I was thinking in terms of databases such as MySQL and Postgres where one process owns the files and client programs communicate with it.
- throwaway09223 3y agoIt's the same for sqlite, postgres, mysql, or any other file. Syncing data to durable storage is totally unrelated to visibility guarantees between multiple threads or processes working within the same file (be it using write, mmap, whatever)
- Animats 3y agoSQLite relies on file system visibility guarantees, because there are multiple processes connected only via the file system. But Postgres and MySQL have one process that touches the underlying file, so they can coordinate internally.
- throwaway09223 3y agoNot really. It's often all just shared memory via mmap. The files are never closed, and sometimes never operated on with read/write because they're just mapped. This behavior is common between (eg) both sqlite and mysql. (Postgres defaults otherwise, but architecturally it is not different) The visibility issues you're raising, basically implementing views and transactions, would not be applicable to any of these systems. You're confusing durability guarantees (fsync, msync) with visibility guarantees -- the latter of which are just standard memory access issues.
- pixel8account 3y ago>This is the default. It's what most programs need. Is it though? I think single writer+many readers is a very common use case. For example I want a compiler to read a file I have open in my IDE. Or I want to be able to open a log file while the application is still running.
- Animats 3y agoIt's fine to read an append-only file. But if the file is a unit file, you can't read the version being written until it is done. So you have file integrity.