4 ms·
Renaming is commonly proposed for rewriting files but does not provide the guarantees people usually think it does. Posix renames are only atomic during normal
by dan-robertson 6y ago
Renaming is commonly proposed for rewriting files but does not provide the guarantees people usually think it does. Posix renames are only atomic during normal operation. But here atomic merely means that other syscalls will not return results for some partially-renamed state. The operation does not need to be atomic in the case of a crash (where here “atomic” means “you can’t read the file system in some partially renamed state after a crash”, a different definition but one which people commonly assume when they hear that rename is atomic). I’m not aware of any normal Linux file system where renames are always atomic over a crash
- dmm 6y agoZFS is really well behaved. It never reorders metadata in an observable way so renames should truely be atomic. In the event of a crash you'll get the old or the new name.
- fiddlerwoaroof 6y agoWhat I’d prefer to atomicity is automatic file versioning such that a file descriptor opened at time t0 always points to the same contents, unless you refresh it. You’d also need CoW and/or writing patches rather than rewriting the whole file and a GC/compactor for cleaning up transient versions of a file. The workflows this would be enable is that, when vim wants syntax checking, it writes a transient version of the file to the fs without updating the “HEAD” pointer and then hands a fd to the syntax checker that points to that new transient version.
- a1369209993 6y agoYeah, the way you're 'supposed' (as near as I can tell) to do this is: read 'foo' write 'foo~' hardlink 'foo~' onto 'foo' unlink 'foo~' on crash: if 'foo' exists: it's the correct state for the file unlink 'foo~' continue otherwise: hardlink 'foo~' onto 'foo' unlink 'foo~' I'm not sure Posix actually requires link and unlink to be atomic enough on crash for this work, but it seems to be fine for any sane filesystem (where the fs itself doesn't fall apart in presence of crashes).
- tomtomtom777 6y agoThis doesn't work because the link() syscall fails if the target exists. You use a hardlink only for atomic creation of the initial data: write 'foo~' hardlink 'foo~' onto 'foo' To atomically update you simply use rename: read 'foo' write 'foo~' rename 'foo~' into 'foo' I don't know what dan-robertson's claim is based on, but this surely is atomic even on a crash, as the filesystem will never be in a state where 'foo' contains only partially updated data. That would require some severe fs corruption.
- ynik 6y agoIt doesn't require severe FS corruption. In reality "write 'foo~'" just starts an asynchronous write; it's possible for rename to happen before that async write is complete. So in case of a system crash you might end up seeing partial data in 'foo'. fsync() may or may not protect against that depending on what system we're talking about.
- tomtomtom777 6y agoSure, you are still dependent on constraints to durability and atomicity of the filesystem write of the new data. But this is handled properly in most modern filesystems (ext3,ext4,zfs) by default (as data=ordered ensures data writes precede metadata writes). The point of rename() is to prevent a version to ever exist that mixes the old and the new version, which it always does, regardless of the filesystem.
- deleted 6y ago[deleted]
- tomtomtom777 6y agoHow can a rename not be atomic over a crash? What does a partially renamed state even mean? Surely, if I do "write foo~", "rename foo~ foo", regardless of any crash, foo is either pointing to the old data or the new data. There simply isn't any inode with any "partially renamed state" mixing the two.