3 ms·
I think it's not just between files, but within a file. The backup process opens some N byte file and starts copying it. Some portion of the file is backed up
by kazinator 5y ago
I think it's not just between files, but within a file.
The backup process opens some N byte file and starts copying it. Some portion of the file is backed up until byte K. At that point the application writes a transaction whereby some of the data is written above K, and some below K. The backup continues and backs up half the transaction above K, combining that with the old data below K before the transaction happened.
If the application structures its scattered write transactions such that they write in increasing offset order, then maybe it's okay.
- liuliu 5y agoI see. Yes, that makes sense. Unless the application do write / rename trick, otherwise the backup process will see torn write.
- yakubin 5y agoYou write "transaction", but Unix filesystems don't know anything about any transactions from the application's point of view. To them it's just a bunch of write(2) calls. How is a ZFS snapshot going to make sure that it snapshots the state either before or after what an application considers a "transaction", but implements with a sequence of write(2), fsync(2) etc. calls? (EDIT: The rest of this paragraph is rescinded. See [1]) If an application uses SQLite, BerkeleyDB etc., then you don't need ZFS. XFS will be sufficient (and maybe even better?). If an application doesn't use any transaction-processing library ("database"), then I don't see how filesystem snapshots fix anything here --- you still can get an intermediate state in a snapshot. As far as I understand, the only thing snapshots make sure of is that you're looking at a consistent state of a filesystem from the point of view of the filesystem, so e.g. if there was a sequence of operations "remove file a/b", "create file c/d", then you're not going to record a state with both files "a/b" and "c/d" present --- something that could happen with a backup tool not using filesystem snapshots, if it reads directory "a" first and directory "c" second. But there are no assurances about application transactions. So if KeePassXC has a bug, then if there is a sequence of operations "remove password A", "add password B", then a backup tool may pick up a version of KeePassXC database which is going to contain both passwords A and B, even if the backup tool uses filesystem snapshots. [1]: At first I didn't get the example from TFA about Firefox and SQLite, but I've reconsidered and it makes sense now. I'll still say that you may get an intermediate state in a snapshot. However, snapshot prevents some classes of inconsistencies (though not all). Call me convinced about ZFS.
- kazinator 5y agoWhat ZFS will do it not capture a state before or after the transaction, but it will copy a state that the file actually has at some point during the transaction. The snapshot will have the state of the file which contains write_0, write_1, ...., write_k; and does not contain the effects of write_k+1, write_k+2, ... The serial copy of the file does not have that property. We may be able to find some k such that the backup contains some writes newer than k, while missing some older ones. The snapshot situation looks, to the application, like a crash and reboot, which it knows how to deal with (with test coverage and everything). The application recognizes that there is an uncommitted transaction and rolls things to the prior state. Or else sees that there is enough info to finish the transaction; just the commit part wasn't done. The missing old writes situation doesn't seem easily recoverable; I have no idea how you would code defensively against that, other than assuming that the backup copy is made serially, and make sure that in every transaction, the writes occur in increasing offset order: the same direction. If the backup and transaction race in the same direction, then it's impossible for the backup to have "holes" whereby some older writes are missing.
- yakubin 5y agoYou're right. (Didn't notice your reply before updating my comment.)
- a1369209993 5y ago> assuming that the backup copy is made serially, and make sure that in every transaction, the writes occur in increasing offset order: the same direction. If the backup and transaction race in the same direction, then it's impossible for the backup to have "holes" whereby some older writes are missing. Actually, that's the opposite of what you want: if the backup and application are racing in the same direction, then every write has a coinflip opportunity to either be included in the backup (if the application is slightly ahead and writes that block just before the backup reads it), or be ommited from the backup (if the backup is slightly ahead and reads that block just before the application writes it). Whereas if the application writes from the end of the file to the start, the write where it meets the backup coming the opposite direction will be arbitrarily kept or lost, but in theory (but not in practice), any writes before it (toward the end of the file) will always be kept, and any writes after it (toward the start of the file) will always be lost, which is the same semantics you'd get from a snapshot at the time of that write.