3 ms·

I think we need to be a bit more concrete than analogies for my brain to process. So here we go: If your application relies on multi-file collaborated persiste

by liuliu 4y ago

I think we need to be a bit more concrete than analogies for my brain to process. So here we go: If your application relies on multi-file collaborated persisted states (for example, an append-only log and a database snapshot), application can make reasonable assumptions on when a file state is committed, such as a database snapshot is `fsync`ed before the append-only log started. Ideally, even on a filesystem without transaction support, this order is preserved in time. However, it may not be preserved for a "file-to-file" backup system because it can loop over the append-only log file first before the database snapshot, causing ordering issues. That could result a "ABA" problem, where the append-only log is corresponding to an older database snapshot, and potentially causes issues. That has been said, is this a common scenario (multi-file persisted state) for applications? (I believe SQLite handles this particular ordering issue fine). I am not sure, and just want to call out filesystem snapshot solves a very particular problem. If your application relies only one file for the state, or these files are orthogonal to each other (.xlsx or .docx or any .markdown, .mp4, .jpg files), it is a non-issue. And if you need to backup a database, you better do that with the database provided tools.

kazinator 4y ago
I think it's not just between files, but within a file. The backup process opens some N byte file and starts copying it. Some portion of the file is backed up until byte K. At that point the application writes a transaction whereby some of the data is written above K, and some below K. The backup continues and backs up half the transaction above K, combining that with the old data below K before the transaction happened. If the application structures its scattered write transactions such that they write in increasing offset order, then maybe it's okay.
- liuliu 4y ago
  I see. Yes, that makes sense. Unless the application do write / rename trick, otherwise the backup process will see torn write.
- yakubin 4y ago
  You write "transaction", but Unix filesystems don't know anything about any transactions from the application's point of view. To them it's just a bunch of write(2) calls. How is a ZFS snapshot going to make sure that it snapshots the state either before or after what an application considers a "transaction", but implements with a sequence of write(2), fsync(2) etc. calls? (EDIT: The rest of this paragraph is rescinded. See [1]) If an application uses SQLite, BerkeleyDB etc., then you don't need ZFS. XFS will be sufficient (and maybe even better?). If an application doesn't use any transaction-processing library ("database"), then I don't see how filesystem snapshots fix anything here --- you still can get an intermediate state in a snapshot. As far as I understand, the only thing snapshots make sure of is that you're looking at a consistent state of a filesystem from the point of view of the filesystem, so e.g. if there was a sequence of operations "remove file a/b", "create file c/d", then you're not going to record a state with both files "a/b" and "c/d" present --- something that could happen with a backup tool not using filesystem snapshots, if it reads directory "a" first and directory "c" second. But there are no assurances about application transactions. So if KeePassXC has a bug, then if there is a sequence of operations "remove password A", "add password B", then a backup tool may pick up a version of KeePassXC database which is going to contain both passwords A and B, even if the backup tool uses filesystem snapshots. [1]: At first I didn't get the example from TFA about Firefox and SQLite, but I've reconsidered and it makes sense now. I'll still say that you may get an intermediate state in a snapshot. However, snapshot prevents some classes of inconsistencies (though not all). Call me convinced about ZFS.
  - kazinator 4y ago
    What ZFS will do it not capture a state before or after the transaction, but it will copy a state that the file actually has at some point during the transaction. The snapshot will have the state of the file which contains write_0, write_1, ...., write_k; and does not contain the effects of write_k+1, write_k+2, ... The serial copy of the file does not have that property. We may be able to find some k such that the backup contains some writes newer than k, while missing some older ones. The snapshot situation looks, to the application, like a crash and reboot, which it knows how to deal with (with test coverage and everything). The application recognizes that there is an uncommitted transaction and rolls things to the prior state. Or else sees that there is enough info to finish the transaction; just the commit part wasn't done. The missing old writes situation doesn't seem easily recoverable; I have no idea how you would code defensively against that, other than assuming that the backup copy is made serially, and make sure that in every transaction, the writes occur in increasing offset order: the same direction. If the backup and transaction race in the same direction, then it's impossible for the backup to have "holes" whereby some older writes are missing.

by liuliu 4y ago

kazinator 4y ago

I think it's not just between files, but within a file. The backup process opens some N byte file and starts copying it. Some portion of the file is backed up until byte K. At that point the application writes a transaction whereby some of the data is written above K, and some below K. The backup continues and backs up half the transaction above K, combining that with the old data below K before the transaction happened. If the application structures its scattered write transactions such that they write in increasing offset order, then maybe it's okay.

liuliu 4y ago
I see. Yes, that makes sense. Unless the application do write / rename trick, otherwise the backup process will see torn write.
yakubin 4y ago
You write "transaction", but Unix filesystems don't know anything about any transactions from the application's point of view. To them it's just a bunch of write(2) calls. How is a ZFS snapshot going to make sure that it snapshots the state either before or after what an application considers a "transaction", but implements with a sequence of write(2), fsync(2) etc. calls? (EDIT: The rest of this paragraph is rescinded. See [1]) If an application uses SQLite, BerkeleyDB etc., then you don't need ZFS. XFS will be sufficient (and maybe even better?). If an application doesn't use any transaction-processing library ("database"), then I don't see how filesystem snapshots fix anything here --- you still can get an intermediate state in a snapshot. As far as I understand, the only thing snapshots make sure of is that you're looking at a consistent state of a filesystem from the point of view of the filesystem, so e.g. if there was a sequence of operations "remove file a/b", "create file c/d", then you're not going to record a state with both files "a/b" and "c/d" present --- something that could happen with a backup tool not using filesystem snapshots, if it reads directory "a" first and directory "c" second. But there are no assurances about application transactions. So if KeePassXC has a bug, then if there is a sequence of operations "remove password A", "add password B", then a backup tool may pick up a version of KeePassXC database which is going to contain both passwords A and B, even if the backup tool uses filesystem snapshots. [1]: At first I didn't get the example from TFA about Firefox and SQLite, but I've reconsidered and it makes sense now. I'll still say that you may get an intermediate state in a snapshot. However, snapshot prevents some classes of inconsistencies (though not all). Call me convinced about ZFS.
- kazinator 4y ago
  What ZFS will do it not capture a state before or after the transaction, but it will copy a state that the file actually has at some point during the transaction. The snapshot will have the state of the file which contains write_0, write_1, ...., write_k; and does not contain the effects of write_k+1, write_k+2, ... The serial copy of the file does not have that property. We may be able to find some k such that the backup contains some writes newer than k, while missing some older ones. The snapshot situation looks, to the application, like a crash and reboot, which it knows how to deal with (with test coverage and everything). The application recognizes that there is an uncommitted transaction and rolls things to the prior state. Or else sees that there is enough info to finish the transaction; just the commit part wasn't done. The missing old writes situation doesn't seem easily recoverable; I have no idea how you would code defensively against that, other than assuming that the backup copy is made serially, and make sure that in every transaction, the writes occur in increasing offset order: the same direction. If the backup and transaction race in the same direction, then it's impossible for the backup to have "holes" whereby some older writes are missing.