4 ms·

It sounds like you are suggesting that checksums and other integrity features obviate the need to back up your important files.

boomboomsubban 9y ago
They're part of the same process, you need the checksums to know your backup is reliable and to tell you when your main files become corrupt. Backups alone aren't enough either.
- zzzcpan 9y ago
  No, it doesn't work like that, backup is not redundancy. To detect a bad block you first need to read it, you can't just know when a file becomes corrupt. And reading every block is very expensive and doesn't provide predictable guarantees. Instead proper backup has built in redundancy to account for lost blocks and so does proper data storage. Checksums are not visible to users in neither case. And backup has to be incremental to prevent disasters.
_emacsomancer_ 9y ago
Of course not. But using fss lacking these features the backups end up 'faithfully' (and uselessly) propogating bitrotted files.
xoa 9y ago
I don't understand how you could have possibly gotten that, particularly given that I very explicitly finished with "that ZFS or the like are far better primitives for a data storage scheme then older FS". Primitives. Other components are still important. But backups in no way whatsoever inherently provide integrity, that's not the threat scenario they deal with. A non-verifying backup (quite typical, especially historically) will cheerfully and unquestioningly backup corrupted data, and can become corrupted itself over time. When you're considering data maintenance over the course of not just years but decades, "oh just backup" isn't at all enough without an integrity story to go with it. Now of course it is possible to build levels of verification higher up the stack, but the problem with a lot of the ones available to general user is that they either leave holes in various places or they're just plain too much of a PITA (and "PITA" here includes a significant enough level of slow down) or both. It's much easier to do a good job if the FS itself is reasonably trustable and has verification and basic repair built in. FWIW I do use tools like par2 for cold or cool backups of ZFS dumps to targets like tape or Glacier now, but it's still a big help to my sanity to have as much of that dealt with at a low, automated level for hot/warm systems where convenience is critical.

boomboomsubban 9y ago

They're part of the same process, you need the checksums to know your backup is reliable and to tell you when your main files become corrupt. Backups alone aren't enough either.

zzzcpan 9y ago
No, it doesn't work like that, backup is not redundancy. To detect a bad block you first need to read it, you can't just know when a file becomes corrupt. And reading every block is very expensive and doesn't provide predictable guarantees. Instead proper backup has built in redundancy to account for lost blocks and so does proper data storage. Checksums are not visible to users in neither case. And backup has to be incremental to prevent disasters.

_emacsomancer_ 9y ago

Of course not. But using fss lacking these features the backups end up 'faithfully' (and uselessly) propogating bitrotted files.

xoa 9y ago

I don't understand how you could have possibly gotten that, particularly given that I very explicitly finished with "that ZFS or the like are far better primitives for a data storage scheme then older FS". Primitives. Other components are still important. But backups in no way whatsoever inherently provide integrity, that's not the threat scenario they deal with. A non-verifying backup (quite typical, especially historically) will cheerfully and unquestioningly backup corrupted data, and can become corrupted itself over time. When you're considering data maintenance over the course of not just years but decades, "oh just backup" isn't at all enough without an integrity story to go with it. Now of course it is possible to build levels of verification higher up the stack, but the problem with a lot of the ones available to general user is that they either leave holes in various places or they're just plain too much of a PITA (and "PITA" here includes a significant enough level of slow down) or both. It's much easier to do a good job if the FS itself is reasonably trustable and has verification and basic repair built in. FWIW I do use tools like par2 for cold or cool backups of ZFS dumps to targets like tape or Glacier now, but it's still a big help to my sanity to have as much of that dealt with at a low, automated level for hot/warm systems where convenience is critical.