6 ms·

> Data can be corrupted in anywhere in this chain. Of course, but most of this kind of corruption is trivial to recover from with just retries, that's why chec

> Data can be corrupted in anywhere in this chain. Of course, but most of this kind of corruption is trivial to recover from with just retries, that's why checksumming there works, but not on the filesystem level, where it can only detect corruption. What makes a bit more sense is something like erasure coding. Filesystem with erasure coding would probably be useful for all the broken FTLs on SD cards, eMMCs, SSDs, but at the end of the day wouldn't provide any significant level of resilience for a single disk to be worth the effort and the overhead. And checksumming alone still won't be useful.

vardump 7y ago
> Of course, but most of this kind of corruption is trivial to recover from with just retries... Retries won't help you when the data was corrupted on its way to be stored.
- zzzcpan 7y ago
  > Retries won't help you when the data was corrupted on its way to be stored. Why not? This is exactly how such corruption is traditionally fixed. On crc error data is rejected before it is stored and retry then fixes the problem. And it's not that big of a deal either, only applications can get data checksummed in the earliest possible moment anyway. The hard part is automatically recovering from data loss and corruption later, which also makes it even unnecessary to verify if data is stored without corruption.
  - throw0101a 7y ago
    > > Retries won't help you when the data was corrupted on its way to be stored. > Why not? 1. Data is fine in memory 2. Data is fine over PCIe. 3. Data is corrupted by firmware bug in storage controller. 4. Data (now corrupted) is sent to disk, which generates checksums for each LBA, and sends the bits to the storage medium. 5. Some time later the data is requested, disk reads bits from sectors, checksums pass, corrupted data is sent back up the stack to the application. This is not a fantasy scenario: * http://www.cs.toronto.edu/~bianca/papers/fast08.pdf http://www.cs.toronto.edu/~bianca/papers/fast08.pdf * https://storagemojo.com/2007/09/19/cerns-data-corruption-research/ https://storagemojo.com/2007/09/19/cerns-data-corruption-res...
    - zzzcpan 7y ago
      You are missing the point: checksums can't save you from data corruption and data loss. They are unnecessary without automatic recovery mechanisms and even make things worse.
      - throw0101a 7y ago
        > You are missing the point: checksums can't save you from data corruption and data loss. No, I am not. The point is: checksums allow you to know that data corruption happened in the first place. Knowing this, you can then make informed decisions. What saves you from "data corruption and data loss" is backups. But how do you know you have to go to backups if you're not aware that your data has problems? > They are unnecessary without automatic recovery mechanisms and even make things worse. Checksums allow you do know that there is a problem in the first place. If you don't know there's a problem, how can you invoke "recovery mechanisms", automatic or otherwise? If you use (say) ZFS with some level of RAID-Z over multiple disks, then you get an "automatic recover mechanism". But even for a single-disk situation, it would be handy (for me at least), to know a file is bad so I can restore it from (say) my Time Machine backup. But I have to know that the file is corrupted in the first place. And checksums would give me that.
        zzzcpan 7y ago
        > What saves you from "data corruption and data loss" is backups. But how do you know you have to go to backups if you're not aware that your data has problems? It doesn't actually. It introduces inconsistencies unless on first detected corruption your filesystem stops all reads and writes until you recover corrupted block from backup and you might not even have it in backup, because backup is not synchronous replication. > Checksums allow you do know that there is a problem in the first place. If you don't know there's a problem, how can you invoke "recovery mechanisms", automatic or otherwise? You do your own appropriate checksumming suitable for recovery mechanism that you have. It's hardly as primitive as you think, because you have to respond fast, without corruption, transparently on request, you also have to schedule a check whether you actually need to heal anything, maybe it was a transient error, then mark it as corrupted, find what to resync from where, all of which also has to be fast, which means a bit complicated algorithms with sort of checksum databases on each replica, etc.
throw0101a 7y ago
> Of course, but most of this kind of corruption is trivial to recover from with just retries, that's why checksumming there works, but not on the filesystem level, where it can only detect corruption. FS-level checksum saves you from the situation when the underlying storage thinks everything is okay but it is not: > This means there was a data error on the drive. But it’s worse than a typical data error — this is an error that was not detected by the hardware. * http://changelog.complete.org/archives/9769-silent-data-corruption-is-real http://changelog.complete.org/archives/9769-silent-data-corr... * https://research.cs.wisc.edu/wind/Publications/zfs-corruption-fast10.pdf https://research.cs.wisc.edu/wind/Publications/zfs-corruptio... * https://news.ycombinator.com/item?id=13851349 https://news.ycombinator.com/item?id=13851349 (via) It's not an either-or situation, but rather both-and hardware- and FS-level checks.
- zzzcpan 7y ago
  > FS-level checksum saves you from the situation when the underlying storage thinks everything is okay but it is not: It doesn't save you from anything, it just turns bit flips into full blocks of data loss and introduces overhead. If your application can deal with data loss it can deal with bit flips too. Filesystem-level checksumming is still useless.
  - throw0101a 7y ago
    It tells you that your data is corrupted, which gives you options on dealing with the situation. > If your application can deal with data loss it can deal with bit flips too. No, it may not. The file may mostly be okay except for a place where a single bit flip causes the value to go from positive to negative: * https://en.wikipedia.org/wiki/Signed_number_representations https://en.wikipedia.org/wiki/Signed_number_representations Hardware not-handling bit flips is a documented phenomenon per my citations above. It does cause issues further up the stack. FS-level checksumming gives people the option to at least detect these errors, and depending on the situation, also perhaps fix them.
    - zzzcpan 7y ago

vardump 7y ago

> Of course, but most of this kind of corruption is trivial to recover from with just retries... Retries won't help you when the data was corrupted on its way to be stored.

zzzcpan 7y ago
> Retries won't help you when the data was corrupted on its way to be stored. Why not? This is exactly how such corruption is traditionally fixed. On crc error data is rejected before it is stored and retry then fixes the problem. And it's not that big of a deal either, only applications can get data checksummed in the earliest possible moment anyway. The hard part is automatically recovering from data loss and corruption later, which also makes it even unnecessary to verify if data is stored without corruption.
- throw0101a 7y ago
  > > Retries won't help you when the data was corrupted on its way to be stored. > Why not? 1. Data is fine in memory 2. Data is fine over PCIe. 3. Data is corrupted by firmware bug in storage controller. 4. Data (now corrupted) is sent to disk, which generates checksums for each LBA, and sends the bits to the storage medium. 5. Some time later the data is requested, disk reads bits from sectors, checksums pass, corrupted data is sent back up the stack to the application. This is not a fantasy scenario: * http://www.cs.toronto.edu/~bianca/papers/fast08.pdf http://www.cs.toronto.edu/~bianca/papers/fast08.pdf * https://storagemojo.com/2007/09/19/cerns-data-corruption-research/ https://storagemojo.com/2007/09/19/cerns-data-corruption-res...
  - zzzcpan 7y ago
    You are missing the point: checksums can't save you from data corruption and data loss. They are unnecessary without automatic recovery mechanisms and even make things worse.
    - throw0101a 7y ago
      > You are missing the point: checksums can't save you from data corruption and data loss. No, I am not. The point is: checksums allow you to know that data corruption happened in the first place. Knowing this, you can then make informed decisions. What saves you from "data corruption and data loss" is backups. But how do you know you have to go to backups if you're not aware that your data has problems? > They are unnecessary without automatic recovery mechanisms and even make things worse. Checksums allow you do know that there is a problem in the first place. If you don't know there's a problem, how can you invoke "recovery mechanisms", automatic or otherwise? If you use (say) ZFS with some level of RAID-Z over multiple disks, then you get an "automatic recover mechanism". But even for a single-disk situation, it would be handy (for me at least), to know a file is bad so I can restore it from (say) my Time Machine backup. But I have to know that the file is corrupted in the first place. And checksums would give me that.
      - zzzcpan 7y ago
        > What saves you from "data corruption and data loss" is backups. But how do you know you have to go to backups if you're not aware that your data has problems? It doesn't actually. It introduces inconsistencies unless on first detected corruption your filesystem stops all reads and writes until you recover corrupted block from backup and you might not even have it in backup, because backup is not synchronous replication. > Checksums allow you do know that there is a problem in the first place. If you don't know there's a problem, how can you invoke "recovery mechanisms", automatic or otherwise? You do your own appropriate checksumming suitable for recovery mechanism that you have. It's hardly as primitive as you think, because you have to respond fast, without corruption, transparently on request, you also have to schedule a check whether you actually need to heal anything, maybe it was a transient error, then mark it as corrupted, find what to resync from where, all of which also has to be fast, which means a bit complicated algorithms with sort of checksum databases on each replica, etc.

throw0101a 7y ago

> Of course, but most of this kind of corruption is trivial to recover from with just retries, that's why checksumming there works, but not on the filesystem level, where it can only detect corruption. FS-level checksum saves you from the situation when the underlying storage thinks everything is okay but it is not: > This means there was a data error on the drive. But it’s worse than a typical data error — this is an error that was not detected by the hardware. * http://changelog.complete.org/archives/9769-silent-data-corruption-is-real http://changelog.complete.org/archives/9769-silent-data-corr... * https://research.cs.wisc.edu/wind/Publications/zfs-corruption-fast10.pdf https://research.cs.wisc.edu/wind/Publications/zfs-corruptio... * https://news.ycombinator.com/item?id=13851349 https://news.ycombinator.com/item?id=13851349 (via) It's not an either-or situation, but rather both-and hardware- and FS-level checks.

zzzcpan 7y ago
> FS-level checksum saves you from the situation when the underlying storage thinks everything is okay but it is not: It doesn't save you from anything, it just turns bit flips into full blocks of data loss and introduces overhead. If your application can deal with data loss it can deal with bit flips too. Filesystem-level checksumming is still useless.
- throw0101a 7y ago
  It tells you that your data is corrupted, which gives you options on dealing with the situation. > If your application can deal with data loss it can deal with bit flips too. No, it may not. The file may mostly be okay except for a place where a single bit flip causes the value to go from positive to negative: * https://en.wikipedia.org/wiki/Signed_number_representations https://en.wikipedia.org/wiki/Signed_number_representations Hardware not-handling bit flips is a documented phenomenon per my citations above. It does cause issues further up the stack. FS-level checksumming gives people the option to at least detect these errors, and depending on the situation, also perhaps fix them.
  - zzzcpan 7y ago