3 ms·

You are missing the point: checksums can't save you from data corruption and data loss. They are unnecessary without automatic recovery mechanisms and even make

You are missing the point: checksums can't save you from data corruption and data loss. They are unnecessary without automatic recovery mechanisms and even make things worse.

throw0101a 7y ago
> You are missing the point: checksums can't save you from data corruption and data loss. No, I am not. The point is: checksums allow you to know that data corruption happened in the first place. Knowing this, you can then make informed decisions. What saves you from "data corruption and data loss" is backups. But how do you know you have to go to backups if you're not aware that your data has problems? > They are unnecessary without automatic recovery mechanisms and even make things worse. Checksums allow you do know that there is a problem in the first place. If you don't know there's a problem, how can you invoke "recovery mechanisms", automatic or otherwise? If you use (say) ZFS with some level of RAID-Z over multiple disks, then you get an "automatic recover mechanism". But even for a single-disk situation, it would be handy (for me at least), to know a file is bad so I can restore it from (say) my Time Machine backup. But I have to know that the file is corrupted in the first place. And checksums would give me that.
- zzzcpan 7y ago
  > What saves you from "data corruption and data loss" is backups. But how do you know you have to go to backups if you're not aware that your data has problems? It doesn't actually. It introduces inconsistencies unless on first detected corruption your filesystem stops all reads and writes until you recover corrupted block from backup and you might not even have it in backup, because backup is not synchronous replication. > Checksums allow you do know that there is a problem in the first place. If you don't know there's a problem, how can you invoke "recovery mechanisms", automatic or otherwise? You do your own appropriate checksumming suitable for recovery mechanism that you have. It's hardly as primitive as you think, because you have to respond fast, without corruption, transparently on request, you also have to schedule a check whether you actually need to heal anything, maybe it was a transient error, then mark it as corrupted, find what to resync from where, all of which also has to be fast, which means a bit complicated algorithms with sort of checksum databases on each replica, etc.
  - throw0101a 7y ago
    What do you do as a living? Because I think we are living in different realities. I, as a sysadmin, have been helped by (ZFS) checksums multiple times. This includes times where I had to go to tape manually because the really expensive SAN has said "everything is fine" when it wasn't. So telling me "checksums do not help" is contrary to my life experience.
    - zzzcpan 7y ago
      I mostly speak from experience designing, implementing and operating a distributed database/storage system and traditional storages, databases before that.
      - throw0101a 7y ago
        And you've never seen the storage stack give back bad data? You seem to be saying "checksums don't matter because there may be nothing we can do about about the corrupted data". So we should just blindly accept all the bits from the lower levels of the stack? We should do nothing to verify that those levels of the stack are working correctly? You've mentioned application-level checksumming: yes, that would be good. But until every single dev adds code to checksum (and correct?) their data structures and file formats, having more checksums allows for beter information about when corruption occurs IMHO.
        zzzcpan 7y ago
        My point is checksumming is not a solution for data corruption, never was, never will be. It's only a tiny part of more complex algorithms and systems that detect and fix corrupted data automatically, never something usable on its own.
        throw0101a 7y ago
        Checksumming is necessary, but not sufficient, for proper data integrity. But if you don't have it, you can't do anything else. So why does it seem you are agruing against checksums?

You are missing the point: checksums can't save you from data corruption and data loss. They are unnecessary without automatic recovery mechanisms and even make things worse.

throw0101a 7y ago

> You are missing the point: checksums can't save you from data corruption and data loss. No, I am not. The point is: checksums allow you to know that data corruption happened in the first place. Knowing this, you can then make informed decisions. What saves you from "data corruption and data loss" is backups. But how do you know you have to go to backups if you're not aware that your data has problems? > They are unnecessary without automatic recovery mechanisms and even make things worse. Checksums allow you do know that there is a problem in the first place. If you don't know there's a problem, how can you invoke "recovery mechanisms", automatic or otherwise? If you use (say) ZFS with some level of RAID-Z over multiple disks, then you get an "automatic recover mechanism". But even for a single-disk situation, it would be handy (for me at least), to know a file is bad so I can restore it from (say) my Time Machine backup. But I have to know that the file is corrupted in the first place. And checksums would give me that.

zzzcpan 7y ago
> What saves you from "data corruption and data loss" is backups. But how do you know you have to go to backups if you're not aware that your data has problems? It doesn't actually. It introduces inconsistencies unless on first detected corruption your filesystem stops all reads and writes until you recover corrupted block from backup and you might not even have it in backup, because backup is not synchronous replication. > Checksums allow you do know that there is a problem in the first place. If you don't know there's a problem, how can you invoke "recovery mechanisms", automatic or otherwise? You do your own appropriate checksumming suitable for recovery mechanism that you have. It's hardly as primitive as you think, because you have to respond fast, without corruption, transparently on request, you also have to schedule a check whether you actually need to heal anything, maybe it was a transient error, then mark it as corrupted, find what to resync from where, all of which also has to be fast, which means a bit complicated algorithms with sort of checksum databases on each replica, etc.
- throw0101a 7y ago
  What do you do as a living? Because I think we are living in different realities. I, as a sysadmin, have been helped by (ZFS) checksums multiple times. This includes times where I had to go to tape manually because the really expensive SAN has said "everything is fine" when it wasn't. So telling me "checksums do not help" is contrary to my life experience.
  - zzzcpan 7y ago
    I mostly speak from experience designing, implementing and operating a distributed database/storage system and traditional storages, databases before that.
    - throw0101a 7y ago
      And you've never seen the storage stack give back bad data? You seem to be saying "checksums don't matter because there may be nothing we can do about about the corrupted data". So we should just blindly accept all the bits from the lower levels of the stack? We should do nothing to verify that those levels of the stack are working correctly? You've mentioned application-level checksumming: yes, that would be good. But until every single dev adds code to checksum (and correct?) their data structures and file formats, having more checksums allows for beter information about when corruption occurs IMHO.
      - zzzcpan 7y ago
        My point is checksumming is not a solution for data corruption, never was, never will be. It's only a tiny part of more complex algorithms and systems that detect and fix corrupted data automatically, never something usable on its own.
        throw0101a 7y ago
        Checksumming is necessary, but not sufficient, for proper data integrity. But if you don't have it, you can't do anything else. So why does it seem you are agruing against checksums?