5 ms·

The issue is not a loudly failing disk. If you get read errors, or if the disk is simply offline, it's easy to recalculate the data. But what if you have silen

The issue is not a loudly failing disk. If you get read errors, or if the disk is simply offline, it's easy to recalculate the data. But what if you have silent data corruption: a single bit? Even if you have parity (raid5) or a duplicate (raid1), how do you know which part is correct, and which part to repair? Without checksums. I've had to restore 1TB from Crashplan. They apparently had checksums in their software, and something like 10-15 files were damaged. Without checksums, you simply don't know if your data is damaged, even on a single hard drive. ZFS gives you data integrity with checksums, and redundancy with mirrors or raid-z. This is why a regular scrub is recommended, you verify the checksums and repair if possible.

rrauenza 7y ago
I was won over to btrfs for my new NAS (~6 years ago!) when I saw a demo (which I expect is also valid on zfs) where the presenter starts with a mirrored setup, finds the exact block offset location of the file's contents and then overwrites a byte in it with a zero using /usr/bin/dd against the raw disk, /dev/sdX. He then reads the file, uncorrupted, with a kernel warning about a bad checksum that was fixed (from the mirror) during the read. This demonstration is why I switched to btrfs (and ECC) for my new home server/NAS. That, along with compression sealed the deal. I was disappointed about not having dedupe at the time, but it probably really wouldn't have saved me that much space. (I keep meaning to run the dedupe tool to see what it might save me now.)
loeg 7y ago
Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent. This sort of gets into it but isn't a fantastic source: https://www.enterprisestorageforum.com/storage-hardware/drive-reliability-studies-2.html https://www.enterprisestorageforum.com/storage-hardware/driv... See also https://en.wikipedia.org/wiki/Data_corruption#Silent https://en.wikipedia.org/wiki/Data_corruption#Silent and https://en.wikipedia.org/wiki/Hard_disk_drive#ERRORRATESHANDLING https://en.wikipedia.org/wiki/Hard_disk_drive#ERRORRATESHAND... . Notably, Apple's new filesystem introduced in 2017 does not detect data corruption; it only checksums metadata. https://en.wikipedia.org/wiki/Apple_File_System#Data_integrity https://en.wikipedia.org/wiki/Apple_File_System#Data_integri... Apparently Apple Engineers are of the opinion that Apple SSD and HDD media simply don't have silent data corruption. They have some pretty smart engineers, so maybe they're on to something. It does beggar the question, though: why have metadata checksums at all? http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ Anyway, I'm not exactly supporting threat models that ignore silent data corruption. Just pointing out that they're not exactly as ridiculous as they might seem on their face.
- lmm 7y ago
  > Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent. Linux md will do the completely wrong thing with this: it will treat that as a drive error, and if you accumulate a lot in a short time (such as rebuilding an arrary) then it will conclude that this is a failed drive and kick it out of the array.
  - loeg 7y ago
    It could always be worse — it could kick out the other drive!
    - lmm 7y ago
      Well, with modern disk sizes it will almost certainly kick out all the drives sooner or later, making it essentially impossible to ever rebuild an array after a disk failure if you weren't scrubbing regularly before the disk failure (and dicey even if you were).
      - loeg 7y ago
        I totally believe you but this has never actually been a problem for me with Linux md. I tend to replace drives when the very first error is detected, and for spinning rust I usually get a very small number of errors initially. Also, md on my system seems to automatically run scrubs weekly. Currently observing 0 for Raw_Read_Error_Rate and Reallocated_Sector_Ct for all drives, with 24k Power_On_Hours (>2.5 years).
        lmm 7y ago
        Weekly scrubs put you in a much better situation than not having them. I suspect your distribution must have done something to close the gap, because vanilla md does not put them in place and the tutorials do not always make it clear that setting up scrubs is not a nice-to-have thing but something that's absolutely critical if you want to have a hope of avoiding data loss. How many array rebuilds have you done? You only see the read errors when you go to actually read the sector in question, and even in a week I wouldn't want to rely on not having any sectors go bad.

rrauenza 7y ago

I was won over to btrfs for my new NAS (~6 years ago!) when I saw a demo (which I expect is also valid on zfs) where the presenter starts with a mirrored setup, finds the exact block offset location of the file's contents and then overwrites a byte in it with a zero using /usr/bin/dd against the raw disk, /dev/sdX. He then reads the file, uncorrupted, with a kernel warning about a bad checksum that was fixed (from the mirror) during the read. This demonstration is why I switched to btrfs (and ECC) for my new home server/NAS. That, along with compression sealed the deal. I was disappointed about not having dedupe at the time, but it probably really wouldn't have saved me that much space. (I keep meaning to run the dedupe tool to see what it might save me now.)

loeg 7y ago

Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent. This sort of gets into it but isn't a fantastic source: https://www.enterprisestorageforum.com/storage-hardware/drive-reliability-studies-2.html https://www.enterprisestorageforum.com/storage-hardware/driv... See also https://en.wikipedia.org/wiki/Data_corruption#Silent https://en.wikipedia.org/wiki/Data_corruption#Silent and https://en.wikipedia.org/wiki/Hard_disk_drive#ERRORRATESHANDLING https://en.wikipedia.org/wiki/Hard_disk_drive#ERRORRATESHAND... . Notably, Apple's new filesystem introduced in 2017 does not detect data corruption; it only checksums metadata. https://en.wikipedia.org/wiki/Apple_File_System#Data_integrity https://en.wikipedia.org/wiki/Apple_File_System#Data_integri... Apparently Apple Engineers are of the opinion that Apple SSD and HDD media simply don't have silent data corruption. They have some pretty smart engineers, so maybe they're on to something. It does beggar the question, though: why have metadata checksums at all? http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ Anyway, I'm not exactly supporting threat models that ignore silent data corruption. Just pointing out that they're not exactly as ridiculous as they might seem on their face.

lmm 7y ago
> Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent. Linux md will do the completely wrong thing with this: it will treat that as a drive error, and if you accumulate a lot in a short time (such as rebuilding an arrary) then it will conclude that this is a failed drive and kick it out of the array.
- loeg 7y ago
  It could always be worse — it could kick out the other drive!
  - lmm 7y ago
    Well, with modern disk sizes it will almost certainly kick out all the drives sooner or later, making it essentially impossible to ever rebuild an array after a disk failure if you weren't scrubbing regularly before the disk failure (and dicey even if you were).
    - loeg 7y ago
      I totally believe you but this has never actually been a problem for me with Linux md. I tend to replace drives when the very first error is detected, and for spinning rust I usually get a very small number of errors initially. Also, md on my system seems to automatically run scrubs weekly. Currently observing 0 for Raw_Read_Error_Rate and Reallocated_Sector_Ct for all drives, with 24k Power_On_Hours (>2.5 years).
      - lmm 7y ago
        Weekly scrubs put you in a much better situation than not having them. I suspect your distribution must have done something to close the gap, because vanilla md does not put them in place and the tutorials do not always make it clear that setting up scrubs is not a nice-to-have thing but something that's absolutely critical if you want to have a hope of avoiding data loss. How many array rebuilds have you done? You only see the read errors when you go to actually read the sector in question, and even in a week I wouldn't want to rely on not having any sectors go bad.