2 ms·
Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attem
by loeg 7y ago
Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent.
This sort of gets into it but isn't a fantastic source: https://www.enterprisestorageforum.com/storage-hardware/drive-reliability-studies-2.html https://www.enterprisestorageforum.com/storage-hardware/driv...
See also https://en.wikipedia.org/wiki/Data_corruption#Silent https://en.wikipedia.org/wiki/Data_corruption#Silent and https://en.wikipedia.org/wiki/Hard_disk_drive#ERRORRATESHANDLING https://en.wikipedia.org/wiki/Hard_disk_drive#ERRORRATESHAND... .
Notably, Apple's new filesystem introduced in 2017 does not detect data corruption; it only checksums metadata. https://en.wikipedia.org/wiki/Apple_File_System#Data_integrity https://en.wikipedia.org/wiki/Apple_File_System#Data_integri... Apparently Apple Engineers are of the opinion that Apple SSD and HDD media simply don't have silent data corruption. They have some pretty smart engineers, so maybe they're on to something. It does beggar the question, though: why have metadata checksums at all? http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/
Anyway, I'm not exactly supporting threat models that ignore silent data corruption. Just pointing out that they're not exactly as ridiculous as they might seem on their face.
- lmm 7y ago> Silent data corruption is not the typical failure mode for HDDs. Sure, it happens. Much more likely is that drive's internal ECC detects the bad sector, attempts to reread it several times, and if it fails, you get EIO back. Given this model, the various kinds of RAID know which drive had the erroneous sector, because it isn't silent. Linux md will do the completely wrong thing with this: it will treat that as a drive error, and if you accumulate a lot in a short time (such as rebuilding an arrary) then it will conclude that this is a failed drive and kick it out of the array.
- loeg 7y agoIt could always be worse — it could kick out the other drive!
- lmm 7y agoWell, with modern disk sizes it will almost certainly kick out all the drives sooner or later, making it essentially impossible to ever rebuild an array after a disk failure if you weren't scrubbing regularly before the disk failure (and dicey even if you were).
- loeg 7y agoI totally believe you but this has never actually been a problem for me with Linux md. I tend to replace drives when the very first error is detected, and for spinning rust I usually get a very small number of errors initially. Also, md on my system seems to automatically run scrubs weekly. Currently observing 0 for Raw_Read_Error_Rate and Reallocated_Sector_Ct for all drives, with 24k Power_On_Hours (>2.5 years).
- lmm 7y agoWeekly scrubs put you in a much better situation than not having them. I suspect your distribution must have done something to close the gap, because vanilla md does not put them in place and the tutorials do not always make it clear that setting up scrubs is not a nice-to-have thing but something that's absolutely critical if you want to have a hope of avoiding data loss. How many array rebuilds have you done? You only see the read errors when you go to actually read the sector in question, and even in a week I wouldn't want to rely on not having any sectors go bad.
- throwaway-9320 7y agoGiven enough writes, any SSD will eventually fail. As a related anecdote, I did some stress-testing on a ZFS filesystem with two Kingston NVMe SSD-s in mirrored configuration and at one point I got two random read errors on one drive, which subsequently got corrected by ZFS. Silent failures might not be the best phrase to use for these situations, but ZFS does let you know that corruption has taken place and you have to acknowledge that message for it to disappear.
- loeg 7y agoIf the SSD controller isn't bugged, lifetime write limits don't result in silent data corruption; they just result in errored writes and inability to persist new data. Apple may have decided to only qual and sell SSDs that they've tested to failure, rather than throwing darts at random. It's not clear to me if you're describing silent data corruption or detected data corruption (at the controller level), but either way Apple isn't selling Kingston NVMe SSDs; they have their own SSD controller and vendor the raw flash.