3 ms·

This article is surprisingly naive. I’ve worked at a well known enterprise storage company. The biggest thing drilled into us was that we couldn’t trust anythi

This article is surprisingly naive. I’ve worked at a well known enterprise storage company. The biggest thing drilled into us was that we couldn’t trust anything reported to us by the hard drives. Hard drives can report back to the OS that the write succeeded even when it didn’t. When something like this happens you’re in for a world of hurt, especially over time. This isn’t a bad sector, it’s corrupt data.

louwrentius 6y ago
Sure, at scale this kind of thing will happen. At scale, even small risks become a certainty. My article was more geared towards home / small SMB users. Silent data corruption will happen. Will you hit that at home? Probably not. But maybe people should stop buying Synologies, QNAPS and all that kind of hardware, I don't know ;-)
- remote_phone 6y ago
  “Probably not.” How can you even begin to quantify this risk if you have no idea what the problem is, or if you’ve never actually checked? That’s irresponsible to tell people “nah you being run into this problem” or “Eh, RAID5 works great!” when you’ve never looked into the stats or supported people who end up getting all their data lost? This is a real problem and anyone not running two drive redundancy in 2020 on data that they care about is going to lose their data eventually. If they don’t care about the data, then sure it’s not a big deal. I myself have 2 Synologies where I back up one to the other. The backup Synology is raid 5 because I don’t care so much about losing data, but my main drive is RAID 6.
  - magnawave 6y ago
    I think his point was about drives silently returning corrupt data, not that you shouldn’t have redundancy. Absolutely you should have redundancy and backups. At the (tens of) thousands of drives scale (and where you treat drives as cattle) having some extra checks and balances for what you write is an excellent idea. Even better is doing that in a distributed way so multiple machines make that same decision. Occasionally you will run into a drive where the firmware has jumped the shark or some such. (Or CPUs or memory or bus issue for that matter). But generally speaking drives DO know when they are returning bad data and will error before they will do that. The odds are about as good as other forms of hardware errors that will eat your data.
paulryanrogers 6y ago
What is the alternative? To read back the data immediately, periodically, separate-block-level check sums?
- mjb 6y ago
  Yes, that’s what you need to do if you want to approach the theoretical durability of any storage system. If you’ve got durability ‘budget’ elsewhere in your design, you might not have to do this.

louwrentius 6y ago

Sure, at scale this kind of thing will happen. At scale, even small risks become a certainty. My article was more geared towards home / small SMB users. Silent data corruption will happen. Will you hit that at home? Probably not. But maybe people should stop buying Synologies, QNAPS and all that kind of hardware, I don't know ;-)

remote_phone 6y ago
“Probably not.” How can you even begin to quantify this risk if you have no idea what the problem is, or if you’ve never actually checked? That’s irresponsible to tell people “nah you being run into this problem” or “Eh, RAID5 works great!” when you’ve never looked into the stats or supported people who end up getting all their data lost? This is a real problem and anyone not running two drive redundancy in 2020 on data that they care about is going to lose their data eventually. If they don’t care about the data, then sure it’s not a big deal. I myself have 2 Synologies where I back up one to the other. The backup Synology is raid 5 because I don’t care so much about losing data, but my main drive is RAID 6.
- magnawave 6y ago
  I think his point was about drives silently returning corrupt data, not that you shouldn’t have redundancy. Absolutely you should have redundancy and backups. At the (tens of) thousands of drives scale (and where you treat drives as cattle) having some extra checks and balances for what you write is an excellent idea. Even better is doing that in a distributed way so multiple machines make that same decision. Occasionally you will run into a drive where the firmware has jumped the shark or some such. (Or CPUs or memory or bus issue for that matter). But generally speaking drives DO know when they are returning bad data and will error before they will do that. The odds are about as good as other forms of hardware errors that will eat your data.

paulryanrogers 6y ago

What is the alternative? To read back the data immediately, periodically, separate-block-level check sums?

mjb 6y ago
Yes, that’s what you need to do if you want to approach the theoretical durability of any storage system. If you’ve got durability ‘budget’ elsewhere in your design, you might not have to do this.