3 ms·
It sounds like cost is the primary factor, but diversity can also help mitigate the risk of unknown bugs. Lets say you found that HPE SSDs were super reliable
by kube-system 6y ago
It sounds like cost is the primary factor, but diversity can also help mitigate the risk of unknown bugs.
Lets say you found that HPE SSDs were super reliable over your test period so you decided to put everything on a bunch of new drives.
Then this hits, and 100% of your drives fail at the same time: https://www.pcmag.com/news/time-to-patch-hpe-ssds-will-fail-after-32768-hours https://www.pcmag.com/news/time-to-patch-hpe-ssds-will-fail-...
While these stats are a great analysis of random failures -- that's only one type of risk.
- hinkley 6y agoI have seen homogeneity 'kill' twice in my career. The first time was unprovisioned hardware, then second time we lost half of the hard drives assigned for dev machines in the space of about 9 weeks. I had so many processes in place by that point that it was a huge inconvenience (mostly due to whole disk encryption) but zero data loss. The worst thing you can do is to put all of your eggs into one basket. And bulk ordering might get you a set of drives all from the same batch. Bad batches will tend to all fail for a similar reason. Drives fail on a bell curve, right? So the first drive may fail way before any others, but in a RAID array, rebuilds are stressful. Eventually you will hit a statistical cluster. Multiple drives failing close together. If that happens during a rebuild, you will lose a RAID 5 array. If you are very lucky, your RAID 10 array loses two drives in the same mirror. If you have two failures during a long rebuild, even RAID 6 won't save you. I just bought a Synology box for home. This is my third and probably final RAID enclosure for personal use. I was having trouble finding Backblaze-tested drive models to populate it, so I filled it with drives from a Drobo and kept looking. Initially I had populated the Drobo with 4 drives I bought at once. When one failed, I bought 2 HGST drives and replaced a pair. When the new drives arrived, I started trying to cycle them through, and one of the drobo drives failed. I'll give you two guesses which one. There is, as far as I can tell, no prosumer multi-disk filesystem that uses consistent hashing to stripe+mirror files across an arbitrary number of disks, instead of the heavy linear algebra RAID5 relies on. It requires touching the whole disk on every rebuild, and I believe that's why Object Storage is slowly taking over from the top end. It's a simpler form of redundancy. I hope that it's worked its way down to my price range by the time the motherboard on the Synology burns out.
- andy4blaze 6y agoAndy from Backblaze here. Nice thinking about the bulk ordering and considerations for RAID. All things we have considered. We use our own Reed-Solomon encoding with a 17/3 set-up across 20 drives across 20 different systems, we call that a Tome. Then we have a specific protocol we follow as drives fail in a Tome to protect the data at all costs. We have the luxury for example to stop writing to a given Tome as we have plenty of others available. This takes a lot of the stress off of the system. Your thoughts on bulk buys and bad drive batches/models is solid. We test drives in small batches first, and we follow drive failures so we don't get to the point of hitting the wall. It would be great to mix and match drives, but you end up with a system that maxes out at the least performant drive. So not optimal.
- hinkley 6y agoI recall reading about your triple redundancy with R-S codes, but it's good to restate it for each audience. From the behavior of my RAID (which also uses Reed Solomon, doesn't it?) it feels like repairing an array takes time proportional to the size of the drive, not the size of the contents, and it feels like a waste. But it's possible that my comfort levels for available disk space are a lot more conservative than other people's, and so the difference is less pronounced in a 'normal' storage situation. For instance, an array that's at 80% capacity takes 25% longer to rebuild than I wish it would, whereas an array that's at 66% capacity takes 50% longer.
- TheDong 6y ago> it feels like repairing an array takes time proportional to the size of the drive, not the size of the contents This is only true for drive-level raid rather than filesystem level raid, or a non-raid solution like ceph's replication. ZFS's filesystem raid can repair a raid in time proportional to the amount of data stored in it. mdadm and raid controllers aren't aware of which parts of the block device are in use or not, and thus have to repair the whole drive. It's exceedingly likely that backblaze's solution does not require repairing entire block devices, but rather is likely to be closer to ceph, where only the in-use portion of a failed drive must be considered / must find a new home. I think raid and distributed storage systems (like backblaze or ceph) are more different than they are alike. > From the behavior of my RAID (which also uses Reed Solomon, doesn't it?) Maybe. mdadm raid5 doesn't, nor does mdadm raid1 or raid10. I think mdadm's raid6 does.