5 ms·

It's interesting that, despite Seagate being relatively bad compared to HGST, Backblaze keeps installing more and more Seagate drives and not that many HGST dri

by ebg13 6y ago

It's interesting that, despite Seagate being relatively bad compared to HGST, Backblaze keeps installing more and more Seagate drives and not that many HGST drives. I guess the analysis that's missing here is the cost per drive hour?

atYevP 6y ago
Yev from Backblaze here -> Yes, we tend to weigh heavily in favor of price and Seagate thus far as been affordable with performance that meets our requirements, so that's why you'll see those drives in higher numbers. We're also trying to diversify our fleet a bit, so it's not purely one vendor, but we do try to keep an eye on cost since we run a pretty lean ship!
AnIdiotOnTheNet 6y ago
Yeah, pretty much the same reason people keep buying Seagate drives generally: They cost less and they aren't so much worse that it is worth paying a higher price for a drive that's less likely to fail.
kube-system 6y ago
It sounds like cost is the primary factor, but diversity can also help mitigate the risk of unknown bugs. Lets say you found that HPE SSDs were super reliable over your test period so you decided to put everything on a bunch of new drives. Then this hits, and 100% of your drives fail at the same time: https://www.pcmag.com/news/time-to-patch-hpe-ssds-will-fail-after-32768-hours https://www.pcmag.com/news/time-to-patch-hpe-ssds-will-fail-... While these stats are a great analysis of random failures -- that's only one type of risk.
- hinkley 6y ago
  I have seen homogeneity 'kill' twice in my career. The first time was unprovisioned hardware, then second time we lost half of the hard drives assigned for dev machines in the space of about 9 weeks. I had so many processes in place by that point that it was a huge inconvenience (mostly due to whole disk encryption) but zero data loss. The worst thing you can do is to put all of your eggs into one basket. And bulk ordering might get you a set of drives all from the same batch. Bad batches will tend to all fail for a similar reason. Drives fail on a bell curve, right? So the first drive may fail way before any others, but in a RAID array, rebuilds are stressful. Eventually you will hit a statistical cluster. Multiple drives failing close together. If that happens during a rebuild, you will lose a RAID 5 array. If you are very lucky, your RAID 10 array loses two drives in the same mirror. If you have two failures during a long rebuild, even RAID 6 won't save you. I just bought a Synology box for home. This is my third and probably final RAID enclosure for personal use. I was having trouble finding Backblaze-tested drive models to populate it, so I filled it with drives from a Drobo and kept looking. Initially I had populated the Drobo with 4 drives I bought at once. When one failed, I bought 2 HGST drives and replaced a pair. When the new drives arrived, I started trying to cycle them through, and one of the drobo drives failed. I'll give you two guesses which one. There is, as far as I can tell, no prosumer multi-disk filesystem that uses consistent hashing to stripe+mirror files across an arbitrary number of disks, instead of the heavy linear algebra RAID5 relies on. It requires touching the whole disk on every rebuild, and I believe that's why Object Storage is slowly taking over from the top end. It's a simpler form of redundancy. I hope that it's worked its way down to my price range by the time the motherboard on the Synology burns out.
  - andy4blaze 6y ago
    Andy from Backblaze here. Nice thinking about the bulk ordering and considerations for RAID. All things we have considered. We use our own Reed-Solomon encoding with a 17/3 set-up across 20 drives across 20 different systems, we call that a Tome. Then we have a specific protocol we follow as drives fail in a Tome to protect the data at all costs. We have the luxury for example to stop writing to a given Tome as we have plenty of others available. This takes a lot of the stress off of the system. Your thoughts on bulk buys and bad drive batches/models is solid. We test drives in small batches first, and we follow drive failures so we don't get to the point of hitting the wall. It would be great to mix and match drives, but you end up with a system that maxes out at the least performant drive. So not optimal.
    - hinkley 6y ago
      I recall reading about your triple redundancy with R-S codes, but it's good to restate it for each audience. From the behavior of my RAID (which also uses Reed Solomon, doesn't it?) it feels like repairing an array takes time proportional to the size of the drive, not the size of the contents, and it feels like a waste. But it's possible that my comfort levels for available disk space are a lot more conservative than other people's, and so the difference is less pronounced in a 'normal' storage situation. For instance, an array that's at 80% capacity takes 25% longer to rebuild than I wish it would, whereas an array that's at 66% capacity takes 50% longer.
      - TheDong 6y ago
        > it feels like repairing an array takes time proportional to the size of the drive, not the size of the contents This is only true for drive-level raid rather than filesystem level raid, or a non-raid solution like ceph's replication. ZFS's filesystem raid can repair a raid in time proportional to the amount of data stored in it. mdadm and raid controllers aren't aware of which parts of the block device are in use or not, and thus have to repair the whole drive. It's exceedingly likely that backblaze's solution does not require repairing entire block devices, but rather is likely to be closer to ceph, where only the in-use portion of a failed drive must be considered / must find a new home. I think raid and distributed storage systems (like backblaze or ceph) are more different than they are alike. > From the behavior of my RAID (which also uses Reed Solomon, doesn't it?) Maybe. mdadm raid5 doesn't, nor does mdadm raid1 or raid10. I think mdadm's raid6 does.
hinkley 6y ago
As I recall, Backblaze does what some other companies have also figured out: amortized maintenance costs across racks. Long ago we hit a point where power and cooling, not floor space, were the biggest bottlenecks for many data centers. Those numbers shift around, but during that era a bunch of people noticed that it's less labor intensive to fill an entire rack at a time. Put a few extra racks in as spares, and as hardware fails you power them off and power up new machines in the spare rack. When X servers in a rack are dead, you migrate everything else out of the rack, then strip it to the rails and put brand new hardware in it. You increase your server density twice in that case. Event-driven maintenance is disruptive to other work. It tends to have a ticking clock involved, so you likely assign it to more experienced people, which is 2 opportunity costs in one. Rack-at-a-time is batch processing. You can schedule it, you can assign multiple people to it, and speed is less of an issue. I might have the new guys stripping a decommed rack, use building a new one as training for four people and one teacher. You put a third of it together, I'll come tell you why the wiring is wrong, lather, rinse, repeat.

by ebg13 6y ago

atYevP 6y ago

Yev from Backblaze here -> Yes, we tend to weigh heavily in favor of price and Seagate thus far as been affordable with performance that meets our requirements, so that's why you'll see those drives in higher numbers. We're also trying to diversify our fleet a bit, so it's not purely one vendor, but we do try to keep an eye on cost since we run a pretty lean ship!

AnIdiotOnTheNet 6y ago

Yeah, pretty much the same reason people keep buying Seagate drives generally: They cost less and they aren't so much worse that it is worth paying a higher price for a drive that's less likely to fail.

kube-system 6y ago

It sounds like cost is the primary factor, but diversity can also help mitigate the risk of unknown bugs. Lets say you found that HPE SSDs were super reliable over your test period so you decided to put everything on a bunch of new drives. Then this hits, and 100% of your drives fail at the same time: https://www.pcmag.com/news/time-to-patch-hpe-ssds-will-fail-after-32768-hours https://www.pcmag.com/news/time-to-patch-hpe-ssds-will-fail-... While these stats are a great analysis of random failures -- that's only one type of risk.

hinkley 6y ago
I have seen homogeneity 'kill' twice in my career. The first time was unprovisioned hardware, then second time we lost half of the hard drives assigned for dev machines in the space of about 9 weeks. I had so many processes in place by that point that it was a huge inconvenience (mostly due to whole disk encryption) but zero data loss. The worst thing you can do is to put all of your eggs into one basket. And bulk ordering might get you a set of drives all from the same batch. Bad batches will tend to all fail for a similar reason. Drives fail on a bell curve, right? So the first drive may fail way before any others, but in a RAID array, rebuilds are stressful. Eventually you will hit a statistical cluster. Multiple drives failing close together. If that happens during a rebuild, you will lose a RAID 5 array. If you are very lucky, your RAID 10 array loses two drives in the same mirror. If you have two failures during a long rebuild, even RAID 6 won't save you. I just bought a Synology box for home. This is my third and probably final RAID enclosure for personal use. I was having trouble finding Backblaze-tested drive models to populate it, so I filled it with drives from a Drobo and kept looking. Initially I had populated the Drobo with 4 drives I bought at once. When one failed, I bought 2 HGST drives and replaced a pair. When the new drives arrived, I started trying to cycle them through, and one of the drobo drives failed. I'll give you two guesses which one. There is, as far as I can tell, no prosumer multi-disk filesystem that uses consistent hashing to stripe+mirror files across an arbitrary number of disks, instead of the heavy linear algebra RAID5 relies on. It requires touching the whole disk on every rebuild, and I believe that's why Object Storage is slowly taking over from the top end. It's a simpler form of redundancy. I hope that it's worked its way down to my price range by the time the motherboard on the Synology burns out.
- andy4blaze 6y ago
  Andy from Backblaze here. Nice thinking about the bulk ordering and considerations for RAID. All things we have considered. We use our own Reed-Solomon encoding with a 17/3 set-up across 20 drives across 20 different systems, we call that a Tome. Then we have a specific protocol we follow as drives fail in a Tome to protect the data at all costs. We have the luxury for example to stop writing to a given Tome as we have plenty of others available. This takes a lot of the stress off of the system. Your thoughts on bulk buys and bad drive batches/models is solid. We test drives in small batches first, and we follow drive failures so we don't get to the point of hitting the wall. It would be great to mix and match drives, but you end up with a system that maxes out at the least performant drive. So not optimal.
  - hinkley 6y ago
    I recall reading about your triple redundancy with R-S codes, but it's good to restate it for each audience. From the behavior of my RAID (which also uses Reed Solomon, doesn't it?) it feels like repairing an array takes time proportional to the size of the drive, not the size of the contents, and it feels like a waste. But it's possible that my comfort levels for available disk space are a lot more conservative than other people's, and so the difference is less pronounced in a 'normal' storage situation. For instance, an array that's at 80% capacity takes 25% longer to rebuild than I wish it would, whereas an array that's at 66% capacity takes 50% longer.
    - TheDong 6y ago
      > it feels like repairing an array takes time proportional to the size of the drive, not the size of the contents This is only true for drive-level raid rather than filesystem level raid, or a non-raid solution like ceph's replication. ZFS's filesystem raid can repair a raid in time proportional to the amount of data stored in it. mdadm and raid controllers aren't aware of which parts of the block device are in use or not, and thus have to repair the whole drive. It's exceedingly likely that backblaze's solution does not require repairing entire block devices, but rather is likely to be closer to ceph, where only the in-use portion of a failed drive must be considered / must find a new home. I think raid and distributed storage systems (like backblaze or ceph) are more different than they are alike. > From the behavior of my RAID (which also uses Reed Solomon, doesn't it?) Maybe. mdadm raid5 doesn't, nor does mdadm raid1 or raid10. I think mdadm's raid6 does.

hinkley 6y ago

As I recall, Backblaze does what some other companies have also figured out: amortized maintenance costs across racks. Long ago we hit a point where power and cooling, not floor space, were the biggest bottlenecks for many data centers. Those numbers shift around, but during that era a bunch of people noticed that it's less labor intensive to fill an entire rack at a time. Put a few extra racks in as spares, and as hardware fails you power them off and power up new machines in the spare rack. When X servers in a rack are dead, you migrate everything else out of the rack, then strip it to the rails and put brand new hardware in it. You increase your server density twice in that case. Event-driven maintenance is disruptive to other work. It tends to have a ticking clock involved, so you likely assign it to more experienced people, which is 2 opportunity costs in one. Rack-at-a-time is batch processing. You can schedule it, you can assign multiple people to it, and speed is less of an issue. I might have the new guys stripping a decommed rack, use building a new one as training for four people and one teacher. You put a third of it together, I'll come tell you why the wiring is wrong, lather, rinse, repeat.