6 ms·
I think you're over-estimating the work taken to replace a drive -- their storage is redundant using a custom Reed-Solomon error correction setup[1,2] (meaning
by cyphar 7y ago
I think you're over-estimating the work taken to replace a drive -- their storage is redundant using a custom Reed-Solomon error correction setup[1,2] (meaning no downtime while replacing a drive so none of the surrounding work you mentioned is necessary), and at the scale they're operating on they have staff just replacing drives (they are averaging 5 failures a day and from memory they had higher failure rates a few years ago) or bringing up new storage pods and other maintenance work.
While they do have multi-failure redundancy (17 data, 3 parity), it wouldn't be a good idea to push the limits regularly because that means you're risking the loss of actual customer data. And adding more parity to give you extra wiggle-room would result in both storage efficiency losses and performance losses. It probably wouldn't be advisable to expand the "stripe" size to cover an entire vault (to increase the independent disk failure tolerance without reducing the data-to-parity ratio) because you'd kill performance (the matrix multiplications necessary to do Reed-Solomon have polynomial complexity).
[1]: https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect...
[2]: https://www.backblaze.com/blog/reed-solomon/ https://www.backblaze.com/blog/reed-solomon/
- linsomniac 7y agoWhen I said "just let failed drives sit", I didn't imagine loss of redundancy, I imagined that they would re-balance it to other drives.
- cyphar 7y agoAh, okay. Unless you have hot spares already in the storage pod (which they might do), I don't think it'd be trivial -- or even a good idea -- to mess with the data/parity shards after the fact.