4 ms·

> In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done

> In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second. I want to know where you can find a drive that can write 12TB/sec of data! (In other words, you clearly missed half the problem. To add a new replacement drive, you have to be able to write to it the data from an original drive. Also RS code calculation is fast these days, but it ain’t that fast)

riobard 8y ago
Additionally, it implies that data needs to be spread across 10,000 drives, which is unrealistic anyway.
- brianwski 8y ago
  Disclaimer: I wrote the blog post. :-) > spread across 10,000 drives, which is unrealistic I claim it is also undesirable. Backblaze specifically made the conscious decision that the parts of any one single "large file" (these can be up to 10 TBytes each) are all stored within the same "vault". A vault is 20 computers in 20 separate racks. This allows a single vault to check the consistency and integrity of a large file periodically without communicating to other vaults in the datacenter. The vaults have been a really good unit of scaling for Backblaze. If the vaults can maintain their performance, then we know we can just stamp out more vaults because there is almost no communication between vaults.
- Dylan16807 8y ago
  As to the one second claim, I think that's a math error, because even 10k * 10MB is only 100GB. But spreading the data over 10k drives isn't unrealistic, it's a different architecture. Pick a different 20 drives for each file. Working it through: Assume 200 machines with 50 drives each. Each machine has to read 1TB, transmit it over the network, do a parity calculation, and write out ~50GB. With dual 10gbps ports the bottleneck is the network, and if we dedicate one on each machine to the rebuild we get a 15 minute clock. Not that having such a monolithic architecture is worth the complication and extra bugs.
londons_explore 8y ago
It's assuming that the newly recovered data is spread across free space of the other 10,000 drives, a few megabytes on each. It will also be recovered by reading the recovery data, which should also be approximately evenly spread across the 10,000 drives.
deleted 8y ago
[deleted]

riobard 8y ago

Additionally, it implies that data needs to be spread across 10,000 drives, which is unrealistic anyway.

brianwski 8y ago
Disclaimer: I wrote the blog post. :-) > spread across 10,000 drives, which is unrealistic I claim it is also undesirable. Backblaze specifically made the conscious decision that the parts of any one single "large file" (these can be up to 10 TBytes each) are all stored within the same "vault". A vault is 20 computers in 20 separate racks. This allows a single vault to check the consistency and integrity of a large file periodically without communicating to other vaults in the datacenter. The vaults have been a really good unit of scaling for Backblaze. If the vaults can maintain their performance, then we know we can just stamp out more vaults because there is almost no communication between vaults.
Dylan16807 8y ago
As to the one second claim, I think that's a math error, because even 10k * 10MB is only 100GB. But spreading the data over 10k drives isn't unrealistic, it's a different architecture. Pick a different 20 drives for each file. Working it through: Assume 200 machines with 50 drives each. Each machine has to read 1TB, transmit it over the network, do a parity calculation, and write out ~50GB. With dual 10gbps ports the bottleneck is the network, and if we dedicate one on each machine to the rebuild we get a 15 minute clock. Not that having such a monolithic architecture is worth the complication and extra bugs.

londons_explore 8y ago

It's assuming that the newly recovered data is spread across free space of the other 10,000 drives, a few megabytes on each. It will also be recovered by reading the recovery data, which should also be approximately evenly spread across the 10,000 drives.

deleted 8y ago

[deleted]