4 ms·
I've very disappointed their recovery time is 6 days! Recovery workload should be spread across the whole cluster, so that the recovered data gets distributed
by londons_explore 8y ago
I've very disappointed their recovery time is 6 days!
Recovery workload should be spread across the whole cluster, so that the recovered data gets distributed evenly. In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second. Maybe 10 seconds with some sluggish tail machines.
Why do you need it done in under a second? While the data is down one replica, it is at dramatically higher risk. Also, drive failures can be dramatically accelerated, for example in the case of a bad software release erasing data - you need to be able to move data faster than bad software gets released. And releasing software at a rate of one machine per second still means a release takes 3 hours!
- ChuckMcM 8y agoBe careful here, it isn't 6 days until data is recovered it is 6 days until it is fully protected again, there is a big difference. During the 6 days the data would be available it just might have to be reconstructed on the fly by the error correcting rather than read directly. In most systems we assume that "primary traffic" (read/write stuff) is prioritized over "rebuild traffic" which is recovering lost shards. So when you specify these things it is best to specify "how long to rebuild a shard while the array is providing storage services at its maximum specified rate." This assures the customer that if they have a 24/7/365 non-stop traffic pattern their data will still stay protected in the face of drive failures.
- brianwski 8y agoDisclaimer: I'm the author of the blog post. :-) > During the 6 days the data would be available it just might have to be reconstructed on the fly by the error correcting rather than read directly. Correct. More specifically, the FIRST time the data is accessed in any 24 hour period it must ALWAYS be reconstructed from the Reed-Solomon encoded parts on 17 other drives on 17 other machines. Any 17 is fine, so it's totally fine if 1 or 2 drives are not available. Once reconstructed it is stored in a set of front end cache computers that have fast SSDs for this purpose. The second time the same file is accessed in a 24 hour period, it will be fetched out of the SSD cache layer so it won't even hit the spinning drives and won't care if all 20 drives are offline. > "primary traffic" (read/write stuff) is prioritized over "rebuild traffic" Yes. Backblaze balances between the two if only one drive has failed, but as a tome (20 drive group spread across 20 computers) becomes more badly degraded Backblaze begins favoring the rebuild. When two drives have failed out of 20, Backblaze stops allowing any writes to that tome because more writes will tend to fail yet another drive. Fewer writes offloads the tome. But we still allow reads. At Backblaze, we have never been 3 drives degraded out of 20 (knock on wood), but if this ever occurs the 20 drive tome is now running without parity -> so in that case we even stop allowing reads AT ALL until we are returned to at least 1 drive of fully redundant parity.
- jsjohnst 8y ago> In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second. I want to know where you can find a drive that can write 12TB/sec of data! (In other words, you clearly missed half the problem. To add a new replacement drive, you have to be able to write to it the data from an original drive. Also RS code calculation is fast these days, but it ain’t that fast)
- riobard 8y agoAdditionally, it implies that data needs to be spread across 10,000 drives, which is unrealistic anyway.
- brianwski 8y agoDisclaimer: I wrote the blog post. :-) > spread across 10,000 drives, which is unrealistic I claim it is also undesirable. Backblaze specifically made the conscious decision that the parts of any one single "large file" (these can be up to 10 TBytes each) are all stored within the same "vault". A vault is 20 computers in 20 separate racks. This allows a single vault to check the consistency and integrity of a large file periodically without communicating to other vaults in the datacenter. The vaults have been a really good unit of scaling for Backblaze. If the vaults can maintain their performance, then we know we can just stamp out more vaults because there is almost no communication between vaults.
- Dylan16807 8y agoAs to the one second claim, I think that's a math error, because even 10k * 10MB is only 100GB. But spreading the data over 10k drives isn't unrealistic, it's a different architecture. Pick a different 20 drives for each file. Working it through: Assume 200 machines with 50 drives each. Each machine has to read 1TB, transmit it over the network, do a parity calculation, and write out ~50GB. With dual 10gbps ports the bottleneck is the network, and if we dedicate one on each machine to the rebuild we get a 15 minute clock. Not that having such a monolithic architecture is worth the complication and extra bugs.
- 8y ago
- TheDong 8y ago> you need to be able to move data faster than bad software gets released. And releasing software at a rate of one machine per second still means a release takes 3 hours! Since others have already demonstrated why the remainder of your comment is overly simplistic, I'll tackle this bit. Generally, software releases are not rolled out at a constant rate to all machines. A typical thing to do is to release it to staging, then to a "canary" subset of machines (e.g. to 1% or 5% of the machines).. Once all seems well there (e.g. metrics are clean and the canaries have handled X writes, reads, and simulated drive failures), it can be rolled out to a larger subset, and eventually to all machines. In that way, the release can take whatever total amount of time is desired while still catching any such bugs fairly reliably. Ideally, at backlblaze they could ensure that their canary instances are "data-redundancy aware" such that even if the 5% they roll to for the canary test all explode, data is still safe. Regardless, any talk of "recovering data faster than software releases" is completely silly and totally misses the reality of how releases are done, how recovery is done, and what sort of bugs might happen. The math based on faulty assumptions about rate is also pointless.