7 ms·
This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives makin
by londons_explore 8y ago
This analysis is simplistic.
Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc.
When calculating your chance of failure, you have to include that, or your result is bogus.
Eg. Model A of drive has a failure rate of 1% per year, but when failed the symptom is failure of the drive to spin up from cold, however if already spinning it will keep working as normal.
3 years later, the datacenter goes down due to a grid power outage and a dispute with diesel suppliers so the generators go down. It's a controlled shutdown, so you believe no data is lost.
2 days later when grid power is back on, you boot everything back up, only to find out that 3% of drives have failed.
Not a problem. Our 17 out of 20 redundancy can recover up to 15% failure!
However, each customers data is split into files around 8MB, which are in turn split into the 20 redundancy chunks. Each customer stores say 1TB with you. That means each customer has ~100k files.
The chances that you only have 16 good drives for a file is about (0.97^16 * 0.03^4)2019*18 = 0.3%
Yet your customer has 100k files! The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-(
- ChuckMcM 8y agoI was going to post essentially the same thing, so here is an upvote :-) While I always find storage analysis interesting (I spent 5 years at NetApp where it was sort of a religion :-)) some of the assumptions that Brian was tossing out are not good ones to make. (like the lack of correlation, or that Drive Savers will exist as a company 10 years from now). Still it does help you to understand the they take data availability seriously which is the underlying message.
- brianwski 8y agoDisclaimer: I'm the author of the blog post. :-) > some of the assumptions that Brian was tossing out are not good ones to make. We COMPLETELY welcome other analysis and listing other assumptions. Internally, we argued endlessly about why this or that wasn't totally accurate, and finally decided to publish the math WITH all of our assumptions exposed so you could be the judge. If Amazon wants to publish their assumptions for S3 for comparison, we're all ears. > that Drive Savers will exist as a company 10 years from now Absolutely true, this calculation is only good RIGHT NOW. For example, one of the things that came up internally was "well, when drives get more dense the rebuild time rises, so this calculation will no longer be accurate in two years". But at the same time, we have some additional tricks and optimizations to make which we have not done yet to cut the 6 day average drive rebuild time down to 3 days. Also, drives last us about 5 years, so your data will be migrated to totally new drives 5 years from now. Those drives will absolutely have a different drive failure rate (maybe higher, maybe lower) so the calculation will no longer have the same result 5 years from now.
- ChuckMcM 8y agoHey Brian! For the record I love that you are transparent about your assumptions, it is really helpful. I have been in very very similar debates, both at NetApp and at Google of all places. Peter Corbett, the guy who invented the dual parity scheme that NetApp uses, wrote a similar analysis as well for Fast '04 [1]. As someone who likes to geek out on failure proof systems and perfectly secure systems, neither of which are attainable but can be asymptotically approached, I think you are seeing the "there is always one level deeper" kinds of discussions. Personally I think of them as endorsements because if the exceptions get too extreme (say 'what if an asteroid hits?') then you know you've got all the bases covered. Its only a problem if the person analyzing the analysis finds something that you really did not even consider. Then it opens up an opportunity to look at the problem a whole new way. [1] https://www.usenix.org/legacy/publications/library/proceedings/fast04/tech/corbett/corbett_html/ https://www.usenix.org/legacy/publications/library/proceedin...
- brianwski 8y agoGood link! I've both forwarded it on, and will study it when I have some time later.
- jorangreef 8y agoHi Brian, thanks for open-sourcing Backblaze's JavaReedSolomon, which is really well-written. A few months ago I ran into an issue with Reed Solomon coding throughput not saturating the write throughput of 16 drives, and wrote a new Reed Solomon module based on Cauchy matrices: https://github.com/ronomon/reed-solomon https://github.com/ronomon/reed-solomon The Cauchy matrices remove the need for a table lookup to do the Galois multiply, replacing it with pure XOR. Together with other optimizations, this gives nearly 3x-5x more coding throughput for the same (17,3) parameters, assuming you're still using your open-sourced JavaReedSolomon in production. I don't know if Reed Solomon coding throughput is a factor in your rebuild times?
- kdkeyser 8y ago
- thefifthsetpin 8y agoI think that your point is what they were trying to address by saying that anything beyond eight nines is impractical. Their examples of correlated failures included earthquakes and floods which might have huge reach rather than just impacting a single rack, but I think it's the same general idea.
- Johnny555 8y agoThat could be a power surge taking out a whole rack This failure mode, at least, is already accounted for by sharding data across cabinets: Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet. https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect... However, they don't seem to offer multi-datacenter (or multi-region) redundancy so are still susceptible to a datacenter fire/failure. In comparison, AWS S3 distributes data across 3 AZ's (datacenters), and you can further replicate across regions if you choose. Though you pay for that added redundancy in 3 - 4X higher cost.
- gm-conspiracy 8y agoI thought AZs were in the same physical location, just separate networks, no?
- coderholic 8y agoNo, they're different physical locations. From https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Conce...: > Each AWS Region has multiple, isolated locations known as Availability Zones. And from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions-availability-zones https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-re...: > Each region is completely independent. Each Availability Zone is isolated, but the Availability Zones in a region are connected through low-latency links.
- jsjohnst 8y agoNope. They aren’t even in the same city. For example, each of the AZs in us-west-2 are separated by about 50-60 miles. This doesn’t list all the locations, but is a good map to get an idea: https://www.google.com/maps/d/u/0/viewer?ll=50.95842700000002%2C4.130858999999987&spn=26.684562%2C56.25&t=m&msa=0&z=4&source=embed&ie=UTF8&mid=1m6v8XPxwp0Dx4THiakRKFkZwIGE https://www.google.com/maps/d/u/0/viewer?ll=50.9584270000000...
- ChrisLomont 8y ago>The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-( You just made the same mistake you're criticizing. You assumed the 100K files were uniformly and independently spread. They're also likely clustered, and perhaps not even at the same data center. Given the variety of drives Backblaze uses, the drives are also not likely to all be the same model, so your failure method is also unlikely. You also only looked at the case there are exactly 16 good drives. The proper failure estimate is 1 - (odds of 17 good + odds of 18 + odds of 19 + odds of 20). I'm not sure where you got the 20 x 19 x 18 part either. Did you mean 30 choose 16 or something like that? Using the proper 1-... method I get 0.00267, not 0.03.
- londons_explore 8y ago>You just made the same mistake you're criticizing Yes. Even if you take it into account, the vast majority of customers will see data loss, assuming random (but not even) shard distribution. I rounded and approximated to avoid explaining too many probability rules... As you can see, it gives the same result at the end.
- ChrisLomont 8y ago>it gives the same result at the end No, it doesn't. You picked 3% of all drives failing out of the blue. Your next estimate was an order of magnitude too high. Your last assumption of p^# drives is not reasonable. The proof is in reality. Backblaze has run over a decade, with all sorts of hardware failures, server configs, running many drive models through their lifetime, across manufacturers, across technologies, across multiple datacenters, and had not seen the level of failures you claim they will. So I suspect their method of estimating is more accurate than yours. So far it matches reality much better.
- httpz 8y agoAlso, there are only 3 hard drive manufacturers left. If one of them have a bug that affects across their product line, that can take out 1/3 of all hard drives.
- kolpa 8y agoIt's vanishingly unlikely that a bug would affect all their drives (across all recent models, and only after burn-in) simultaneously, unless the drives are managed by a remote server with a SPOF.
- hinkley 8y agoWe're talking about 6+ 9's here. Putting an order of magnitude on your definition of 'vanishingly small' seems compulsory when we're already so far right of the decimal point. How many years between incidents are you talking, and when was the last time a manufacturer had a multigenerational bug? (and is quality improving, or decreasing? That is, are we more or less likely to see failures in the next 10 years than we did in the last 10?)
- voidmain 8y agoThe Crucial M4 SSD had a firmware bug [1] that caused it to fail repeatedly starting after 5,184 power on hours. Disk failures could also be triggered by datacenter environmental factors shared among many drives like temperature or noise. [1] https://www.storagereview.com/node/2676 https://www.storagereview.com/node/2676
- deleted 8y ago[deleted]
- sitkack 8y agoYou never experienced Death Star.
- jrockway 8y agoI'm not sure how anyone, in their disaster recovery plans, ever expect anything less than 100% failure of a data center. The scenario is: a tornado hits the data center. Your power outage causing 3% of drives to fail is just a subset of that.
- viraptor 8y agoIf it's your data center, you can plan the location to prevent most of this. There are locations where the natural hazards can be completely managed. (No tornados, fires, tsunamis, earthquakes, ...) So the power outage is the most likely thing to happen.
- jrockway 8y agoI guess. I feel like "a tornado can never happen" is a lot like those lines in the logs like "error: can't happen". It can't happen, but it does.
- kijin 8y agoTheoretically, yes, it can happen. Realistically, the chance of a tornado taking out the Swedish datacenter built inside a former nuclear bunker under 100ft of granite bedrock is so small that it probably doesn't affect the number of 9's that you can claim.
- not_kurt_godel 8y agoSadly not even Swedish nuclear bunkers are safe from disaster: https://www.theguardian.com/environment/2017/may/19/arctic-stronghold-of-worlds-seeds-flooded-after-permafrost-melts https://www.theguardian.com/environment/2017/may/19/arctic-s... > Arctic stronghold of world’s seeds flooded after permafrost melts > It was designed as an impregnable deep-freeze to protect the world’s most precious seeds from any global disaster and ensure humanity’s food supply forever. But the Global Seed Vault, buried in a mountain deep inside the Arctic circle, has been breached after global warming produced extraordinary temperatures over the winter, sending meltwater gushing into the entrance tunnel. There are always unforeseen and unforeseeable risks associated with any location. You can mitigate them but you can't claim X number of 9s for a single physical datacenter.
- Confusion 8y agoWhich means every customer suffers data loss They suffer partial backup loss. The customer only suffers data loss if they lost their 'master' copy of the data as well during the outage. Iff they don't have a secondary backup solution.
- mmt 8y agoAre they positioning their "B2" product solely as a backup solution, rather than something closer to an S3 competitor? If it's the latter, "data loss" seems an appropriate characterization.
- Confusion 8y agoYou are right; I hadn't realized BackBlaze was also offering cloud storage and thought this was about their backup product.
- cjhanks 8y agoNice analysis.
- mmt 8y ago> Correlated failures are common in drives. This is why, when I was building DIY arrays for startups (around the same time Backblaze published their first pod design [1]), I went through the extra effort of sourcing disks from as many different vendors as possible. Although it was somewhat more time consuming and limited how good a price I could get and how fast the delivery could be, it meant that, for any given disk drive size, it meant I could build an array as large as 12 where no 2 drives were identical in model and manufacturing batch [2]. Of course, it's still a vanishingly rare risk, and "nobody" cares about hardware any more. It does help to remember, at least once in a while, that, on some level, cloud computing really is "someone else's servers" and to hope that someone else still maintains this expertise. [1] though I used SuperMicro SAS expander backplane chasses for performance reasons [2] and firmware from the factory, although this is somewhat irrelevant, as one can explicitly load specific firmware versions, and, IIRC, the advantages of consistent firmware across drives, behind a hardware RAID card, outweighed the disadvantages
- jquast 8y ago> I went through the extra effort of sourcing disks from as many different vendors as possible. This is very good advice! If you already built your array, consider advice: "replace a bad disk with a different brand, whenever possible". Over time, you naturally migrate away from the bad vendors/models/batches. After following this practice, it seems ridiculous to me now to keep replacing the same bad disks with the same vendor+model.
- mmt 8y agoAlthough I wouldn't go so far as to insist on switching brands (especially since, as another commenter pointed out, there has been so much consolidation, there remain only 3), I agree that replacing with at least a different model, or, failing that, a different batch, is a best practice for an already-built homogenous array. Some of this can also be achieved ahead of time if one has multiple arrays with hot spares, by shuffling hot spares around, assuming there's some model diversity between the arrays but not within them. I doubt I'll ever again have the luxury of being able to perform this kind of engineering, however. Even a minor increase in cost or cognitive/procedure complexity or a decrease in convenience just serves to encourage a "let's move everything to the cloud" reaction, so I keep my mouth shut.
- hinkley 8y agoYou could also have a bad batch of drives, causing a bunch of failures to happen over the period of a couple of months. Sources of failure don't take a number. Like in your example, they can and will overlap, which is why when we try to design bullet proof systems we aren't satisfied until every bad even requires a number of separate things to go wrong all at once. But if you have faulty power or bad hardware chipping away at your equipment, your depth of resiliency is degraded until the issue is corrected.
- mrb 8y ago«The chances that you only have 16 good drives for a file is about (0.97^16 ∗ 0.03^4)∗20∗19∗18 = 0.3% Yet your customer has 100k files! The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-(» Your math is completely wrong. In reality 92% of customers suffer no data loss. The chance of a file having 4 failed drives (16 good drives) is: .03^4 = 0.00008100% The chance of a file being irrecoverable is the chance of having 4 or more failed drives: .03^4 + .03^5 + .03^6 + ... = 0.00008351% The chance of a file being recoverable is: 1 - .03^4 - .03^5 - .03^6 - ... = 99.99992151% The chance of a customer's 100k files all being recoverable is: (1 - .03^4 - .03^5 - .03^6 - ...)^100000 = 92.0% Therefore only 8% of customers encounter one or more 8MB file that is irrecoverable.
- barbegal 8y agoThe original maths is correct. You have neglected the fact that the 4 failed drives can be any of the 20. Your calculations are for 4 specific drives failing.
- mrb 8y agoOh you are right. The OP still has a slight error though. C(20,4) = 4845 possible combinations of failing drives (4 out of 20.) Therefore the chance a file is irrecoverable is: .97^16 × .03^4 × 4845 = 0.241% (not 0.3%) But the OP's conclusion is largely correct: every customer will have some irrecoverable files.
- majewsky 8y agoI think you're not considering that failed drives will be replaced and the data on them reconstructed from the other shards. This failure mode requires 4 of 20 drives to fail in such a short amount of time that reconstruction cannot be completed.
- mrb 8y agoYes, but this was the OP's scenario: what happens when 3% of drives all fail at the same time when the DC is powered back on. Edit: actually the math is still wrong. The chance any 4 out of 20 drive is failing is: .03^4 × C(20,4) = .03^4 × 4845 = 0.392% — There is no need to multiply by .97^16 as the status of the other 16 drives is irrelevant.
- KindOne 8y agoYear 2038 seems a bit simple to test. Can you just set the BIOS date to January 19 2038 03:00:00 UTC and wait 15 minutes to see what happens?
- jdcarter 8y agoI recently looked into the research on this topic, and it absolutely agrees with you--failures are not independent. Papers of note: https://www.usenix.org/legacy/event/fast08/tech/full_papers/jiang/jiang.pdf https://www.usenix.org/legacy/event/fast08/tech/full_papers/... http://www.cs.toronto.edu/~bianca/papers/fast07.pdf http://www.cs.toronto.edu/~bianca/papers/fast07.pdf Take note of section 5 in both papers for their statistical models and how they match with real-world data.
- hinkley 8y agoThere was a statistics class in college that kicked my ass. I never quite understood how to determine if two variables were independent or dependent. You get a vastly different answer if you get it wrong. I run into people all the time that seem to have the same problem, to the point that it makes me wary of any software developer putting forth numbers that seem fantastical. My gut reaction is that if Backblaze wants to keep reporting their disaster preparedness numbers that they need the assistance of an actuary to calculate them.