4 ms·
I'm not sure how anyone, in their disaster recovery plans, ever expect anything less than 100% failure of a data center. The scenario is: a tornado hits the da
by jrockway 8y ago
I'm not sure how anyone, in their disaster recovery plans, ever expect anything less than 100% failure of a data center. The scenario is: a tornado hits the data center.
Your power outage causing 3% of drives to fail is just a subset of that.
- viraptor 8y agoIf it's your data center, you can plan the location to prevent most of this. There are locations where the natural hazards can be completely managed. (No tornados, fires, tsunamis, earthquakes, ...) So the power outage is the most likely thing to happen.
- jrockway 8y agoI guess. I feel like "a tornado can never happen" is a lot like those lines in the logs like "error: can't happen". It can't happen, but it does.
- kijin 8y agoTheoretically, yes, it can happen. Realistically, the chance of a tornado taking out the Swedish datacenter built inside a former nuclear bunker under 100ft of granite bedrock is so small that it probably doesn't affect the number of 9's that you can claim.
- not_kurt_godel 8y agoSadly not even Swedish nuclear bunkers are safe from disaster: https://www.theguardian.com/environment/2017/may/19/arctic-stronghold-of-worlds-seeds-flooded-after-permafrost-melts https://www.theguardian.com/environment/2017/may/19/arctic-s... > Arctic stronghold of world’s seeds flooded after permafrost melts > It was designed as an impregnable deep-freeze to protect the world’s most precious seeds from any global disaster and ensure humanity’s food supply forever. But the Global Seed Vault, buried in a mountain deep inside the Arctic circle, has been breached after global warming produced extraordinary temperatures over the winter, sending meltwater gushing into the entrance tunnel. There are always unforeseen and unforeseeable risks associated with any location. You can mitigate them but you can't claim X number of 9s for a single physical datacenter.
- Dylan16807 8y agoNot a tornado, though. > There are always unforeseen and unforeseeable risks associated with any location. You can mitigate them but you can't claim X number of 9s for a single physical datacenter. What is X, here? I'm pretty sure I can claim 99% for a single datacenter.
- Hello71 8y agohell, I can probably do a solid 9% just visiting the local coffee shop twice a week.
- not_kurt_godel 8y agoWhat I meant is you can't necessarily amortize loss in the event of a localized catastrophe. Failure modes in a single location are by definition not always statistically independent. You could have 99.99999% durability for 20 years, but if something happens to the datacenter that causes total loss, you're SOL. Geographical redundancy vastly reduces the risk of freak occurrences that you can't predict.
- kijin 8y agoIf a datacenter boasts flawless durability for 19 years and loses everything in the 20th year, then they have an infinite number of 9's for the first 19 years and zero for the 20th year. It's all about probability. Nobody can promise 100%, but that doesn't mean that all those 9's are meaningless. They mean a lot for budgeting, and even more for insurance purposes -- which is exactly what we as a civilization have come up with as a way to amortize loss in the event of a local catastrophe. Your premiums are going to be much higher if you don't have enough 9's in a critical part of your money-making infrastructure. No one here is saying that you don't need geographical redundancy. First we need to figure out how many 9's we can realistically expect in order to determine how much redundancy makes financial sense.
- not_kurt_godel 8y ago> No one here is saying that you don't need geographical redundancy I mean, that's kind of what Backblaze is saying in the article, isn't it? They don't have geographical redundancy, yet there's not a single mention of that fact or the importance thereof in an entire article dedicated to teaching the unwashed masses about the limitations of mathematical theory in analyzing durability, even going so far as to say: > somewhere around the 8th nine we start moving from practical to purely academic... it’s far more likely that...Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers [emphasis my own] Seems like a pretty serious omission given their claimed authority as "the bottom line for data durability" and being "like all the other serious cloud providers" who do have geo redundancy, don't ya think?
- arminiusreturns 8y agoThe first Datacenter I ever brought up to T2 was tornado proof (former military base).
- stubish 8y agoIts just risk analysis. The cost involved in splitting things over multiple data centers vs. the chance of your single DC getting wiped out. Declaring bankruptcy if it happens will be the best business decision in many cases, or insurance if the risk is higher or owners can't afford the loss or have liability.