7 ms·
This was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line: Because at
by garettmd 8y ago
This was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line:
Because at these probability levels, it’s far more likely that:
- An armed conflict takes out data center(s).
- Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers.
- There’s a prolonged billing problem and your account data is deleted.
The point that once you get to a certain point of durability (at least as far as hardware/software is concerned) you're chasing diminishing returns for any improvement. But the risks that are still there (and have been big issues for people lately) are billing issues. I think it's an important point that the operational procedures (even in non-technical areas like billing and support) are critical factors in data "durability"
- schoen 8y agoI wonder if there's a general term in engineering for the case where a particular risk has been reduced well below the likelihood of more serious but exotic risks. I've heard about this most in cryptography where we can sometimes say that the risk of, say, a system failing due to an unexpected hash collision is drastically less than the risk of the system failing due to cosmic radiation or various natural disasters. At that point it doesn't seem important or worthwhile to consider this risk, because it's dwarfed by the others. This seems like a form of the same argument, and I wonder where else this arises and how people describe it.
- matthewmcg 8y agoI think in other contexts, you might say that these signals are "below the noise floor."
- entropicdrifter 8y agoAs an audio engineer (as well as a dev), I definitely agree with this use of that phrase.
- matt4077 8y agoIt’s not a perfect match, but Rumsfeldian “unknown unknowns” come to mind. Specifically: every X-nines durability design will be compromised by some failure mode you didn’t think of.
- macintux 8y agoI tend to assume pessimistically that the durability design will itself cause a problem. Redundant switches to survive a hardware failure, e.g., strikes me as inviting trouble.
- jzwinck 8y agoIndeed. I once had critical systems routed via a large redundant Cisco switch which claimed to be 1+1. Turns out there was a single "supervisor" component which failed (after just a year or two) and made the pair of switches useless. Apparently the designer worked in a team where nobody does anything when the boss is out.
- schoen 8y agoI don't think unknown unknowns are what I'm thinking of here. In this case the argument involves a very specific risk, and sometimes a very specific lower bound for its probability. For example, in the hash collision case the argument says that it's not worth worrying about the (known) probability of a software error due to an unexpected hash collision because it's dominated by the (known) probability of a comparable error due to cosmic radiation. (The former probability can be calculated using the birthday paradox formula, and the latter has been characterized experimentally in different kinds of semiconductor chips.) This kind of argument doesn't rely on the idea that there are other risks that we can't identify or quantify. It's about comparing two failure modes that we did think of, in order to argue that one of them is acceptable or at least not worth further attempts to mitigate.
- fanpuns 8y agoAlthough Rumsfeld often gets credit for this statement, it has been around for a long time before him https://en.wikipedia.org/wiki/There_are_known_knowns https://en.wikipedia.org/wiki/There_are_known_knowns
- pixl97 8y agohttps://en.wikipedia.org/wiki/Black_swan_theory https://en.wikipedia.org/wiki/Black_swan_theory
- jorangreef 8y agohttps://en.wikipedia.org/wiki/Information_theory https://en.wikipedia.org/wiki/Information_theory
- klodolph 8y agoI've posted the math here before but if we assume that an asteroid hits the earth every 65 million years and wipes out the dominant life forms, then this fact alone puts your yearly durability at a maximum of ~8 nines. The point about billing is better, though. My other concern is that a software bug, operator error, or malicious operator deletes your data.
- GauntletWizard 8y agoThe event that wiped out the Dinosaurs was 65 million years ago, but the Mesozoic era lasted for ~200 million years. Your point stands, though.
- klodolph 8y agoThe question of how many data points to use is a subtle one, though. I can say that I picked one data point because I was lazy and doing back of the envelope math, which is reasonable because we can be somewhat assured that I didn't choose a number of data points that was convenient for the hypothesis. But if you're choosing two data points, my question is... why two? If you are choosing whether or not to reply based on whether or not the second data point fits with the first, then you're introducing selection bias. The chance that the second data point disagrees with the first by at least as much as the 200 My interval disagrees with the 1/65 My rate is equal to 1-(exp(-65/200)-exp(200/65)) = 0.32, which is not especially high.
- tjoff 8y agoThat's why no sane entity would use one earth. Use two and you can quickly recover.
- klodolph 8y agoOur goal is N+2, that way when one Earth is down for planned maintenance, you can endure unplanned loss of a second Earth. N, of course, is always 1.
- metalliqaz 8y ago
- TheAceOfHearts 8y agoFor a consumer the cheapest and easiest way to backup important documents or files is to encrypt it and store it across multiple storage providers, e.g. Dropbox and Google Drive. They usually give you a reasonable amount of free storage, and it's unlikely all accounts would be terminated or locked at the same time. And of course, you should always have your local backups as well.
- e12e 8y agoAssume you want access to files over the next 20 years. What are the odds Google will have bought Dropbox in that time; and what are the odds an automated system monitor at Google ad words will have disabled your Google account in that time span? Replace with Amazon and/or crashplan as appropriate..
- scarface74 8y agoBackblaze suggests a “3-2-1” backup strategy. You should always have at least one backup on site. If a remote backup becomes inaccessible, you could move over to another remote access provider.