7 ms·
Relevant amusing bit from the Amazon FAQ: "S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to
by idlewords 11y ago
Relevant amusing bit from the Amazon FAQ: "S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years."
I think my favorite part of that is "on average", as if you will be making repeated ten-million-year trials of this effectively brand new technology.
The point is that once you get into several nines of reliability, really rare events that are impossible to model start to dominate your risk budget.
- hyperpape 11y ago"just 0.000001% of disk space was permanently affected." So Google just exhausted their 11 9's for centuries to come.
- MichaelGG 11y agoWhich would be embarrassing if this had happened to a product offering such an SLA. Instead, this is just zone-local HDD storage. It will only affect customers that decided to run a system in a single site, making no snapshots or using cross-region replication, etc. Which may be quite a few seeing as there seems to be this misunderstanding that VMs "in the cloud" have physics-defying redundancy attributes. (Not saying you have this idea, just that multiple clients have expressed surprise when their single-VM setup goes down because "isn't this what the cloud is for?".)
- dmytton 11y agoThis is S3 which isn't comparable to Google's persistent disks. S3 is equivalent to Google Cloud Storage which has "99.999999999%" durability as per https://cloud.google.com/storage/ https://cloud.google.com/storage/ To accurately compare them you'd need to look at AWS EBS: "Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% - 0.2%, where failure refers to a complete or partial loss of the volume, depending on the size and performance of the volume."
- idlewords 11y agoThe point isn't to compare competing cloud services, but draw attention to the fantastical nature of these durability promises.
- azurezyq 11y agoI don't think so, PD and GCS are fundamentally different products. PD is like the local disk, and you cannot assume it last forever. while GCS is durable storage and it is safer. " This outage is wholly Google's responsibility. However, we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data. " In a cloud based infra, one still needs to know the difference. And for the ultra-low numbers, they are from mathematics I think. (# replicas, possibly cross DC/geo)
- emmett 11y agoMany people have far more than 10k objects though. Based on that math, if you stored 10 billion objects in S3 (certainly well within the realm of possibility), you'd lose an object on average every 10 years, which is a length of time one can think about. Your main point (that making high reliability systems more conventionally reliable is building your high wall yet higher still) is definitely valid. But the lossage rate is actually a meaningful number given the extremely large number of objects stored in S3.
- idlewords 11y agoI completely agree with you. Where they are intentionally misleading is in pretending that keeping one object for a zillion years is the same as keeping a zillion objects for one year.
- devindotcom 11y agoKeeping one object for a zillion years sounds like an interesting project, now that you bring it up. Our data integrity is such that it will survive universal cataclysm and the reduction of all data to a single point of infinite energy.[1] [1]guarantee applies to a single bit of data[2] [2]bit value must be 1
- toomuchtodo 11y agoAll I could think of was Hitchhiker's Guide To The Galaxy.
- forrestthewoods 11y agohttp://www.10000yearclock.net/learnmore.html http://www.10000yearclock.net/learnmore.html
- tsunamifury 11y agoFunny enough, its funded by Bezos
- cjensen 11y agoGlacier is also 99.999999999%. But those nines are only true if Amazon's software is bug-free. I've lost data on Glacier on my home account. They personally called me to apologize.
- Tloewald 11y agoSo empirically speaking their reliability is far, far worse than stated just based on that one data point.
- onethumb 11y agoHow so? We'd need to know both how much data cjensen lost and how much data Glacier has in it to tell, neither of which I know from this conversation.
- cjensen 11y agoMy recollection: around 20GB... however, they clearly were willing to try to get some of it back if that was helpful. It was just a redundant offsite backup for me, so there was no need.
- EduardoBautista 11y agoWould have been interesting to see if they had been able to.
- pdabbadabba 11y agoWho knows. Maybe his is the last data Glacier will lose in the next [suitably large number] years. That's bad luck, buddy!
- megablast 11y agoI too choose to extrapolate all my losses over the next 100 billion years.
- 11y ago
- Zenst 11y agoIndeed "on average" covers the entire period and from initial local-node intake of the data it has to then be propergated to another site to increase integrity probability. So if it takes even 5 seconds to propergate to several sites at different locations then the durability scales from when the clock starts. Heck once in every 10,000,000 years, the odds of winning the USA lottery are around one in 175,000,000 and yet people partake and somebody wins. So it is always worth looking at odds from another perspective and accept it is either impossible, probably or possible and as we know the return from investment in durability is logarithmic in cost going up and return getting smaller. Still one of the biggest issues in any datacentre is not just the power, but also the quality of the power and even the slightest noise could and does increase the odds of some electronics going wrong. UPS's (though been many years) will put out a square wave in modulation and the ideal is a perfect sine-wave. It is truly educational to see the quality of many power sources on an oscilloscope. Also equipment on the same circuit can also induce noise, so can vary rack to rack.
- zkhalique 11y agoThe irony is that, by that time, Amazon will almost certainly not be around. So doing it in terms of time is actually a different calculation, and this is based on wrong assumptions :)
- pjc50 11y agoAt this point, correlation matters. In the unlikely event that I lose at least 1 object, what is the conditional probability of losing another? (Forgetting about correlation was a big part of the MBS and LTCM financial failures)
- Tloewald 11y agoAnd Fukushima. Assumptions that variables are independent are often very mistaken.
- dockd 11y agoOr the bigger problem: assuming your future reliability looks just like your past. For example, say they have a problem with year 2038. Everything works great until everything doesn't work at all.
- mikecb 11y agoThey had a bigger problem: modeling with the wrong distribution.
- Tloewald 11y agoAnother problem is that any errors in assumptions or omissions made when calculating the odds will be enormously magnified. I wouldn't trust any of these figures unless they have ongoing efforts to test them empirically. E.g. create distributed databases of 100 trillion objects, mess with them in various ways, and perform correctness checks on them.
- mikeash 11y agoDo they spell out what assumptions they're working under? Because, for example, I'm pretty sure that the odds of a civilization-ending asteroid or comet strike in the next year, while quite low, are higher than what's implied by 99.999999999% durability on trillions of objects."
- mehrdada 11y agoWell, the guarantee needs to be reasonable only as far as you and the courts are alive to sue them. (Corollary: you only need to sustain DDOS attacks when the rest of the internet infrastructure can sustain the traffic).
- Dylan16807 11y agoWhy do you specify trillions of objects? They practically promise data loss at that scale. Also, the typical civilization-ending comet or asteroid is going to destroy 0 or 1 geographically-diverse data centers. Remember that this is a retention promise, not an uptime promise.
- mikeash 11y agoJust because it's how many objects they store. But now that you mention it, it has no relevance to the probability of data loss due to global catastrophe, since it cancels out. If you're proposed reliability is data loss of one in 100 billion per year, that assumes the risk of e.g. global thermonuclear war is no more than that amount, or you're ignoring it for the purposes of the calculation. Which is why I wonder about their assumptions. As for a comet destroying at most one data center, if it really ends civilization then the data will probably be lost before too much longer even if the facility physically survives.
- Taek 11y agoThat's one of the reasons that I'm excited about decentralized storage. Data stored on dozens or hundreds of nodes spread across multiple countries and jurisdictions is much more robust to things like earthquakes, storms, government intervention, and companies going out of business or changing their profitability model. It has a much stronger defence against black swan events. When you are storing many billions of files consuming many millions of disks, you become vulnerable to black swan events, simply because you are rolling the dice so many times.
- leni536 11y agoI'm quite hyped about ipfs. I think they can deliver in few years.
- jsprogrammer 11y agoIs that durability rate per-... user? project? s3-wide? Important detail left out. If there are 10,000,000 projects with 10,000 objects, then a system-wide durability of 99.999999999% would expect to drop 1 object per year.
- cos2pi 11y agoI always think of Feynman's minority report [0] on the Challenger Disaster when these types of risk assessments pop up. [0] http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/roger...
- Dylan16807 11y agoYou are making repeated trials, they just overlap.
- yeukhon 11y agoWell the expectation is different for different players. If Facebook lose 1 photo for 1 customer out of 10M customers, FB wouldn't care. The user may just assume a glitch. For a small business, losing 1 photo for 1 customer out of 10,000 customers can be a big deal, but nonetheless not the worst case. Offer sincere apology, do extra data backup/replication if those data are extremely valuable to customer, and roll out a new service.
- wodenokoto 11y agoYou're thinking of average as a frequentist, and Amazon are thinking as Bayesian. Ever thought about how we can talk about the chance of rain tomorrow? Or the risc that a comet may strike the earth within the next million years?