3 ms·

S3's number is just marketing. It's going to be that durable until a black swan event wipes out 0.1% of all files. I imagine they keep multiple copies of ever

by jewel 12y ago

S3's number is just marketing. It's going to be that durable until a black swan event wipes out 0.1% of all files. I imagine they keep multiple copies of every file, spread across multiple data centers. It's durable against hardware failure of all varieties, but it's still vulnerable to a software bug. It's also possibly vulnerable to certain types of natural disasters. IIRC backblaze employs a similar strategy. They have the advantage that your home computer is still storing a copy, so even if they have a minor catastrophe they can recover by simply having their client re-upload those files.

ghshephard 12y ago
I think the S3 number is a floor, not a ceiling. That is, you should expect to be losing that much of data, but, given an unplanned event, obviously a great deal more data can be lost.
mikeash 12y ago
Human disasters too. I'd wager the odds of a catastrophic massive nuclear war are higher than one in ten million per year. Not that I expect Amazon (or much of anyone) to protect against that, nor would the integrity of my S3 data be a priority afterwards, but it makes the claim kind of absurd. Edit: reading more closely, their durability number is per object. It's one object lost per 100 billion objects per year. The 10 million number comes from a hypothetical situation where you store 10,000 objects. I don't know how many objects S3 stores altogether, but if we say it's a billion (presumably a vast underestimate) then that would imply that the probability of losing all objects in the system over the coming year is one in 100 quadrillion. I don't think this planet is that safe.
- res0nat0r 12y ago
  Not sure why it is absurd due to the redundancy they've implemented + data checksumming. Plus AWS hit two trillion objects a bit over a year ago. http://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ http://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objec...
  - mikeash 12y ago
    As great as S3 is, it's still confined to Earth. Seems to me that the odds of a planet-wide disaster taking out all Amazon infrastructure (as well as certain less important things like human civilization) are higher than the odds they're giving of a catastrophic S3 failure.
    - res0nat0r 12y ago
      Sure. The AFR of your Western Digital desktop HD doesn't factor in and lower it's reliability slightly because of the slim chance that a 747 is going to crash into your house and destroy your home. Barring unforeseen acts of God, the 9's listed above apply and you just have to personally weigh if you think the risk of S3 losing multiple datacenters is high enough for you to risk storing your data there.
      - mikeash 12y ago
        Amazon pretty explicitly includes unforeseen catastrophic events in their durability estimate. "In addition, Amazon S3 is designed to sustain the concurrent loss of data in two facilities." I sure hope the loss of two facilities doesn't fall into the "foreseen" category!
        res0nat0r 12y ago
        Sure, it says they account for that right there in their FAQ, so I guess I don't understand your point. If you think events like the world being destroyed by a meteorite, the Sun dying, or a zombie apocalypse should factor in to their 9's reliability percentage, it shouldn't.
        mikeash 12y ago
        OK, why not? Serious question, here. Things like gigantic hurricanes flooding their data centers should factor into it, right? Risk of war destroying the data center should factor into it, right? (I mean, would you trust S3 to the same degree if all of their data centers were located in Gaza?) So why shouldn't a scenario like "all of our data centers are simultaneously destroyed as part of a worldwide nuclear conflict" factor into it?

by jewel 12y ago

ghshephard 12y ago

I think the S3 number is a floor, not a ceiling. That is, you should expect to be losing that much of data, but, given an unplanned event, obviously a great deal more data can be lost.

mikeash 12y ago

Human disasters too. I'd wager the odds of a catastrophic massive nuclear war are higher than one in ten million per year. Not that I expect Amazon (or much of anyone) to protect against that, nor would the integrity of my S3 data be a priority afterwards, but it makes the claim kind of absurd. Edit: reading more closely, their durability number is per object. It's one object lost per 100 billion objects per year. The 10 million number comes from a hypothetical situation where you store 10,000 objects. I don't know how many objects S3 stores altogether, but if we say it's a billion (presumably a vast underestimate) then that would imply that the probability of losing all objects in the system over the coming year is one in 100 quadrillion. I don't think this planet is that safe.

res0nat0r 12y ago
Not sure why it is absurd due to the redundancy they've implemented + data checksumming. Plus AWS hit two trillion objects a bit over a year ago. http://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ http://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objec...
- mikeash 12y ago
  As great as S3 is, it's still confined to Earth. Seems to me that the odds of a planet-wide disaster taking out all Amazon infrastructure (as well as certain less important things like human civilization) are higher than the odds they're giving of a catastrophic S3 failure.
  - res0nat0r 12y ago
    Sure. The AFR of your Western Digital desktop HD doesn't factor in and lower it's reliability slightly because of the slim chance that a 747 is going to crash into your house and destroy your home. Barring unforeseen acts of God, the 9's listed above apply and you just have to personally weigh if you think the risk of S3 losing multiple datacenters is high enough for you to risk storing your data there.
    - mikeash 12y ago
      Amazon pretty explicitly includes unforeseen catastrophic events in their durability estimate. "In addition, Amazon S3 is designed to sustain the concurrent loss of data in two facilities." I sure hope the loss of two facilities doesn't fall into the "foreseen" category!
      - res0nat0r 12y ago
        Sure, it says they account for that right there in their FAQ, so I guess I don't understand your point. If you think events like the world being destroyed by a meteorite, the Sun dying, or a zombie apocalypse should factor in to their 9's reliability percentage, it shouldn't.
        mikeash 12y ago
        OK, why not? Serious question, here. Things like gigantic hurricanes flooding their data centers should factor into it, right? Risk of war destroying the data center should factor into it, right? (I mean, would you trust S3 to the same degree if all of their data centers were located in Gaza?) So why shouldn't a scenario like "all of our data centers are simultaneously destroyed as part of a worldwide nuclear conflict" factor into it?