4 ms·

> Tape Gateway stores virtual tapes in Amazon S3, Amazon S3 Glacier Flexible Retrieval, and Amazon S3 Glacier Deep Archive, protected by 99.999999999% of durabi

> Tape Gateway stores virtual tapes in Amazon S3, Amazon S3 Glacier Flexible Retrieval, and Amazon S3 Glacier Deep Archive, protected by 99.999999999% of durability. That’s a lot of 9s.

aftbit 4y ago
Does than mean that you should expect to lose 1 byte for every 100 GB that you store?
- primax 4y ago
  The statement is made at an object level typically for AWS
kcb 4y ago
I hate it. Once the number gets so small beyond comprehension it’s hard to believe the validity. It reminds me of the Feynman’s statements during the Space Shuttle Challenger disaster. > Feynman was disturbed by two aspects of this practice. First, NASA management assigned a probability of failure to each individual bolt, sometimes claiming a probability of 1 in 10^8, i.e. one in one hundred million. Feynman pointed out that it is impossible to calculate such a remote possibility with any scientific rigor. https://en.wikipedia.org/wiki/Rogers_Commission_Report https://en.wikipedia.org/wiki/Rogers_Commission_Report Unless S3 can survive nuclear war or an asteroid hitting the earth, I don’t buy it.
- lopkeny12ko 4y ago
  There is rigor behind the claim. Check out this talk from a few years ago: https://www.youtube.com/watch?v=DzRyrvUF-C0 https://www.youtube.com/watch?v=DzRyrvUF-C0
  - dgacmu 4y ago
    There's rigor within the assumptions. The problem is when the assumptions are wrong - most notably around failure independence. Data center fire? Accounted for. Simultaneous terrorist attack / asteroid (as per GP), / nuclear war on all data centers? Maybe not. Where does insider threat fit in on that spectrum? We don't know. I'm personally fine with the claim as I understand the methodology and assumptions but it has to be interpreted carefully.
- chaxor 4y ago
  One bit change in a sea of 100TB is quite a small percentage. Could work that way.
- daxfohl 4y ago
  Yeah, the most depressing part of my job is when I have to calculate the SLO metrics for our directors. The first pass of stuff we control is always way within SLO. Then we have to subtract stuff where some third party messes up, or like the request doesn't even get to our service and it always puts us in the red.
harshaw 4y ago
Think about it in terms of erasure encoding where you can have enough shards on enough hosts. That's how you get the durability.
- ff317 4y ago
  Yeah but that's also incredibly naive. Inevitably there will be some un-calculated-for shortcuts that make all those copies not as failure-independent as they seem. If nothing else, what are the odds some random Amazon engineer makes a software rollout or hardware upgrade mistake and takes out a whole lot of copies in the process?
vbezhenar 4y ago
It means that 1 bit out of 12.5 GB is not durable. Not a lot if you ask me.

> Tape Gateway stores virtual tapes in Amazon S3, Amazon S3 Glacier Flexible Retrieval, and Amazon S3 Glacier Deep Archive, protected by 99.999999999% of durability. That’s a lot of 9s.

aftbit 4y ago

Does than mean that you should expect to lose 1 byte for every 100 GB that you store?

primax 4y ago
The statement is made at an object level typically for AWS

kcb 4y ago

I hate it. Once the number gets so small beyond comprehension it’s hard to believe the validity. It reminds me of the Feynman’s statements during the Space Shuttle Challenger disaster. > Feynman was disturbed by two aspects of this practice. First, NASA management assigned a probability of failure to each individual bolt, sometimes claiming a probability of 1 in 10^8, i.e. one in one hundred million. Feynman pointed out that it is impossible to calculate such a remote possibility with any scientific rigor. https://en.wikipedia.org/wiki/Rogers_Commission_Report https://en.wikipedia.org/wiki/Rogers_Commission_Report Unless S3 can survive nuclear war or an asteroid hitting the earth, I don’t buy it.

lopkeny12ko 4y ago
There is rigor behind the claim. Check out this talk from a few years ago: https://www.youtube.com/watch?v=DzRyrvUF-C0 https://www.youtube.com/watch?v=DzRyrvUF-C0
- dgacmu 4y ago
  There's rigor within the assumptions. The problem is when the assumptions are wrong - most notably around failure independence. Data center fire? Accounted for. Simultaneous terrorist attack / asteroid (as per GP), / nuclear war on all data centers? Maybe not. Where does insider threat fit in on that spectrum? We don't know. I'm personally fine with the claim as I understand the methodology and assumptions but it has to be interpreted carefully.
chaxor 4y ago
One bit change in a sea of 100TB is quite a small percentage. Could work that way.
daxfohl 4y ago
Yeah, the most depressing part of my job is when I have to calculate the SLO metrics for our directors. The first pass of stuff we control is always way within SLO. Then we have to subtract stuff where some third party messes up, or like the request doesn't even get to our service and it always puts us in the red.

harshaw 4y ago

Think about it in terms of erasure encoding where you can have enough shards on enough hosts. That's how you get the durability.

ff317 4y ago
Yeah but that's also incredibly naive. Inevitably there will be some un-calculated-for shortcuts that make all those copies not as failure-independent as they seem. If nothing else, what are the odds some random Amazon engineer makes a software rollout or hardware upgrade mistake and takes out a whole lot of copies in the process?

vbezhenar 4y ago

It means that 1 bit out of 12.5 GB is not durable. Not a lot if you ask me.