3 ms·

What does 99.999999999% durability mean exactly? Does it mean a probability of 0.000000001% (1 in 100_000_000_000) that your bits will randomly disappear? Is t

What does 99.999999999% durability mean exactly? Does it mean a probability of 0.000000001% (1 in 100_000_000_000) that your bits will randomly disappear? Is that yearly? One interpretation is that about 1 bit per 100 GB will randomly flip each year. That or S3 Glacier is expecting to hit a catastrophic event every 100 billion years (which doesn't seem nearly frequent enough).

mrich 4y ago
This explains it pretty well: https://blog.synology.com/data-durability https://blog.synology.com/data-durability > Data durability: the ability to keep the stored data consistent, intact without the influence of bit rot, drive failures, or any form of corruption. 99.999999999% (11 nines) durability means that if you store 10 million objects, then you expect to lose an object of your data every 10,000 years.
- wongarsu 4y ago
  Doesn't AWS only spread your data over three availability zones, which are three data centers in the same city, often located in close proximity of each other? I'd rate the possibility of Amsterdam suffering a catastrophic flood or nuclear strike that wipes out three data centers at once at more frequently than once every 10,000 years.
  - contravariant 4y ago
    I mean we're already pretty certain that their current design isn't enough to last when the sun explodes so that puts a hard limit of all data disappearing once every 10 billion years or so.
    - gw99 4y ago
      Think we'll have an AWS region in Alpha Centauri by then... Edit: and it'll still break when us-east-1 is destroyed by the sun exploding.
- Aachen 4y ago
  "an object" Shouldn't that be multiplied by the number of bytes of this object? I still don't quite understand the math apparently because then we'd all be losing 10KB+ files yearly.
  - mbreese 4y ago
    I don’t think so. The question is: for any given file, what is the likelihood that if I restore it, there will be some corruption? (Total loss, bitrot, anything) For the purposes of this calculation, you would probably use the average size of a stored file, determine the number of chunks that was split into, and then the likelihood of losing/corrupting a chunk. By looking at the durability of objects, you (a) reinforce the object-store nature of S3, and (b) avoid reporting “durability per kb”, which isn’t linear, and more difficult to interpret. Honestly, the durability number is so high, that adding the bytes as a unit would just make it difficult to reason about. And S3 doesn’t store “bytes” — it stores objects (made of bytes). So, that’s the metric they chose to focus on.
  - xani_ 4y ago
    If we assume for the sake of argument simple RAID 6 setup, to lose say a block of data you'd need to * have a bad block on one drive * have a bad block in *EXACTLY SAME LOCATION* on the second drive * have a bad block in *EXACTLY SAME LOCATION* on the third drive * all of that happening within one scrub period (which is usually once a month). And that's just simple RAID6 setup without external redundancy. As you can see it would be far easier to lose data due to just correlated drive failures, than from random block corruption.
wongarsu 4y ago
Also note how Amazon doesn't claim that the service has 11 nines of data durability, but that it is designed for 11 nines of durability. That's not quite the same. Backblaze apparently gets the same number by looking at the failure rate of their drives, and how long it takes to recover from a drive failure (reaching the same redundancy as before the failure). By being able to recover from three drive failures before the first drive is rebuilt, they get 11 nines of durability [1]. Or in other words: 0.000000001% of drive failures are non-recoverable and lose data. I assume AWS does basically the same calculation, adjusted for whatever technology they use. 1: https://www.theregister.com/2018/07/19/data_durability_statements/ https://www.theregister.com/2018/07/19/data_durability_state...
sliken 4y ago
Not sure exactly. But Backblaze has been pretty transparent on their approach. Backblaze takes user data, splits it into 17 pieces, add 3 pieces of redundancy, and split it across 20 racks. Backblaze has a tiered response system, my memory is something along the lines of 1 of 20 disks dying = schedule for replacement the next business day. The 2nd disk triggers a quicker/higher priority response. 3rd disk triggers pages and all hands on deck. Given that disks have a annual failure rate on the order of 1%, and losing the first disk has a chance of 1% * 1/365, if a second disk is 4 hours that 1% * 1/(3656), and a 3rd disk is 1%/(36524) you end up with lots of zeros: 0.01(1/365) 0.01 * 1/(3656) 0.01 * 1/(365*24) .00000000000000014281 That's just to remove the redundancy, you still need to lose another disk before data is lost. Sure there's other failure methods, including those that could take out an entire data center. I did find a statement saying that data stored in the US-West region would be stored in both Sacramento and Phoenix, but not sure if that means each B2 object uploaded would be in both.

mrich 4y ago

This explains it pretty well: https://blog.synology.com/data-durability https://blog.synology.com/data-durability > Data durability: the ability to keep the stored data consistent, intact without the influence of bit rot, drive failures, or any form of corruption. 99.999999999% (11 nines) durability means that if you store 10 million objects, then you expect to lose an object of your data every 10,000 years.

wongarsu 4y ago
Doesn't AWS only spread your data over three availability zones, which are three data centers in the same city, often located in close proximity of each other? I'd rate the possibility of Amsterdam suffering a catastrophic flood or nuclear strike that wipes out three data centers at once at more frequently than once every 10,000 years.
- contravariant 4y ago
  I mean we're already pretty certain that their current design isn't enough to last when the sun explodes so that puts a hard limit of all data disappearing once every 10 billion years or so.
  - gw99 4y ago
    Think we'll have an AWS region in Alpha Centauri by then... Edit: and it'll still break when us-east-1 is destroyed by the sun exploding.
Aachen 4y ago
"an object" Shouldn't that be multiplied by the number of bytes of this object? I still don't quite understand the math apparently because then we'd all be losing 10KB+ files yearly.
- mbreese 4y ago
  I don’t think so. The question is: for any given file, what is the likelihood that if I restore it, there will be some corruption? (Total loss, bitrot, anything) For the purposes of this calculation, you would probably use the average size of a stored file, determine the number of chunks that was split into, and then the likelihood of losing/corrupting a chunk. By looking at the durability of objects, you (a) reinforce the object-store nature of S3, and (b) avoid reporting “durability per kb”, which isn’t linear, and more difficult to interpret. Honestly, the durability number is so high, that adding the bytes as a unit would just make it difficult to reason about. And S3 doesn’t store “bytes” — it stores objects (made of bytes). So, that’s the metric they chose to focus on.
- xani_ 4y ago
  If we assume for the sake of argument simple RAID 6 setup, to lose say a block of data you'd need to * have a bad block on one drive * have a bad block in *EXACTLY SAME LOCATION* on the second drive * have a bad block in *EXACTLY SAME LOCATION* on the third drive * all of that happening within one scrub period (which is usually once a month). And that's just simple RAID6 setup without external redundancy. As you can see it would be far easier to lose data due to just correlated drive failures, than from random block corruption.

wongarsu 4y ago

Also note how Amazon doesn't claim that the service has 11 nines of data durability, but that it is designed for 11 nines of durability. That's not quite the same. Backblaze apparently gets the same number by looking at the failure rate of their drives, and how long it takes to recover from a drive failure (reaching the same redundancy as before the failure). By being able to recover from three drive failures before the first drive is rebuilt, they get 11 nines of durability [1]. Or in other words: 0.000000001% of drive failures are non-recoverable and lose data. I assume AWS does basically the same calculation, adjusted for whatever technology they use. 1: https://www.theregister.com/2018/07/19/data_durability_statements/ https://www.theregister.com/2018/07/19/data_durability_state...

sliken 4y ago

Not sure exactly. But Backblaze has been pretty transparent on their approach. Backblaze takes user data, splits it into 17 pieces, add 3 pieces of redundancy, and split it across 20 racks. Backblaze has a tiered response system, my memory is something along the lines of 1 of 20 disks dying = schedule for replacement the next business day. The 2nd disk triggers a quicker/higher priority response. 3rd disk triggers pages and all hands on deck. Given that disks have a annual failure rate on the order of 1%, and losing the first disk has a chance of 1% * 1/365, if a second disk is 4 hours that 1% * 1/(3656), and a 3rd disk is 1%/(36524) you end up with lots of zeros: 0.01(1/365) 0.01 * 1/(3656) 0.01 * 1/(365*24) .00000000000000014281 That's just to remove the redundancy, you still need to lose another disk before data is lost. Sure there's other failure methods, including those that could take out an entire data center. I did find a statement saying that data stored in the US-West region would be stored in both Sacramento and Phoenix, but not sure if that means each B2 object uploaded would be in both.