4 ms·

This explains it pretty well: https://blog.synology.com/data-durability https://blog.synology.com/data-durability > Data durability: the ability to keep the s

by mrich 4y ago

This explains it pretty well: https://blog.synology.com/data-durability https://blog.synology.com/data-durability > Data durability: the ability to keep the stored data consistent, intact without the influence of bit rot, drive failures, or any form of corruption. 99.999999999% (11 nines) durability means that if you store 10 million objects, then you expect to lose an object of your data every 10,000 years.

wongarsu 4y ago
Doesn't AWS only spread your data over three availability zones, which are three data centers in the same city, often located in close proximity of each other? I'd rate the possibility of Amsterdam suffering a catastrophic flood or nuclear strike that wipes out three data centers at once at more frequently than once every 10,000 years.
- contravariant 4y ago
  I mean we're already pretty certain that their current design isn't enough to last when the sun explodes so that puts a hard limit of all data disappearing once every 10 billion years or so.
  - gw99 4y ago
    Think we'll have an AWS region in Alpha Centauri by then... Edit: and it'll still break when us-east-1 is destroyed by the sun exploding.
Aachen 4y ago
"an object" Shouldn't that be multiplied by the number of bytes of this object? I still don't quite understand the math apparently because then we'd all be losing 10KB+ files yearly.
- mbreese 4y ago
  I don’t think so. The question is: for any given file, what is the likelihood that if I restore it, there will be some corruption? (Total loss, bitrot, anything) For the purposes of this calculation, you would probably use the average size of a stored file, determine the number of chunks that was split into, and then the likelihood of losing/corrupting a chunk. By looking at the durability of objects, you (a) reinforce the object-store nature of S3, and (b) avoid reporting “durability per kb”, which isn’t linear, and more difficult to interpret. Honestly, the durability number is so high, that adding the bytes as a unit would just make it difficult to reason about. And S3 doesn’t store “bytes” — it stores objects (made of bytes). So, that’s the metric they chose to focus on.
- xani_ 4y ago
  If we assume for the sake of argument simple RAID 6 setup, to lose say a block of data you'd need to * have a bad block on one drive * have a bad block in *EXACTLY SAME LOCATION* on the second drive * have a bad block in *EXACTLY SAME LOCATION* on the third drive * all of that happening within one scrub period (which is usually once a month). And that's just simple RAID6 setup without external redundancy. As you can see it would be far easier to lose data due to just correlated drive failures, than from random block corruption.

by mrich 4y ago

wongarsu 4y ago

Doesn't AWS only spread your data over three availability zones, which are three data centers in the same city, often located in close proximity of each other? I'd rate the possibility of Amsterdam suffering a catastrophic flood or nuclear strike that wipes out three data centers at once at more frequently than once every 10,000 years.

contravariant 4y ago
I mean we're already pretty certain that their current design isn't enough to last when the sun explodes so that puts a hard limit of all data disappearing once every 10 billion years or so.
- gw99 4y ago
  Think we'll have an AWS region in Alpha Centauri by then... Edit: and it'll still break when us-east-1 is destroyed by the sun exploding.

Aachen 4y ago

"an object" Shouldn't that be multiplied by the number of bytes of this object? I still don't quite understand the math apparently because then we'd all be losing 10KB+ files yearly.

mbreese 4y ago
I don’t think so. The question is: for any given file, what is the likelihood that if I restore it, there will be some corruption? (Total loss, bitrot, anything) For the purposes of this calculation, you would probably use the average size of a stored file, determine the number of chunks that was split into, and then the likelihood of losing/corrupting a chunk. By looking at the durability of objects, you (a) reinforce the object-store nature of S3, and (b) avoid reporting “durability per kb”, which isn’t linear, and more difficult to interpret. Honestly, the durability number is so high, that adding the bytes as a unit would just make it difficult to reason about. And S3 doesn’t store “bytes” — it stores objects (made of bytes). So, that’s the metric they chose to focus on.
xani_ 4y ago
If we assume for the sake of argument simple RAID 6 setup, to lose say a block of data you'd need to * have a bad block on one drive * have a bad block in *EXACTLY SAME LOCATION* on the second drive * have a bad block in *EXACTLY SAME LOCATION* on the third drive * all of that happening within one scrub period (which is usually once a month). And that's just simple RAID6 setup without external redundancy. As you can see it would be far easier to lose data due to just correlated drive failures, than from random block corruption.