4 ms·

> Amazon S3 is an object base store. > Highly durable, 11 9’s of reliability. My eye twitched when I read this, considering that I remember more than one aws o

> Amazon S3 is an object base store. > Highly durable, 11 9’s of reliability. My eye twitched when I read this, considering that I remember more than one aws outage that affected s3 availability/reliability. AWS own pages mention 10 nines for /durability/, however, I doubt this durability number, this level is almost unachievable by any end customer product. Could it be some clever durability accounting? I'm quite disappointed on AWS service status reporting when the service may be disrupted so so badly, to the point of not being serve a single request, however status pages not mentioning problems at all, or just reporting "increased error rates".

antonvs 8y ago
The quoted 11 nines durability number is saying that you won't lose your data. Availability has nothing to do with it. It seems the OP used the wrong word and should have said "durability" instead of "reliability". > this level is almost unachievable by any end customer product. Not sure what your point is here, unless you're highlighting the benefits of using a cloud provider. AWS is not using an "end customer product" to implement S3. They're providing a managed service, and their implementation and management is what allows it to achieve those levels of durability.
mleonhard 8y ago
Availability and durability are different. S3 has 11 9's of durability. This means you can expect to lose one out of 100,000,000,000 objects stored in S3, during an unspecified period of time. S3 achieves this by using "erasure codes", distributing chunks across hundreds of hard drives, and quickly replacing lost chunks. Hadoop is a popular storage system that can do erasure coding on your own servers. To make it more durable, you will need many reliable hard drives. But I didn't find any indication that Hadoop detects and replaces lost chunks automatically, so its durability is severely limited compared to S3. S3 has 4 9's of availability. This means you must expect S3 to be inaccessible 0.0001 of the time, or about an hour per year.

antonvs 8y ago

The quoted 11 nines durability number is saying that you won't lose your data. Availability has nothing to do with it. It seems the OP used the wrong word and should have said "durability" instead of "reliability". > this level is almost unachievable by any end customer product. Not sure what your point is here, unless you're highlighting the benefits of using a cloud provider. AWS is not using an "end customer product" to implement S3. They're providing a managed service, and their implementation and management is what allows it to achieve those levels of durability.

mleonhard 8y ago

Availability and durability are different. S3 has 11 9's of durability. This means you can expect to lose one out of 100,000,000,000 objects stored in S3, during an unspecified period of time. S3 achieves this by using "erasure codes", distributing chunks across hundreds of hard drives, and quickly replacing lost chunks. Hadoop is a popular storage system that can do erasure coding on your own servers. To make it more durable, you will need many reliable hard drives. But I didn't find any indication that Hadoop detects and replaces lost chunks automatically, so its durability is severely limited compared to S3. S3 has 4 9's of availability. This means you must expect S3 to be inaccessible 0.0001 of the time, or about an hour per year.