5 ms·

A pretty good post mortem, but one worrisome final comment: "For us, availability is the most important feature of DynamoDB" I would think that durability sho

A pretty good post mortem, but one worrisome final comment: "For us, availability is the most important feature of DynamoDB" I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?

manuelflara 11y ago
They probably said that because this was an availability outage. If this was a postmortem after some data had been lost, they would've ended it with "For us, durability is the most important feature of DynamoDB".
stingraycharles 11y ago
I think this is written from an ops point of view. Dynamo-based systems allow a client to wait until at least X servers have acknowledged the write, and use eventual consistency in case there is some service disruption. I personally didnt know they used meta data tables in Dynamo, and that sounds like they lost one of the most important traits which gave Dynamo its super-high availability abilities, which is having all the cluster state in the clients rather than in the servers.
- sillypog 11y ago
  Thank you for saying that! I just read the Dynamo paper the other day and that was the first thing that jumped out at me when I read this outage description. Now I feel like I understood the paper.
  - zackelan 11y ago
    Despite the name, DynamoDB is not actually an implementation of the Dynamo paper. It's influenced by those ideas, as well as the issues with scaling SimpleDB, but other than that is a from-scratch implementation. http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....
EwanToo 11y ago
I suspect they're simply trying to reassure customers that this size of outage won't happen again, not make some kind of deep mission statement..
- jMyles 11y ago
  If this is true (and I think you're right), then pronouncements about "the most important feature" are probably not timely.
sturadnidge 11y ago
If I'm reading their post-outage actions correctly, they certainly align with availability > durability. There are many things to like about DynamoDB, but one of the things I really dislike is the 'abscence of an error == success' pattern that they implement for many operations. With less frequent metadata requests, I imagine there is a greater chance of silent failures (ie availability looks fine, durability is gone). Could be wrong though, I'm not going to pretend to know more about distributed systems than the DynamoDB developers.
- senderista 11y ago
  Durability is certainly not being compromised. I think they are just referring to increasing lease times. Timeouts are always a tradeoff (false alarm frequency vs. recovery time), and this event has prompted them to re-evaluate the tradeoff. If you suspect the design of DynamoDB is sloppy, I encourage you to read this: http://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-services-uses-formal-methods/fulltext http://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-s...
brudgers 11y ago
Since the \A\ in ACID is atomicity, I suspect the reading context for "availability" is CAP.
deleted 11y ago
[deleted]

manuelflara 11y ago

They probably said that because this was an availability outage. If this was a postmortem after some data had been lost, they would've ended it with "For us, durability is the most important feature of DynamoDB".

stingraycharles 11y ago

I think this is written from an ops point of view. Dynamo-based systems allow a client to wait until at least X servers have acknowledged the write, and use eventual consistency in case there is some service disruption. I personally didnt know they used meta data tables in Dynamo, and that sounds like they lost one of the most important traits which gave Dynamo its super-high availability abilities, which is having all the cluster state in the clients rather than in the servers.

sillypog 11y ago
Thank you for saying that! I just read the Dynamo paper the other day and that was the first thing that jumped out at me when I read this outage description. Now I feel like I understood the paper.
- zackelan 11y ago
  Despite the name, DynamoDB is not actually an implementation of the Dynamo paper. It's influenced by those ideas, as well as the issues with scaling SimpleDB, but other than that is a from-scratch implementation. http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....

EwanToo 11y ago

I suspect they're simply trying to reassure customers that this size of outage won't happen again, not make some kind of deep mission statement..

jMyles 11y ago
If this is true (and I think you're right), then pronouncements about "the most important feature" are probably not timely.

sturadnidge 11y ago

If I'm reading their post-outage actions correctly, they certainly align with availability > durability. There are many things to like about DynamoDB, but one of the things I really dislike is the 'abscence of an error == success' pattern that they implement for many operations. With less frequent metadata requests, I imagine there is a greater chance of silent failures (ie availability looks fine, durability is gone). Could be wrong though, I'm not going to pretend to know more about distributed systems than the DynamoDB developers.

senderista 11y ago
Durability is certainly not being compromised. I think they are just referring to increasing lease times. Timeouts are always a tradeoff (false alarm frequency vs. recovery time), and this event has prompted them to re-evaluate the tradeoff. If you suspect the design of DynamoDB is sloppy, I encourage you to read this: http://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-services-uses-formal-methods/fulltext http://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-s...

brudgers 11y ago

Since the \A\ in ACID is atomicity, I suspect the reading context for "availability" is CAP.

deleted 11y ago

[deleted]