2 ms·
[Disclosure: I used to work at Amazon, but not on Dynamo or Cart, and have no non-public information about those systems.] I think this is based on a flawed ap
by mbrubeck 17y ago
[Disclosure: I used to work at Amazon, but not on Dynamo or Cart, and have no non-public information about those systems.]
I think this is based on a flawed application of metrics from other organizations that do not match Amazon's actual business needs.
The author talks about bank transaction systems with five nines of availability. Sure, you can build a centralized system with a critical core that has near-zero downtime - usually at great operational cost - but there's not a bank in the world that's actually 99.999% available to customers, over the internet. Most retail banks I've used take several hours of outages per month for maintenance on their web services.
Amazon wants its systems not just to keep running but to keep taking orders from millions of people all over the world. This means that they are concerned with the reliability not just of a storage server, but everything needed to connect it to its application servers and to customers. They have found that the cost-effective way to do this is to distribute every component across geographically separated datacenters. Amazon has remained available through real and simulated datacenter-wide outages ranging from power/cooling failures to floods, fire, and hurricanes. No Amazon system lives within a single building, much less a single network switch or rack.
Finally, although the formally provable aspects of Dynamo's "eventual consistency" guarantee may be vague, any team operating a distributed system will study and understand the actual operational characteristics in practice, under both normal conditions and various failure modes. Some systems may have realistic failures that lead to days or hours of inconsistency (in which case the team has deliberately chosen this and will write client software to be aware of it); others might be tuned to achieve consistency within milliseconds under normal operation and to set off alarms within seconds after a failure. I've never known any that would be used in circumstances where "human lifetimes" are a relevant time period.