I have no doubt they would have been using randomised exponential backoff -- these issues are well recognised inside Amazon and best practices are well known.

I have no doubt they would have been using randomised exponential backoff -- these issues are well recognised inside Amazon and best practices are well known. For example, here's a blog post from March this year from Marc Brooker on the topic: http://www.awsarchitectureblog.com/2015/03/backoff.html http://www.awsarchitectureblog.com/2015/03/backoff.html . It may not have been correctly tuned for this scenario however. I'd say the Dynamo team are well aware of what they should have been doing, and kicking themselves for not foreseeing this cascading-failure case. ouch!