4 ms·
One of the most severe S3 outages[1] was triggered by a single NIC flipping bits. S3 maintains (at least back then) the routing table (object to node mapping) i
by vishnugupta 5y ago
One of the most severe S3 outages[1] was triggered by a single NIC flipping bits. S3 maintains (at least back then) the routing table (object to node mapping) in a Merkle Tree data structure and it keeps getting updated via gossip protocol. This bit-flipping machine however, would spread misinformation about the routing table forcing all its neighbours and their neighbours (and so on) to sync their Merkle Trees. So a large number of machines spent more time gossiping than serving the customer request.
I vividly remember the outage as I was at Amazon during that time, in fact I was in AWS group but a different team. I guess the ticket was escalated to sev-1 as large number of tier-1 services had begun depending on S3. The COE action items took months to get rolled out.
While it's one thing to say be ready for all kinds of failure, including hardware, it takes massive failures to internalise the lessons. AWS is now paranoid about being resilient against all kinds of failures including earthquake and lightening.
[1] https://status.aws.amazon.com/s3-20080720.html https://status.aws.amazon.com/s3-20080720.html
- pm90 5y agoIt always takes that one incident to reprioritize reliability. Kudos to AWS for actually taking action to change the org to be more resilient though. Most organizations that have reliability issues will hire a QE expert or have some Quality initiatives that start hot but fizzle out as leadership loses interest until another disaster hits again.
- sokoloff 5y ago> So a large number of machines spent more time gossiping than serving the customer request. Machines are becoming increasingly capable of automating jobs previously only done by humans.
- Abishek_Muthian 5y agoWhat did you call that single NIC flipping bits event in Amazon? Is the term 'cosmic-ray-induced error' outdated in FAANG? >Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month. [1]https://en.wikipedia.org/wiki/Cosmic_ray#Effect_on_electronics https://en.wikipedia.org/wiki/Cosmic_ray#Effect_on_electroni...
- deleted 5y ago[deleted]