6 ms·

> I think this paper is super important, and anybody who designs or runs big systems should read it and take the core point to heart. Please forgive my naivene

by nuerow 5y ago

> I think this paper is super important, and anybody who designs or runs big systems should read it and take the core point to heart. Please forgive my naiveness but isn't this article just bundling well known and basic failure modes, which are already covered in distributed system 101 courses, into a new keyword? The article covers retry storms, cache invalidation and resets (which even AWS documented well[1]), and brownouts. These are basic distributed system failure modes. What is there to gain by coining a redundant keyword to refer to basic failure modes in distributed systems which are already covered in intro courses? [1] https://aws.amazon.com/builders-library/caching-challenges-and-strategies/ https://aws.amazon.com/builders-library/caching-challenges-a...

andrewf 5y ago
I think there's value in rolling the different failure modes up into a single concept for discussion and recall. And the whole thing comes in at 5 pages! "The thing that caused this problem isn't necessarily what's sustaining it" is something I'll commonly call out to people managing an incident, often before we know what the failure modes are. I looked through the AWS Builders' Library for articles which seemed to cover metastable failures. (Disclosure: Work there. Disclaimer: Writing here for myself, not my employer). A system designer with more time might be interested in: https://aws.amazon.com/builders-library/avoiding-fallback-in-distributed-systems/ https://aws.amazon.com/builders-library/reliability-and-constant-work/ https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/ https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload/ https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/ If the Google SRE book is your preferred school of thought, you might like: https://sre.google/sre-book/load-balancing-datacenter/ https://sre.google/sre-book/handling-overload/ https://sre.google/sre-book/addressing-cascading-failures/

by nuerow 5y ago

andrewf 5y ago

I think there's value in rolling the different failure modes up into a single concept for discussion and recall. And the whole thing comes in at 5 pages! "The thing that caused this problem isn't necessarily what's sustaining it" is something I'll commonly call out to people managing an incident, often before we know what the failure modes are. I looked through the AWS Builders' Library for articles which seemed to cover metastable failures. (Disclosure: Work there. Disclaimer: Writing here for myself, not my employer). A system designer with more time might be interested in: https://aws.amazon.com/builders-library/avoiding-fallback-in-distributed-systems/ https://aws.amazon.com/builders-library/reliability-and-constant-work/ https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/ https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload/ https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/ If the Google SRE book is your preferred school of thought, you might like: https://sre.google/sre-book/load-balancing-datacenter/ https://sre.google/sre-book/handling-overload/ https://sre.google/sre-book/addressing-cascading-failures/