3 ms·
I'm sure that this Google outage was the combination of several independent issues. Perhaps there are the kind of safeguards you mentioned in place, but they re
by cranekam 6y ago
I'm sure that this Google outage was the combination of several independent issues. Perhaps there are the kind of safeguards you mentioned in place, but they returned an incorrect view of the world because of a bug exposed by a network partition? The point is that with huge distributed systems the things take them down (at least once they're mature) are typically the result of several failures that compound and interact in a way that wasn't foreseen. IMO it's more likely that the suggestions you made (blocks and/or automation) are already in place but failed in some more complex way.
Source: worked at a FAANG for a decade, saw many big incidents. Almost all were the confluence of several smaller issues that, had they happened alone, wouldn't have been newsworthy.