3 ms·
From the article capacity was reduced
by erhk 6y ago
From the article capacity was reduced
- mrighele 6y agoYou're right. Still, a better approach could have been adopted, like blocking a change that would cause some service to go over quota, or have the automatic system change the quotas gradually, thus giving time for alerts and human intervention
- cranekam 6y agoI'm sure that this Google outage was the combination of several independent issues. Perhaps there are the kind of safeguards you mentioned in place, but they returned an incorrect view of the world because of a bug exposed by a network partition? The point is that with huge distributed systems the things take them down (at least once they're mature) are typically the result of several failures that compound and interact in a way that wasn't foreseen. IMO it's more likely that the suggestions you made (blocks and/or automation) are already in place but failed in some more complex way. Source: worked at a FAANG for a decade, saw many big incidents. Almost all were the confluence of several smaller issues that, had they happened alone, wouldn't have been newsworthy.