4 ms·
There's nothing in the incident report that says this was due to human action. An automated process could just have easily pushed configs to the wrong environme
by packetslave 13y ago
There's nothing in the incident report that says this was due to human action. An automated process could just have easily pushed configs to the wrong environment (either through a bug or a mis-configuration of the process itself). Not saying that's what happened, but the IR doesn't say.
In my experience (having personally been the trigger for a widespread outage that got us in the news), Google takes a fairly sophisticated view of outages: SREs as a rule care less about crucifying the engineer who did it, and much more about questions like:
what was the REAL root cause that caused this to happen?
why didn't our processes/tools STOP it from happening?
did our monitoring detect it?
could we have detected it faster?
did we fix it or mitigate it fast enough?
did things like communication, escalations, handoffs between teams, etc. work effectively?
what can we do better next time?
When I first started, my director took a bunch of us out to a Noogler lunch. We sat down with our plates, and he said "Ok, let's talk about how to get fired." Basically, mistakes happen. Bugs happen. If you cause a huge outage, that isn't necessarily a negative reflection on you, and shouldn't hurt your perf. If you cause a huge outage because you willfully ignored procedure, went around established safety controls, didn't monitor to make sure your changes didn't turn all the pretty dashboards red, THEN you're going to have a problem.