3 ms·
These comments crop up regularly on outage posts. There's probably some psychological bias that explains it but I can't remember... I think the last time Google
by throwaway53453 6y ago
These comments crop up regularly on outage posts. There's probably some psychological bias that explains it but I can't remember... I think the last time Google went down there were comments blaming it on Summer Interns :)
- dmlittle 6y agoNo, it was SolarWinds that blamed the intern for their security issues. In all seriousness... if a single employee can accidentally single-handedly screw things up you've got a process problem. There should be checks and balances to make sure that what you're trying to do make sense. These might be in the form of a code-review, automated validation, a staging environments, etc. Yes, these might be things that don't exist at a scrappy startup and that is fine as they'll get there eventually but for any large company not having good processes is reckless.
- throwaway53453 6y agoThese issues happen despite all those things. You can bet that Google has that and more in place. Just like in aerospace engineering, things go wrong when dozens of failures happen across the critical path, not any individual piece.
- dmlittle 6y agoYes, and issues will always occur. What I was trying to point out that it's never an individual's fault alone. Sure someone might have tripped the final domino but they might not have had anything to do with everything else that happened. That's why blameless postmortems are a thing. Blameless (to me) doesn't mean that you don't name any person but rather that there's no blame assigned to anyone. I want to know what Person A did to cause the incident but there shouldn't be any retribution against Person A.
- rzzzt 6y agoFrequency illusion aka. Baader-Meinhof phenomenon? https://en.wikipedia.org/wiki/Frequency_illusion https://en.wikipedia.org/wiki/Frequency_illusion