3 ms·

> When an engineer causes an incident, we fixed it and then did a review to understand how we can improve the process to prevent such an incident in future Yes

> When an engineer causes an incident, we fixed it and then did a review to understand how we can improve the process to prevent such an incident in future Yes, sure most of the time its the process's fault. But not always. The big issue is that unlike an incident review where agreed, measurable fixes will be put in place, in business there is no such thing. Sure, Sundar admits that he made the wrong call. But, why? what information lead to that call, what changes are going to be made to how decisions are made in the future? how can the rest of the company be sure that this is a one time thing, and not a systemic information bias.

nindalf 4y ago
I assume there’s been a post-mortem for this incident, considering the cost to the company in terms of morale, pain for former employees, damage to hiring brand etc. They’ve probably learned lessons from this so they don’t repeat them in future. It’s hard to make the results of the post-mortem public because most of this would be decisions taken by specific people, instead of systems misfiring. They would not want to publicly throw the people under the bus.
- KaiserPro 4y ago
  > I assume there’s been a post-mortem for this incident That requires trust. The sysadmin team where I work have a weekly incident review, its vaguely public and although I don't often go, trust is built because results are not private or hidden. > It’s hard to make the results of the post-mortem public because most of this would be decisions taken by specific people, Public yes, but wider inside the company, less so (there are some caveats.) but as for throwing people under a bus, that's never the aim for an incident review. Sure if someone clearly acted with malice, then perhaps action might be taken. The point of incident reviews is to work out what happened, why, and if its likely to happen again. The general rule is that if you allow people to speak honestly in a review, and not punish them after, you get a better result.

nindalf 4y ago

I assume there’s been a post-mortem for this incident, considering the cost to the company in terms of morale, pain for former employees, damage to hiring brand etc. They’ve probably learned lessons from this so they don’t repeat them in future. It’s hard to make the results of the post-mortem public because most of this would be decisions taken by specific people, instead of systems misfiring. They would not want to publicly throw the people under the bus.

KaiserPro 4y ago
> I assume there’s been a post-mortem for this incident That requires trust. The sysadmin team where I work have a weekly incident review, its vaguely public and although I don't often go, trust is built because results are not private or hidden. > It’s hard to make the results of the post-mortem public because most of this would be decisions taken by specific people, Public yes, but wider inside the company, less so (there are some caveats.) but as for throwing people under a bus, that's never the aim for an incident review. Sure if someone clearly acted with malice, then perhaps action might be taken. The point of incident reviews is to work out what happened, why, and if its likely to happen again. The general rule is that if you allow people to speak honestly in a review, and not punish them after, you get a better result.