4 ms·

To be fair, do "incident reviews" mean anything? Most people blame someone else when things go south. How many people would complete a survey with, "Honestly, I

To be fair, do "incident reviews" mean anything? Most people blame someone else when things go south. How many people would complete a survey with, "Honestly, I'm not a very good programmer, I had no idea this piece of code was exploit-able."

ochronus 3y ago
Yes, they do – if done well. These are supposed to be blameless postmortems, focusing on what we can do better in the system to prevent similar issues from happening in the future. The answer is never “get better programmers”. Typical action items from the couple of recent postmortems I led/were a part of were along the lines of “decrease this alert's threshold, so we know about it earlier” ; “we need to invest in a proper local testing environment” ; “let's make our service independent of that other service” etc.
MadnessASAP 3y ago
A particular bit that I appreciate about my workplaces "incident review" process is that it is anonymous and non-punitave. They exist solely to understand the root causes that lead to an incident. They also don't stop asking why, an employee did something stupid? Why? Were they not properly trained? Why not? Were they pressured? By whom and why? Why did they pressure the first? Etc, etc. They're can of course be a parallel disciplinary investigation of the situation so requires, but they don't share notes and you're not obligated to tell them both the same story. It's a model I wish more organizations would adopt. People make mistakes or deliberately do things that are wrong for all kinds of reasons. Pretending they don't or won't is a poor plan.
piva00 3y ago
I have written my fair share of post-mortems with a conclusion of "my knowledge of the topic/domain was limited at the time of implementation and hence created issue X due to Y". It all depends on the organisation's culture, if management is always looking for blame you'll get meaningless incident reviews caused by fear, if there's trust you'll get a larger share of honest reviews with actionable points to minimising the risk of it happening again.

ochronus 3y ago

Yes, they do – if done well. These are supposed to be blameless postmortems, focusing on what we can do better in the system to prevent similar issues from happening in the future. The answer is never “get better programmers”. Typical action items from the couple of recent postmortems I led/were a part of were along the lines of “decrease this alert's threshold, so we know about it earlier” ; “we need to invest in a proper local testing environment” ; “let's make our service independent of that other service” etc.

MadnessASAP 3y ago

A particular bit that I appreciate about my workplaces "incident review" process is that it is anonymous and non-punitave. They exist solely to understand the root causes that lead to an incident. They also don't stop asking why, an employee did something stupid? Why? Were they not properly trained? Why not? Were they pressured? By whom and why? Why did they pressure the first? Etc, etc. They're can of course be a parallel disciplinary investigation of the situation so requires, but they don't share notes and you're not obligated to tell them both the same story. It's a model I wish more organizations would adopt. People make mistakes or deliberately do things that are wrong for all kinds of reasons. Pretending they don't or won't is a poor plan.

piva00 3y ago

I have written my fair share of post-mortems with a conclusion of "my knowledge of the topic/domain was limited at the time of implementation and hence created issue X due to Y". It all depends on the organisation's culture, if management is always looking for blame you'll get meaningless incident reviews caused by fear, if there's trust you'll get a larger share of honest reviews with actionable points to minimising the risk of it happening again.