3 ms·

> What I want from a post mortem is to know how we could prevent, detect or mitigate similar incidents in future and to make those changes to code or process.

by Aeolun 2y ago

> What I want from a post mortem is to know how we could prevent, detect or mitigate similar incidents in future and to make those changes to code or process. The answer this post gives to that bizarre question that always gets asked, is ‘nothing’, unless you want to significantly adjust the speed that we deliver features. Any added process or check is going to impose overhead and make the team a little bit less happy. Ocassionally you’ll have a unicorn situation where there is actually a relatively simple fix, but those are few and far between. In extremis, you’re reduced to a situation in which you have zero incidents, but you also have zero work getting done.

hamandcheese 2y ago
> Ocassionally you’ll have a unicorn situation where there is actually a relatively simple fix, but those are few and far between. Perhaps we have different backgrounds, but even in late stage startups I find there is an abundance of low hanging fruit and simple fixes. I'm sure it's different at Google, though.
scott_w 2y ago
That’s simply not true. Some processes are good, some post-mortem outcomes will focus on improving deployment speed (so you can revert changes faster), or improve your monitors so you can detect (and mitigate) incidents faster. On the other hand, enforcing a manual external QA check on every release WILL slow things down. You’re repeating the same mistake as the article by assuming “process” sits on a grade that naturally slows work down. This is because you’re not being precise in your reasoning. Look at the specifics and make a decision based on the facts in front of you.
- braza 2y ago
  > This is because you’re not being precise in your reasoning. Look at the specifics and make a decision based on the facts in front of you I agree with the premise here, but in my experience running incidents review the issue that I see it’s a mixture of a performatic safetycism with reactivity. Processes are the cheap bandaid to fix design, architectural and cultural issues. Most of the net positive micro-reforms that we had after incident reviews were the ones that invested in safety nets, faster recoveries, and guardrailing than a new process and will tax everyone.
  - scott_w 2y ago
    > Processes are the cheap bandaid to fix design, architectural and cultural issues. They can be, yes. I have a friend that thinks I'm totally insane by wanting to release code to production multiple times a day. His sweet spot is once every 2 weeks because he wants QA to check over every change. Most of his employers can manage once a month at best, and once a quarter is more typical. > Most of the net positive micro-reforms that we had after incident reviews were the ones that invested in safety nets, faster recoveries, and guardrailing than a new process and will tax everyone. I 100% agree with this. Your comment also reminded me to say that incident reviews are necessary but not sufficient. You also need engineering leadership reviewing at a higher-level to make bigger organisational or technical changes to further improve things.

by Aeolun 2y ago

hamandcheese 2y ago

> Ocassionally you’ll have a unicorn situation where there is actually a relatively simple fix, but those are few and far between. Perhaps we have different backgrounds, but even in late stage startups I find there is an abundance of low hanging fruit and simple fixes. I'm sure it's different at Google, though.

scott_w 2y ago

That’s simply not true. Some processes are good, some post-mortem outcomes will focus on improving deployment speed (so you can revert changes faster), or improve your monitors so you can detect (and mitigate) incidents faster. On the other hand, enforcing a manual external QA check on every release WILL slow things down. You’re repeating the same mistake as the article by assuming “process” sits on a grade that naturally slows work down. This is because you’re not being precise in your reasoning. Look at the specifics and make a decision based on the facts in front of you.

braza 2y ago
> This is because you’re not being precise in your reasoning. Look at the specifics and make a decision based on the facts in front of you I agree with the premise here, but in my experience running incidents review the issue that I see it’s a mixture of a performatic safetycism with reactivity. Processes are the cheap bandaid to fix design, architectural and cultural issues. Most of the net positive micro-reforms that we had after incident reviews were the ones that invested in safety nets, faster recoveries, and guardrailing than a new process and will tax everyone.
- scott_w 2y ago
  > Processes are the cheap bandaid to fix design, architectural and cultural issues. They can be, yes. I have a friend that thinks I'm totally insane by wanting to release code to production multiple times a day. His sweet spot is once every 2 weeks because he wants QA to check over every change. Most of his employers can manage once a month at best, and once a quarter is more typical. > Most of the net positive micro-reforms that we had after incident reviews were the ones that invested in safety nets, faster recoveries, and guardrailing than a new process and will tax everyone. I 100% agree with this. Your comment also reminded me to say that incident reviews are necessary but not sufficient. You also need engineering leadership reviewing at a higher-level to make bigger organisational or technical changes to further improve things.