3 ms·
Vivid post-mortems are valuable if they serve their purpose: Motivating action that prevents future post-mortems. But post-mortems themselves are not valuable
by mechanical_fish 13y ago
Vivid post-mortems are valuable if they serve their purpose: Motivating action that prevents future post-mortems.
But post-mortems themselves are not valuable at all. Quite the opposite: Every post-mortem signifies that something has died. Resurrecting the dead has probably consumed a lot of time and resources that could have been better spent on something else.
An organization should never write the same post-mortem twice, and should see the rate of post-mortems diminish over time, to the point where they are incredibly rare. But that doesn't always happen.
Instead, fetishizing post-mortems can be a real problem. It is easy to accidentally build an engineering culture that glorifies firefighting. Every day an expensive team of trained technicians comes to work and dives into the usual series of daily emergencies. Each emergency winds up with a glowing after-action report that gets filed with the other reports. People compete to tell the most entertaining tales of command-line heroics in the face of hilariously broken systems. It can actually be kind of fun, like a game. Everyone feels very productive and important, because without constant vigilance the software would die dozens of times every day. And the post-mortems may be of the highest quality, because every employee has ample opportunity to practice writing them.
In such an environment it's astonishingly easy to lose the plot: It is better to be the organization whose software consistently works, and which never writes post-mortems because nothing ever dies. A boring report is a quality report!
- philwelch 13y agoI think it's good to build a culture where having to write a postmortem is a mark of shame. It seems incredibly perverse to do otherwise, and while I appreciate the thrill of a good crisis, any professional should endeavor to do no more and no less than sleep soundly knowing that nothing will break and page them at night. (Developers aren't in the on-call rotation? That's your first problem.)