4 ms·

I’ve seen it a lot in well-intentioned post-mortems. Some bad code gets pushed to production and causes an outage. The responsible team does a thorough review,

by yojo 5y ago

I’ve seen it a lot in well-intentioned post-mortems. Some bad code gets pushed to production and causes an outage. The responsible team does a thorough review, and in a bout of self-flagellation proposes a long list of extra processes that add steps and checks essentially amounting to “don’t trust anyone to commit anything right ever.” If a level-headed manager doesn’t step in, this pile of process gets implemented, output drops, and people hate their jobs. I’ve seen a similar thing with integration tests, as on a team with decent unit test practices production bugs are often a result of bad assumptions about another team’s API. Eventually a mountain of brittle end-to-end tests cripples a team until they get managerial buy-in to rip them all out and the cycle starts over.

Aeolun 5y ago
We are there now. It’s nice if people keep asking ‘how can we prevent this’, but at some point you have to say ‘we cannot’.
dredmorbius 5y ago
This is the dark side / failure mode of checklist-based processes. Yes, checklists and process standardisation will avoid numerous types of errors and issues. However one of your checklist processes must be to assess the checklist itself, weeding and refactoring the checklist itself from time to time. This includes recognising that the refactored checklist will itself have further issues and shakedown bugs. Ultimately process is a compromise between what can be kept in working or near-at-hand memory, and what's both sufficient and required for the particular problem domain.
- deleted 5y ago
  [deleted]
- yojo 5y ago
  That’s great, I’ve never thought of implementing “process refactoring” as a standard recurring process, but I will now.
SilasX 5y ago
That … seems like a company that isn’t doing postmortems correctly. You’re not supposed to robotically implement every “what would have prevented this”. It’s just there to explore the space of such things and provide your options. “This is so rare and low impact that we change nothing” is a valid option! The purpose of keeping a record of past postmortems is that you can potentially go back and say “oh gosh, it isn’t actually rare, and so maybe the more elaborate countermeasures are justified”. I’ve seen the opposite extreme happen, where a devops guy was adamantly against writing a postmortem at all, because “it wasn’t our fault” and “it probably won’t happen again”. Facepalm.
- yojo 5y ago
  Oh this is definitely me describing a failure mode of postmortems, not an indictment of the process generally. It usually happens with well intentioned junior teams that lack experience/perspective. I’m a firm believer in learning from mistakes, and making the right trade offs between reliability and velocity. Postmortems are a fantastic tool for facilitating this. As you point out though, the implementation matters and you need to be conscious of the cost/benefit of the recommendations that come out of the exercise.
tdumitrescu 5y ago
> The responsible team does a thorough review, and in a bout of self-flagellation proposes a long list of extra processes that add steps and checks essentially amounting to “don’t trust anyone to commit anything right ever.” If that list of extra checks is automated, i.e. added to your deployment scripts rather than manually followed by a human, then that's great. Every bit of repetitive decision-making and tribal knowledge that you can eliminate from your team's day-to-day functioning is a win.

by yojo 5y ago

Aeolun 5y ago

We are there now. It’s nice if people keep asking ‘how can we prevent this’, but at some point you have to say ‘we cannot’.

dredmorbius 5y ago

This is the dark side / failure mode of checklist-based processes. Yes, checklists and process standardisation will avoid numerous types of errors and issues. However one of your checklist processes must be to assess the checklist itself, weeding and refactoring the checklist itself from time to time. This includes recognising that the refactored checklist will itself have further issues and shakedown bugs. Ultimately process is a compromise between what can be kept in working or near-at-hand memory, and what's both sufficient and required for the particular problem domain.

deleted 5y ago
[deleted]
yojo 5y ago
That’s great, I’ve never thought of implementing “process refactoring” as a standard recurring process, but I will now.

SilasX 5y ago

That … seems like a company that isn’t doing postmortems correctly. You’re not supposed to robotically implement every “what would have prevented this”. It’s just there to explore the space of such things and provide your options. “This is so rare and low impact that we change nothing” is a valid option! The purpose of keeping a record of past postmortems is that you can potentially go back and say “oh gosh, it isn’t actually rare, and so maybe the more elaborate countermeasures are justified”. I’ve seen the opposite extreme happen, where a devops guy was adamantly against writing a postmortem at all, because “it wasn’t our fault” and “it probably won’t happen again”. Facepalm.

yojo 5y ago
Oh this is definitely me describing a failure mode of postmortems, not an indictment of the process generally. It usually happens with well intentioned junior teams that lack experience/perspective. I’m a firm believer in learning from mistakes, and making the right trade offs between reliability and velocity. Postmortems are a fantastic tool for facilitating this. As you point out though, the implementation matters and you need to be conscious of the cost/benefit of the recommendations that come out of the exercise.

tdumitrescu 5y ago

> The responsible team does a thorough review, and in a bout of self-flagellation proposes a long list of extra processes that add steps and checks essentially amounting to “don’t trust anyone to commit anything right ever.” If that list of extra checks is automated, i.e. added to your deployment scripts rather than manually followed by a human, then that's great. Every bit of repetitive decision-making and tribal knowledge that you can eliminate from your team's day-to-day functioning is a win.