10 ms·

Blameless post-mortems and a just culture (2012)

xrd 7y ago
I love blameless postmortems. And, it isn't really whether the team or the team lead decides to do a blameless postmortem. It's whether the company values looking at things this way. And, those values change over time, and shift when the times are good or bad (financially). I would really love to read about a blameless postmortem that happened in a dying company where everyone was backstabbing each other. Now, that would be a good story! In fairness, this story does convey that. The "second story" part is great. I do wish there was a little more about how management came around to seeing it, how they rectified bonuses and pay and PIPs. There is an untold journey here.
- pm90 7y ago
  > I would really love to read about a blameless postmortem that happened in a dying company where everyone was backstabbing each other. Now, that would be a good story! It doesn't have to be a dying company. Many companies that try to transition from startup -> small company -> medium company -> megacorp tend to run into periods of change and insanity in every one of those transitions. I suspect its one of the reasons why most companies die out at some point: the failure to focus on product and focusing instead on politics. Teams turtle down, avoid responsibility (i.e. blame) at every opportunity, the product turns to shit as nobody wants to move a muscle in case anything breaks.
- kqr 7y ago
  I have been recommended the book "Turning the Ship Around" for an account of how to grow a new good culture in a decisively bad one. It's probably not an exact recipe for being able to hold blameless postmortems, but I suspect it could contain some valuable information. If anyone knows how well it applies to this situation, please do tell me!
  - asplake 7y ago
    TBH I’m not sure of its relevance to postmortems specifically but I’d highly recommend the book. Turn the Ship Around, Marquet. Great audiobook too.
  - di4na 7y ago
    TTSA is a great book and may help. But it is a leadership book. I still strongly advise it, at least to know nice things are possible.
- di4na 7y ago
  So this came from management. It is John Allspaw that pushed for them, at the time VP of Eng iirc. For how to do it... there is a community at https://www.learningfromincidents.io/ https://www.learningfromincidents.io/ trying to help. The consensus so far is to test waters and do some "in parallel", if necessary rogue-ish. The output is so superior in term of learning that you may then use this result as a trojan horse. You need to carry your water.
- namdnay 7y ago
  I think you can go too far in that direction. I worked for a big European software company, there was a deep culture of never blaming people, so any postmortem inevitably ended with a new process or training or transversal synergy coordinators, when sometimes the root cause was just bad developers
  - GiuseppaAcciaio 7y ago
    Assuming that "bad developers" is indeed the root cause, that still sounds like a systemic issue to me: a) are the developers bad because they were hired that way? maybe the hiring process is broken b) are the developers bad because they are not managed/trained properly? maybe managers aren't doing their job, and maybe it's not even their fault because they did not get training in how to manage people in the first place c) if you implement extra layers (or "synergy coordinators") it probably means that the company hasn't figured out how to communicate properly
    - AmericanChopper 7y ago
      Management is ultimately responsible for everything. Even if you have the worst developer in the world on your team, management decided to hire them, retain them, assign them responsibility, and failed to improve their performance. That said, plenty of people are just self-serving dickheads. Could they be reformed? Probably. Is that the company’s responsibility? Only up to a point. I’ve worked with people that I was responsible for training in DevOps, from a starting point of having only ever done GUI-based Ops. Basically starting from complete technical incompetence, and plenty of them succeeded, because they were good people with good attitudes. I’d wager a small team of people like that would always be more productive than a team of any sized comprised of arrogant, combative, CYA assholes.
kqr 7y ago
Only tangentially related but I have to get it out of my system anyway: I watched the latest season of the F1 documentary on Netflix, and what stood out beyond anything else to me was that every team pitted their own drivers against each other, had a "you win or you lose" mentality, screamed at people for messing up, and so on. All teams except one. One team iterated how they "win as a team and make mistakes as a team", mistakes are an opportunity to learn, and so on. One team showed pretty clearly how they had a good culture surrounding failures. I'm not at all surprised that this one team is the team that has consistently won the last 6 years in a row. Why aren't more teams copying this? Why is this one team not more secretive about their recipe to success?
- op03 7y ago
  Both strategies are known to work - Carrot and Stick. Why? Because the psychological Traits and Needs of all humans are not the same. Most people who build teams have a sense of their own Needs and Traits and pick strategies/team members that align/compensate. https://en.wikipedia.org/wiki/Revised_NEO_Personality_Inventory#Personality_dimensions https://en.wikipedia.org/wiki/Revised_NEO_Personality_Invent... https://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs https://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs There is a distribution of Traits/Needs that you need to be aware of within any population, group, team etc. If you are not aware of those variations and randomly create groups and try to impose strategy, what works and fails will always remain mysterious.
- MikeAmelung 7y ago
  As the other reply kind of hints at, it's a lot easier to do this if you have the absolutely dominant driver on your team. If I was the technical director and we were losing, I'd certainly be like "Lewis, pal, we lose as a team, right?" And then Lewis would be gone and we would just be losers.
  - AmericanChopper 7y ago
    There’s so many reasons for Mercedes success, that the parent comment comes of as being a bit silly. As you mentioned, they have Lewis Hamilton, who a lot of people would argue is the greatest driver of all time. They also have a very clear pecking order, team orders for Bottas to lose to Hamilton are a meme in F1 (“Valtteri, it’s James”). After the season was underway, they really only let them race after Hamilton had secured the drivers championship. Toto’s leadership is certainly very good (which the parent poster is referencing one particular part of), but you also have to remember, that Mercedes have only been dominant for one particular generation of F1 cars (the turbo-hybrid era). The sample size is absurdly small, and if you ask most commentators who’s going to dominate next, most of the would guess Verstappen (or LeClerc), not another Mercedes driver.
Mikhail_Edoshin 7y ago
There's a Youtube talk [Martial Arts Mind][1] by Peter Ralston about a change he underwent when he stopped seeing the opponent as an adversary. It's a very deep subject. Easy to get wrong too :) [1]: https://www.youtube.com/watch?v=wi7ph8GrRDk https://www.youtube.com/watch?v=wi7ph8GrRDk
notyourday 7y ago
Blameless postmortems should come with the equal pay ( as in equal dollar amount, equal value of equity, and equal value of other perks) for everyone in the tech organization. If it does not, then it is a propaganda tool embraced by people who extract more value from the people below them.
- diegoperini 7y ago
  I don't understand the argument. (I didn't vote on your comment)
  - notyourday 7y ago
    If it accepted that is not a person or persons whose mistakes create problems requiring postmortems meaning it was the collective that made mistakes then it should also be accepted that it is never a person or persons whose accomplishments propelled the company forward rather it is the collective. So everyone in the collective should get paid the amount of money for equal contributions to ups and downs, should not they?
    - diegoperini 7y ago
      On a scale from "individual" to "collective", both blame and success can be attributed to either extreme (or middle) on a case by case basis. I don't believe it can be possible to claim a universal generalization for all cases.
      - notyourday 7y ago
        Is there a single case where a company which practices blameless downside practices a collective upside?
        diegoperini 7y ago
        It is rare (so sad) but yes, there is. I work in such a company and I am not the employer.
codesections 7y ago
> We must strive to understand that accidents don’t happen because people gamble and lose. The problem is that some accidents do happen because people gamble and lose – or rather, because people cut corners is ways that probably won't matter but sometimes really, really do. The best approach needs to balance both causes of error. I agree that most organizations tend to err too far in the "blame someone" direction – but that doesn't mean that it's impossible to err by going too far in the other direction.
- di4na 7y ago
  They cut corner because the system tell them to do so. Not because they are fundamentally flawed into liking to gamble. I can make a perfect product with zero gamble, if i get infinite money and noone need to use it.
- hoorayimhelping 7y ago
  As I understand it the key to the blameless and just culture is to keep going to the bottom of it: "You cut corners. You were incentivized to cut corners. Why? Why do those incentives exist? What can we do to remove those incentives in the future?" The core of the philosophy is assuming people come to work to do a good job, and when they don't do a good job, there are reasonable explanations that involve people making locally rational decisions. The goal is to make the system stable in a way that making locally rational decisions translates to globally good decisions.
  - peteradio 7y ago
    And what happens when the finger starts pointing back towards business imposing terrible incentives?
    - jonfw 7y ago
      If you find a business incentive as a root cause, you blamelessly discuss ways you can change that incentive.
leggomylibro 7y ago
I once worked somewhere that did blameless technical postmortems very well. Whenever something broke, we kept asking questions until we had uncovered what went wrong, and we constantly reminded each other that our failings were in the process, not the people. It worked great, and we ended up with much more robust systems than most other places I'd worked. Sadly, they didn't apply the same ideals to social faux pas. We had an older gentleman working for us who, on hearing a younger member of the team propose an idea, said: "okay, I'll play the straight man: why won't this cause [...]" In the poor guy's generation, "straight man" was a stand-up comedy term for the person who acts deadpan and surprised while their partner makes with the comedy. But after explaining his actions and the expectations/assumptions behind them, he was reprimanded, ostracized in the office, and left before long. Even though I liked working on the technical systems at that job, I felt isolated and uncomfortable speaking with my peers after that. I've never been good with people, and what if I made a mistake? That sort of terror is what non-technical folks feel when you ask them to start taking on new responsibilities, and it's why these blameless systems are so important.
- the_af 7y ago
  "Straight man" still has that meaning today, just like "deadpan" and "foil". It's not a generational thing. It was probably a contextual thing; my guess is that nobody expected terms of comedy and theater at the workplace. But don't blame it on youth...
  - whatshisface 7y ago
    >my guess is that nobody expected terms of comedy and theater at the workplace That's a bit of a stretch, metaphors are the grease for the engine of the English language. In fact I've even heard this specific phrase many times before, recently. It's a metaphor in active use. The company was probably looking for a reason to get rid of him.
    - the_af 7y ago
      In fact I argued it's a metaphor in active use! But I also guessed metaphors taken out of context are likely to be misinterpreted. I know plenty of native English speakers who don't know what "straight man" means.
quartz 7y ago
I'm a cofounder at Kintaba (https://kintaba.com https://kintaba.com) where we spend a lot of time with companies that are implementing postmortems as part of their larger incident management process and it has been fascinating to see how varied the adoption of the practice is even in SV despite the value being well accepted for over a decade in tech (longer in other research circles). I often recommend anyone who is interested in the topic to check out Sidney Dekker's Field Guide to Understanding Human Error [1]. It's a very approachable read and goes into great detail about the underlying theories of safety research that support the value of blame-free cultures and postmortems and addresses common counter-arguments, particularly around the idea that lack of blame = lack of accountability. Also worth checking out the (free) google SRE Book chapters on Incident Management [2] and Postmortem Culture [3]. [1] https://www.amazon.com/Field-Guide-Understanding-Human-Error/dp/1472439058/ https://www.amazon.com/Field-Guide-Understanding-Human-Error... [2] https://landing.google.com/sre/sre-book/chapters/managing-incidents/ https://landing.google.com/sre/sre-book/chapters/managing-in... [3] https://landing.google.com/sre/sre-book/chapters/postmortem-culture/ https://landing.google.com/sre/sre-book/chapters/postmortem-...

xrd 7y ago

I love blameless postmortems. And, it isn't really whether the team or the team lead decides to do a blameless postmortem. It's whether the company values looking at things this way. And, those values change over time, and shift when the times are good or bad (financially). I would really love to read about a blameless postmortem that happened in a dying company where everyone was backstabbing each other. Now, that would be a good story! In fairness, this story does convey that. The "second story" part is great. I do wish there was a little more about how management came around to seeing it, how they rectified bonuses and pay and PIPs. There is an untold journey here.

pm90 7y ago
> I would really love to read about a blameless postmortem that happened in a dying company where everyone was backstabbing each other. Now, that would be a good story! It doesn't have to be a dying company. Many companies that try to transition from startup -> small company -> medium company -> megacorp tend to run into periods of change and insanity in every one of those transitions. I suspect its one of the reasons why most companies die out at some point: the failure to focus on product and focusing instead on politics. Teams turtle down, avoid responsibility (i.e. blame) at every opportunity, the product turns to shit as nobody wants to move a muscle in case anything breaks.
kqr 7y ago
I have been recommended the book "Turning the Ship Around" for an account of how to grow a new good culture in a decisively bad one. It's probably not an exact recipe for being able to hold blameless postmortems, but I suspect it could contain some valuable information. If anyone knows how well it applies to this situation, please do tell me!
- asplake 7y ago
  TBH I’m not sure of its relevance to postmortems specifically but I’d highly recommend the book. Turn the Ship Around, Marquet. Great audiobook too.
- di4na 7y ago
  TTSA is a great book and may help. But it is a leadership book. I still strongly advise it, at least to know nice things are possible.
di4na 7y ago
So this came from management. It is John Allspaw that pushed for them, at the time VP of Eng iirc. For how to do it... there is a community at https://www.learningfromincidents.io/ https://www.learningfromincidents.io/ trying to help. The consensus so far is to test waters and do some "in parallel", if necessary rogue-ish. The output is so superior in term of learning that you may then use this result as a trojan horse. You need to carry your water.
namdnay 7y ago
I think you can go too far in that direction. I worked for a big European software company, there was a deep culture of never blaming people, so any postmortem inevitably ended with a new process or training or transversal synergy coordinators, when sometimes the root cause was just bad developers
- GiuseppaAcciaio 7y ago
  Assuming that "bad developers" is indeed the root cause, that still sounds like a systemic issue to me: a) are the developers bad because they were hired that way? maybe the hiring process is broken b) are the developers bad because they are not managed/trained properly? maybe managers aren't doing their job, and maybe it's not even their fault because they did not get training in how to manage people in the first place c) if you implement extra layers (or "synergy coordinators") it probably means that the company hasn't figured out how to communicate properly
  - AmericanChopper 7y ago
    Management is ultimately responsible for everything. Even if you have the worst developer in the world on your team, management decided to hire them, retain them, assign them responsibility, and failed to improve their performance. That said, plenty of people are just self-serving dickheads. Could they be reformed? Probably. Is that the company’s responsibility? Only up to a point. I’ve worked with people that I was responsible for training in DevOps, from a starting point of having only ever done GUI-based Ops. Basically starting from complete technical incompetence, and plenty of them succeeded, because they were good people with good attitudes. I’d wager a small team of people like that would always be more productive than a team of any sized comprised of arrogant, combative, CYA assholes.

kqr 7y ago

Only tangentially related but I have to get it out of my system anyway: I watched the latest season of the F1 documentary on Netflix, and what stood out beyond anything else to me was that every team pitted their own drivers against each other, had a "you win or you lose" mentality, screamed at people for messing up, and so on. All teams except one. One team iterated how they "win as a team and make mistakes as a team", mistakes are an opportunity to learn, and so on. One team showed pretty clearly how they had a good culture surrounding failures. I'm not at all surprised that this one team is the team that has consistently won the last 6 years in a row. Why aren't more teams copying this? Why is this one team not more secretive about their recipe to success?

op03 7y ago
Both strategies are known to work - Carrot and Stick. Why? Because the psychological Traits and Needs of all humans are not the same. Most people who build teams have a sense of their own Needs and Traits and pick strategies/team members that align/compensate. https://en.wikipedia.org/wiki/Revised_NEO_Personality_Inventory#Personality_dimensions https://en.wikipedia.org/wiki/Revised_NEO_Personality_Invent... https://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs https://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs There is a distribution of Traits/Needs that you need to be aware of within any population, group, team etc. If you are not aware of those variations and randomly create groups and try to impose strategy, what works and fails will always remain mysterious.
MikeAmelung 7y ago
As the other reply kind of hints at, it's a lot easier to do this if you have the absolutely dominant driver on your team. If I was the technical director and we were losing, I'd certainly be like "Lewis, pal, we lose as a team, right?" And then Lewis would be gone and we would just be losers.
- AmericanChopper 7y ago
  There’s so many reasons for Mercedes success, that the parent comment comes of as being a bit silly. As you mentioned, they have Lewis Hamilton, who a lot of people would argue is the greatest driver of all time. They also have a very clear pecking order, team orders for Bottas to lose to Hamilton are a meme in F1 (“Valtteri, it’s James”). After the season was underway, they really only let them race after Hamilton had secured the drivers championship. Toto’s leadership is certainly very good (which the parent poster is referencing one particular part of), but you also have to remember, that Mercedes have only been dominant for one particular generation of F1 cars (the turbo-hybrid era). The sample size is absurdly small, and if you ask most commentators who’s going to dominate next, most of the would guess Verstappen (or LeClerc), not another Mercedes driver.

Mikhail_Edoshin 7y ago

There's a Youtube talk [Martial Arts Mind][1] by Peter Ralston about a change he underwent when he stopped seeing the opponent as an adversary. It's a very deep subject. Easy to get wrong too :) [1]: https://www.youtube.com/watch?v=wi7ph8GrRDk https://www.youtube.com/watch?v=wi7ph8GrRDk

notyourday 7y ago

Blameless postmortems should come with the equal pay ( as in equal dollar amount, equal value of equity, and equal value of other perks) for everyone in the tech organization. If it does not, then it is a propaganda tool embraced by people who extract more value from the people below them.

diegoperini 7y ago
I don't understand the argument. (I didn't vote on your comment)
- notyourday 7y ago
  If it accepted that is not a person or persons whose mistakes create problems requiring postmortems meaning it was the collective that made mistakes then it should also be accepted that it is never a person or persons whose accomplishments propelled the company forward rather it is the collective. So everyone in the collective should get paid the amount of money for equal contributions to ups and downs, should not they?
  - diegoperini 7y ago
    On a scale from "individual" to "collective", both blame and success can be attributed to either extreme (or middle) on a case by case basis. I don't believe it can be possible to claim a universal generalization for all cases.
    - notyourday 7y ago
      Is there a single case where a company which practices blameless downside practices a collective upside?
      - diegoperini 7y ago
        It is rare (so sad) but yes, there is. I work in such a company and I am not the employer.

codesections 7y ago

> We must strive to understand that accidents don’t happen because people gamble and lose. The problem is that some accidents do happen because people gamble and lose – or rather, because people cut corners is ways that probably won't matter but sometimes really, really do. The best approach needs to balance both causes of error. I agree that most organizations tend to err too far in the "blame someone" direction – but that doesn't mean that it's impossible to err by going too far in the other direction.

di4na 7y ago
They cut corner because the system tell them to do so. Not because they are fundamentally flawed into liking to gamble. I can make a perfect product with zero gamble, if i get infinite money and noone need to use it.
hoorayimhelping 7y ago
As I understand it the key to the blameless and just culture is to keep going to the bottom of it: "You cut corners. You were incentivized to cut corners. Why? Why do those incentives exist? What can we do to remove those incentives in the future?" The core of the philosophy is assuming people come to work to do a good job, and when they don't do a good job, there are reasonable explanations that involve people making locally rational decisions. The goal is to make the system stable in a way that making locally rational decisions translates to globally good decisions.
- peteradio 7y ago
  And what happens when the finger starts pointing back towards business imposing terrible incentives?
  - jonfw 7y ago
    If you find a business incentive as a root cause, you blamelessly discuss ways you can change that incentive.

leggomylibro 7y ago

I once worked somewhere that did blameless technical postmortems very well. Whenever something broke, we kept asking questions until we had uncovered what went wrong, and we constantly reminded each other that our failings were in the process, not the people. It worked great, and we ended up with much more robust systems than most other places I'd worked. Sadly, they didn't apply the same ideals to social faux pas. We had an older gentleman working for us who, on hearing a younger member of the team propose an idea, said: "okay, I'll play the straight man: why won't this cause [...]" In the poor guy's generation, "straight man" was a stand-up comedy term for the person who acts deadpan and surprised while their partner makes with the comedy. But after explaining his actions and the expectations/assumptions behind them, he was reprimanded, ostracized in the office, and left before long. Even though I liked working on the technical systems at that job, I felt isolated and uncomfortable speaking with my peers after that. I've never been good with people, and what if I made a mistake? That sort of terror is what non-technical folks feel when you ask them to start taking on new responsibilities, and it's why these blameless systems are so important.

the_af 7y ago
"Straight man" still has that meaning today, just like "deadpan" and "foil". It's not a generational thing. It was probably a contextual thing; my guess is that nobody expected terms of comedy and theater at the workplace. But don't blame it on youth...
- whatshisface 7y ago
  >my guess is that nobody expected terms of comedy and theater at the workplace That's a bit of a stretch, metaphors are the grease for the engine of the English language. In fact I've even heard this specific phrase many times before, recently. It's a metaphor in active use. The company was probably looking for a reason to get rid of him.
  - the_af 7y ago
    In fact I argued it's a metaphor in active use! But I also guessed metaphors taken out of context are likely to be misinterpreted. I know plenty of native English speakers who don't know what "straight man" means.

quartz 7y ago

I'm a cofounder at Kintaba (https://kintaba.com https://kintaba.com) where we spend a lot of time with companies that are implementing postmortems as part of their larger incident management process and it has been fascinating to see how varied the adoption of the practice is even in SV despite the value being well accepted for over a decade in tech (longer in other research circles). I often recommend anyone who is interested in the topic to check out Sidney Dekker's Field Guide to Understanding Human Error [1]. It's a very approachable read and goes into great detail about the underlying theories of safety research that support the value of blame-free cultures and postmortems and addresses common counter-arguments, particularly around the idea that lack of blame = lack of accountability. Also worth checking out the (free) google SRE Book chapters on Incident Management [2] and Postmortem Culture [3]. [1] https://www.amazon.com/Field-Guide-Understanding-Human-Error/dp/1472439058/ https://www.amazon.com/Field-Guide-Understanding-Human-Error... [2] https://landing.google.com/sre/sre-book/chapters/managing-incidents/ https://landing.google.com/sre/sre-book/chapters/managing-in... [3] https://landing.google.com/sre/sre-book/chapters/postmortem-culture/ https://landing.google.com/sre/sre-book/chapters/postmortem-...