5 ms·

One tradition I really appreciate at Google is the post-mortem. Any serious outage or bug will be followed by a detailed post-mortem detailing a timeline of wha

by cletus 14y ago

One tradition I really appreciate at Google is the post-mortem. Any serious outage or bug will be followed by a detailed post-mortem detailing a timeline of what went wrong and why. Sometimes this will go as far as establishing a "war room", possibly for weeks, dragging in people from a number of different areas to address a particular issue. It's not about pointing the finger or otherwise apportioning blame. It's about learning from mistakes and preventing them from happening again. That being said, I've worked at other places where shame is attempted to be used in such situations (which is what this is). In my experience it just fosters a hostile environment.

swah 14y ago
Interesting. That reminded me of this book "Start-up Nation: The Story of Israel's Economic Miracle" which says their military runs the same procedure in a similar way after missions.
- Nimi 14y ago
  As an Israeli, I can vouch for the massive ass-covering that goes on there. In fact, we just had a big public ass-covering demonstration regarding the awful flotilla blockade. Regardless of what one thinks of the political aspects and/or morality of that action, the purpose was to examine why were several soldiers injured and Israel's relationship with Turkey, previously a strategic ally, deteriorated. Both these results were seen as failures of the mission. The ass covering was plain for all to see, on the national news. I agree that the Israeli army has a very substantial contribution to the high level of entrepreneurship and technological achievement in the country, but effective organizational behavior is definitely not something you want to pick up from the IDF.
  - anamax 14y ago
    > effective organizational behavior is definitely not something you want to pick up from the IDF. (1) The IDF is "ineffective" on an absolute scale. (2) The IDF is significantly more effective than most other organizations. While better isn't perfect, you can learn from both.
    - Nimi 14y ago
      Why do you claim that the IDF is more effective than other organizations?
      - ars 14y ago
        http://www.globalfirepower.com/ http://www.globalfirepower.com/ Compare that list to the population size of each country, Israel is a very large outlier. National will is important, but you don't rank on that list by being ineffective.
      - anamax 14y ago
        > Why do you claim that the IDF is more effective than other organizations? I didn't claim that the IDF was more effective than all organizations, I said that it was more effective than most. For example, I'd claim that the IDF is more effective than almost every failed startup.... (You can argue that many fail for reasons outside their control, but I'll respond that such vulnerabilities are a failure.)
    - Retric 14y ago
      It's also a lot easier for large organizations to examine issues of a less sensitive nature. AKA missile detonates in the middle of a field vs school.
- run4yourlives 14y ago
  Almost all armies do this, and they do it really, really well. As a veteran I can tell you that everything you do is critiqued, and everything your boss does is critiqued, and so on. At certain levels it's peer review, at others, hierarchtical. The whole point of it all is to increase your chances of survival. It's a great process, and I've noticed that most military folk are great at speaking plainly and accepting critique, much more so than civilians. I beleive these processes are a huge part of the reason.
wccrawford 14y ago
A good programmer will feel shame from that whole process even if nobody points the finger at them. The blame game just isn't needed. A bad programmer shouldn't work there in the first place, and if they need to be blamed, they're probably doing other things to destroy morale.
- eternalban 14y ago
  s/programmer/worker
- zura 14y ago
  If we stay in common sense ranges, a bug is not something you have to shame or proud. It is just part of the working process.
- damoncali 14y ago
  I find it interesting that you'd feel shame over a bug. Why?
  - Devilboy 14y ago
    This is a very well-known phenomena. If you don't feel anything after a bug in your code cost your company lots of money and your team members many days in debugging and fixing it... then you're like nobody I've ever worked with.
    - jaylevitt 14y ago
      Really? My first major software release took the AOL mail system down for 34 hours. My later work kept it up for 8 years. I consider both equal fodder for bragging, self-immolation, and war stories over drinks. I couldn't tell you which taught me more.
      - PerryCox 14y ago
        I think what the previous commenters are trying to say is that you shouldn't be OK with producing lots of bugs. Sure, you can learn a lot when you make a mistake, but you shouldn't be making the same mistakes.
Diederich 14y ago
My company, LiveOps, does the same thing. Every customer impacting issue gets a post-mortem, and some non-customer impacting problems get one. We have good stability (we hit four 9s recently), but we're always pushing for me. The post-mortem process is very positive and affirming. That's because it's transparent, as it should be.
quux 14y ago
At a prior job I was a key member of a team that did a 2 week long, intense root cause analysis of a prod issue that threatened a multi million dollar contract. I consider it one of the most interesting and rewarding things I've done in my career. All obstacles were removed, we could follow the clues wherever they took us, use any resources etc. In the end it took careful analysis, many simulations and hard work to find and fix the root cause, and no fingers were pointed, instead we learned some good lessons and were better for it. When done right, RCA is a great thing.
lhnn 14y ago
I don't know... maybe it's where I work, but the post-mortems I've been to all too often mandate some useless extra layer of review or process on top of an already byzantine change control scheme. Sometimes, "shit happens", and I wish our availability teams appreciated that. And even if there is justification for additional process, they should let the ops/dev teams come up with ideas/solutions themselves: we know our tools and teams best.
quantumhobbit 14y ago
That sort of sounds like how Morbidity and Mortality conferences at hospitals are supposed to work. It isn't about assigning blame(unless someone really screwed up) but examining the mistake in front of the whole department so that everybody learns from it.
- roguecoder 14y ago
  Right. Ideally, everyone is willing to take responsibility for the work they do and are proud of it even when it contains flaws. If you can't handle having bugs in your code without getting all defensive, you probably shouldn't be coding. The only way to get better is to get feedback.

by cletus 14y ago

swah 14y ago

Interesting. That reminded me of this book "Start-up Nation: The Story of Israel's Economic Miracle" which says their military runs the same procedure in a similar way after missions.

Nimi 14y ago
As an Israeli, I can vouch for the massive ass-covering that goes on there. In fact, we just had a big public ass-covering demonstration regarding the awful flotilla blockade. Regardless of what one thinks of the political aspects and/or morality of that action, the purpose was to examine why were several soldiers injured and Israel's relationship with Turkey, previously a strategic ally, deteriorated. Both these results were seen as failures of the mission. The ass covering was plain for all to see, on the national news. I agree that the Israeli army has a very substantial contribution to the high level of entrepreneurship and technological achievement in the country, but effective organizational behavior is definitely not something you want to pick up from the IDF.
- anamax 14y ago
  > effective organizational behavior is definitely not something you want to pick up from the IDF. (1) The IDF is "ineffective" on an absolute scale. (2) The IDF is significantly more effective than most other organizations. While better isn't perfect, you can learn from both.
  - Nimi 14y ago
    Why do you claim that the IDF is more effective than other organizations?
    - ars 14y ago
      http://www.globalfirepower.com/ http://www.globalfirepower.com/ Compare that list to the population size of each country, Israel is a very large outlier. National will is important, but you don't rank on that list by being ineffective.
    - anamax 14y ago
      > Why do you claim that the IDF is more effective than other organizations? I didn't claim that the IDF was more effective than all organizations, I said that it was more effective than most. For example, I'd claim that the IDF is more effective than almost every failed startup.... (You can argue that many fail for reasons outside their control, but I'll respond that such vulnerabilities are a failure.)
  - Retric 14y ago
    It's also a lot easier for large organizations to examine issues of a less sensitive nature. AKA missile detonates in the middle of a field vs school.
run4yourlives 14y ago
Almost all armies do this, and they do it really, really well. As a veteran I can tell you that everything you do is critiqued, and everything your boss does is critiqued, and so on. At certain levels it's peer review, at others, hierarchtical. The whole point of it all is to increase your chances of survival. It's a great process, and I've noticed that most military folk are great at speaking plainly and accepting critique, much more so than civilians. I beleive these processes are a huge part of the reason.

wccrawford 14y ago

A good programmer will feel shame from that whole process even if nobody points the finger at them. The blame game just isn't needed. A bad programmer shouldn't work there in the first place, and if they need to be blamed, they're probably doing other things to destroy morale.

eternalban 14y ago
s/programmer/worker
zura 14y ago
If we stay in common sense ranges, a bug is not something you have to shame or proud. It is just part of the working process.
damoncali 14y ago
I find it interesting that you'd feel shame over a bug. Why?
- Devilboy 14y ago
  This is a very well-known phenomena. If you don't feel anything after a bug in your code cost your company lots of money and your team members many days in debugging and fixing it... then you're like nobody I've ever worked with.
  - jaylevitt 14y ago
    Really? My first major software release took the AOL mail system down for 34 hours. My later work kept it up for 8 years. I consider both equal fodder for bragging, self-immolation, and war stories over drinks. I couldn't tell you which taught me more.
    - PerryCox 14y ago
      I think what the previous commenters are trying to say is that you shouldn't be OK with producing lots of bugs. Sure, you can learn a lot when you make a mistake, but you shouldn't be making the same mistakes.

Diederich 14y ago

My company, LiveOps, does the same thing. Every customer impacting issue gets a post-mortem, and some non-customer impacting problems get one. We have good stability (we hit four 9s recently), but we're always pushing for me. The post-mortem process is very positive and affirming. That's because it's transparent, as it should be.

quux 14y ago

At a prior job I was a key member of a team that did a 2 week long, intense root cause analysis of a prod issue that threatened a multi million dollar contract. I consider it one of the most interesting and rewarding things I've done in my career. All obstacles were removed, we could follow the clues wherever they took us, use any resources etc. In the end it took careful analysis, many simulations and hard work to find and fix the root cause, and no fingers were pointed, instead we learned some good lessons and were better for it. When done right, RCA is a great thing.

lhnn 14y ago

I don't know... maybe it's where I work, but the post-mortems I've been to all too often mandate some useless extra layer of review or process on top of an already byzantine change control scheme. Sometimes, "shit happens", and I wish our availability teams appreciated that. And even if there is justification for additional process, they should let the ops/dev teams come up with ideas/solutions themselves: we know our tools and teams best.

quantumhobbit 14y ago

That sort of sounds like how Morbidity and Mortality conferences at hospitals are supposed to work. It isn't about assigning blame(unless someone really screwed up) but examining the mistake in front of the whole department so that everybody learns from it.

roguecoder 14y ago
Right. Ideally, everyone is willing to take responsibility for the work they do and are proud of it even when it contains flaws. If you can't handle having bugs in your code without getting all defensive, you probably shouldn't be coding. The only way to get better is to get feedback.