3 ms·

> 4) perform a postmortem to identify the next steps It takes an effective management team to make this step work. If the postmortem isn't blameless, this stem

> 4) perform a postmortem to identify the next steps It takes an effective management team to make this step work. If the postmortem isn't blameless, this stem becomes completely ineffective as employees (not unreasonably) prioritize their own careers over the health of the company.

sidlls 8y ago
Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. I think there's a middle ground between never holding anyone accountable and the desperate situation you describe.
- nordsieck 8y ago
  > Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. It is true that some people are not good at their jobs. It's also true that the only way to discover that people are bad at their jobs is their performance. However, I think the process approach is applicable to a much larger domain than many people give it credit. One of the greatest software development quotes I've read comes from DJB [0]: > For many years I have been systematically identifying error-prone programming habits—by reviewing the literature, analyzing other people’s mistakes, and analyzing my own mistakes—and redesigning my programming environment to eliminate those habits. The difference in security bugs found in qmail and sendmail speaks for itself. This is an example of the superiority of process: it shows that what some people consider a "root cause" is actually only a proximate cause - that the root cause is the programming environment that allows the bugs to occur. [0] https://cr.yp.to/qmail/qmailsec-20071101.pdf https://cr.yp.to/qmail/qmailsec-20071101.pdf
  - peterwwillis 8y ago
    Yep. In many ways the "competency" of a person or group comes down to whether there was a designated person whose job it was to ensure they were following good processes. DJB does it himself, but most people don't do it for themselves.
- andrewaylett 8y ago
  In my experience, it's rare that any blame is sufficiently concentrated as to make assigning it sensible. A failure by a human is more commonly a failure in the process they were following, or in the supervision they should have received. And there's a big difference between "blame-free" and "unaccountable". It's not the fault of a team of developers if they aren't experienced enough to take responsibility for a critical production service, but the solution is not to berate them for being inexperienced: it's to give them the support they should (in hindsight) have had in the first place.
  - late2part 8y ago
    Sometimes that is translated as “a lack of training” which is often a euphemism for someone did something stupid and we have to put in place reactionary consequences (training on what not to do) if they or someone else does it again.
    - user5994461 8y ago
      Lack of experience is a nice word for incompetent. It remind me of an outage a few months ago. Lost a pair of systems, couldn't login and run anything anymore. Turns out, an operator ran a "rm -rf /home/user". He was following a procedure that includes cleaning a subdirectory. The subdirectory didn't exist so he figured out let's clean whatever top path exists!
    - andrewaylett 8y ago
      I typically push back on conclusions that only involve trying to change people and don't also put technical safeguards in place. There are definitely things you can't fully automate, but the goal should be that doing the right thing is really easy and making mistakes is hard.
- thwarted 8y ago
  There's "blame" and there's "fault". Faults are correctable, blame tries to place responsibility on actors. Few faults can be corrected without acknowledging where responsibility lies. There are multiple kinds of responsibility however. Responsibility for the cause and responsibility for fixing it. The same party who caused the issue isn't necessarily qualified (or authorized) to address it. The idea behind blameless postmortems is to be descriptive without calling out bad action, and evaluating the fault holistically, and acknowledging the reason most faults occur is a failure of processes not necessarily a failure of a person. Fixing processes is much more scalable.
- jasonlotito 8y ago
  > Sometimes a root cause is simply that a person or group was not competent in their execution of a task. If an incompetent person is able to take down a competent person's system, the "competent" person is to blame.
- yc-kraln 8y ago
  Sure, sometimes someone made a mistake. But they're not to blame; the process was set up that someone could make a mistake. Change the process so if/when a mistake happens, it doesn't cause a problem.
tonyedgecombe 8y ago
In my experience that's quite rare, what is likely to happen is any postmortem will be used as a political stick to beat someone with.

sidlls 8y ago

Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. I think there's a middle ground between never holding anyone accountable and the desperate situation you describe.

nordsieck 8y ago
> Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. It is true that some people are not good at their jobs. It's also true that the only way to discover that people are bad at their jobs is their performance. However, I think the process approach is applicable to a much larger domain than many people give it credit. One of the greatest software development quotes I've read comes from DJB [0]: > For many years I have been systematically identifying error-prone programming habits—by reviewing the literature, analyzing other people’s mistakes, and analyzing my own mistakes—and redesigning my programming environment to eliminate those habits. The difference in security bugs found in qmail and sendmail speaks for itself. This is an example of the superiority of process: it shows that what some people consider a "root cause" is actually only a proximate cause - that the root cause is the programming environment that allows the bugs to occur. [0] https://cr.yp.to/qmail/qmailsec-20071101.pdf https://cr.yp.to/qmail/qmailsec-20071101.pdf
- peterwwillis 8y ago
  Yep. In many ways the "competency" of a person or group comes down to whether there was a designated person whose job it was to ensure they were following good processes. DJB does it himself, but most people don't do it for themselves.
andrewaylett 8y ago
In my experience, it's rare that any blame is sufficiently concentrated as to make assigning it sensible. A failure by a human is more commonly a failure in the process they were following, or in the supervision they should have received. And there's a big difference between "blame-free" and "unaccountable". It's not the fault of a team of developers if they aren't experienced enough to take responsibility for a critical production service, but the solution is not to berate them for being inexperienced: it's to give them the support they should (in hindsight) have had in the first place.
- late2part 8y ago
  Sometimes that is translated as “a lack of training” which is often a euphemism for someone did something stupid and we have to put in place reactionary consequences (training on what not to do) if they or someone else does it again.
  - user5994461 8y ago
    Lack of experience is a nice word for incompetent. It remind me of an outage a few months ago. Lost a pair of systems, couldn't login and run anything anymore. Turns out, an operator ran a "rm -rf /home/user". He was following a procedure that includes cleaning a subdirectory. The subdirectory didn't exist so he figured out let's clean whatever top path exists!
  - andrewaylett 8y ago
    I typically push back on conclusions that only involve trying to change people and don't also put technical safeguards in place. There are definitely things you can't fully automate, but the goal should be that doing the right thing is really easy and making mistakes is hard.
thwarted 8y ago
There's "blame" and there's "fault". Faults are correctable, blame tries to place responsibility on actors. Few faults can be corrected without acknowledging where responsibility lies. There are multiple kinds of responsibility however. Responsibility for the cause and responsibility for fixing it. The same party who caused the issue isn't necessarily qualified (or authorized) to address it. The idea behind blameless postmortems is to be descriptive without calling out bad action, and evaluating the fault holistically, and acknowledging the reason most faults occur is a failure of processes not necessarily a failure of a person. Fixing processes is much more scalable.
jasonlotito 8y ago
> Sometimes a root cause is simply that a person or group was not competent in their execution of a task. If an incompetent person is able to take down a competent person's system, the "competent" person is to blame.
yc-kraln 8y ago
Sure, sometimes someone made a mistake. But they're not to blame; the process was set up that someone could make a mistake. Change the process so if/when a mistake happens, it doesn't cause a problem.

tonyedgecombe 8y ago

In my experience that's quite rare, what is likely to happen is any postmortem will be used as a political stick to beat someone with.