4 ms·

Expanding on the post, one corollary of "people are rarely the cause of the outage" is that "you can't fix people". Given that Dave accidentally caused an outag

Expanding on the post, one corollary of "people are rarely the cause of the outage" is that "you can't fix people". Given that Dave accidentally caused an outage by pressing some button he should't have, explaining to him why he shouldn't press the button won't prevent anyone else from making the same mistake. Instead, you need to look at the reasons underlying why Dave pressed the button. Is the button confusingly labeled? Is there no playbook on what the process for pressing the button should be? If you fix those underlying problems, you will prevent other people from making the same mistake Dave made.

seangransee 7y ago
Dave actually wasn't the one who pushed the button. I was the button-pusher in this particular outage :) It sucks being the one who caused an outage, but having a culture where people put blame on the process rather than the person takes away a lot of the stress and naturally motivates people to solve the core issue as quickly as possible. The points in the post about psychological safety and underlying systemic causes of an outage are spot on.
closeparen 7y ago
If your response to every problem is "we need a process to make this idiot proof," pretty soon you are going to be up to your eyeballs in CYA training webinars and permission requests and committee reviews. Small groups of competent people who can be trusted to wield powerful tools will start to run circles around you. My employer does a reasonable job with this. Our tools usually make the path of least resistance the right thing. We have guardrails that help prevent accidents. But at the end of the day, our controls presume competence and good faith, and there's always an "I know what I'm doing" button. Actions taken after pressing it incur a bit more personal responsibility.
- jodrellblank 7y ago
  When parent poster says “look at the underlying reasons why”, you jump to untrustworthy idiot employees hampered by CYA red tape. When your employer does the same thing, it leads to “good tools with sensible affordances and guard rails” for competent responsible people. Why is that?
  - closeparen 7y ago
    It's a balance. "Harden the process" is good and necessary in moderation for the reasons already mentioned, but can be taken too far, to the point that the process is so rigid it's paralyzing. A finding of "the process was reasonable and the individual wasn't" or "a process guaranteed to stop this kind of action would be unreasonably heavy" may sometimes need to be on the table.
- geofft 7y ago
  Small groups of competent people (e.g., startups) can also usually do less damage. One quite reasonable approach is to give people small environments they can meaningfully work with freely, but require change control to go outside it. Also, a good postmortem process ought to treat "Why didn't we do this in time" as a problem to solve, too. (To be fair I don't immediately know of good examples of this.)
- charles_f 7y ago
  I'm not reading it this way, illustrated by: > explaining to him why he shouldn't press the button won't prevent anyone else from making the same mistake. How I interpret the point is: don't fix people, fix tool. You can't tell someone not to break the system, you need to make the system less easy to break, labels on buttons, confirmation dialogs, no double negatives, approvals if need be. Reminds me of a deployment with a customer (some 15y ago), I wasn't allowed to "touch prod" myself so I was sitting next to a button pusher doing what I was telling him to do (irony...). At some point there was some trickery involved to reboot HA instances one at a time rather than simultaneously, and you had to press "No" when prompted if you wanted to reboot. Before the dialog showed up, I pre-conditioned the guy, telling him "it's going to ask if you want to reboot, say no. Whatever you do, do not click on yes". This wasn't the first time he deployed. He said "oh, ok" then sure enough clicked on yes, and the IT police was in our room 2 minutes later because of a prod outage we just caused. I have thought many times of what could have prevented that - allowing me to deploy myself, doing more training to that guy, finding a competent replacement to him, more documentation. There is no process that would have worked better to fix the issue than finding a way to remove this dialog box - today that would be done through automation.
- tehlike 7y ago
  This is where netflix memo hits it right. When you burden creative people with processes, you take away the alpha in innovation... I like to think SWEs are in creative businesses... But this doesn't always hold true...
- ikiris 7y ago
  You don't seem to have learned the lesson at all. The whole point is CYA doesn't work. Its not trying to make it idiot proof, it's trying to remove the idiot from being involved at all.

seangransee 7y ago

Dave actually wasn't the one who pushed the button. I was the button-pusher in this particular outage :) It sucks being the one who caused an outage, but having a culture where people put blame on the process rather than the person takes away a lot of the stress and naturally motivates people to solve the core issue as quickly as possible. The points in the post about psychological safety and underlying systemic causes of an outage are spot on.

closeparen 7y ago

If your response to every problem is "we need a process to make this idiot proof," pretty soon you are going to be up to your eyeballs in CYA training webinars and permission requests and committee reviews. Small groups of competent people who can be trusted to wield powerful tools will start to run circles around you. My employer does a reasonable job with this. Our tools usually make the path of least resistance the right thing. We have guardrails that help prevent accidents. But at the end of the day, our controls presume competence and good faith, and there's always an "I know what I'm doing" button. Actions taken after pressing it incur a bit more personal responsibility.

jodrellblank 7y ago
When parent poster says “look at the underlying reasons why”, you jump to untrustworthy idiot employees hampered by CYA red tape. When your employer does the same thing, it leads to “good tools with sensible affordances and guard rails” for competent responsible people. Why is that?
- closeparen 7y ago
  It's a balance. "Harden the process" is good and necessary in moderation for the reasons already mentioned, but can be taken too far, to the point that the process is so rigid it's paralyzing. A finding of "the process was reasonable and the individual wasn't" or "a process guaranteed to stop this kind of action would be unreasonably heavy" may sometimes need to be on the table.
geofft 7y ago
Small groups of competent people (e.g., startups) can also usually do less damage. One quite reasonable approach is to give people small environments they can meaningfully work with freely, but require change control to go outside it. Also, a good postmortem process ought to treat "Why didn't we do this in time" as a problem to solve, too. (To be fair I don't immediately know of good examples of this.)
charles_f 7y ago
I'm not reading it this way, illustrated by: > explaining to him why he shouldn't press the button won't prevent anyone else from making the same mistake. How I interpret the point is: don't fix people, fix tool. You can't tell someone not to break the system, you need to make the system less easy to break, labels on buttons, confirmation dialogs, no double negatives, approvals if need be. Reminds me of a deployment with a customer (some 15y ago), I wasn't allowed to "touch prod" myself so I was sitting next to a button pusher doing what I was telling him to do (irony...). At some point there was some trickery involved to reboot HA instances one at a time rather than simultaneously, and you had to press "No" when prompted if you wanted to reboot. Before the dialog showed up, I pre-conditioned the guy, telling him "it's going to ask if you want to reboot, say no. Whatever you do, do not click on yes". This wasn't the first time he deployed. He said "oh, ok" then sure enough clicked on yes, and the IT police was in our room 2 minutes later because of a prod outage we just caused. I have thought many times of what could have prevented that - allowing me to deploy myself, doing more training to that guy, finding a competent replacement to him, more documentation. There is no process that would have worked better to fix the issue than finding a way to remove this dialog box - today that would be done through automation.
tehlike 7y ago
This is where netflix memo hits it right. When you burden creative people with processes, you take away the alpha in innovation... I like to think SWEs are in creative businesses... But this doesn't always hold true...
ikiris 7y ago
You don't seem to have learned the lesson at all. The whole point is CYA doesn't work. Its not trying to make it idiot proof, it's trying to remove the idiot from being involved at all.