4 ms·

Firing the person who happened to be at the wheel when a mistake like this occurs never seems like the right choice to me, especially if their performance to-da

Firing the person who happened to be at the wheel when a mistake like this occurs never seems like the right choice to me, especially if their performance to-date had otherwise been good. Everybody has off days, or just instances where circumstances misalign in just the wrong way. To pretend otherwise is silly; instead, it's the leader's/team's responsibility to ensure that those sort of off days don't lead to massive losses via redundancy & the sort of measures we're talking about here & in the OP. Firing somebody in these circumstances just acts to severely reduce morale, since we all secretly know in our hearts that it very easily could have been us. Firing in this case just seems retributive. It's not going to bring the lost data back, and you've just eliminated the very person who could have told you most about the chain of events leading to the incident in question to help you guard against it in the future. These incidents usually sound simple at the surface level ("I clicked the button in the wrong window") but often hint at deeper, perhaps even organizational, issues. A lack of team focus on reliability/quality, a lack of communication or trust about decisions made (or not made) by higher ups, or so on. And they are probably the single least likely person to cause a similar incident again -- that person will now likely be double and triple checking their commands for eternity.

jacobsenscott 6y ago
Agree. There is never a single cause to this kind of error. It takes a village. Someone didn't name things properly, someone else didn't store backups properly, someone else gave everyone root access to production, etc. It was inevitable the database would be deleted - doesn't matter who actually did it. If your CTO scattered those landmines all over then "not stepping right" is not an error. It just sucks.
greedo 6y ago
Sometimes. And sometimes they make the same mistake over and over. We had an admin in charge of our storage. He had worked with our old vendor's SAN for years, then we got a new SAN. Trained him/certified him etc. He "accidentally" shut down the entire SAN. That brought down the entire company for over 9 hours. Fast forward two years later, he screwed up again and caused a storage outage affecting about 1100 VMs. Luckily not much data loss, but a painful outage. Then a month ago, he offlines part of the SAN. Some people never learn, and recognizing this early is usually better than letting someone continue to risk things.
- mehrdadn 6y ago
  3 mistakes in... >2 years? I feel like it's really hard to tell if the problem is really the person at that point. Have you had others perform the same job for a similar duration to see if they avoid the same mistakes?
  - greedo 6y ago
    This is just the mistakes made in the SAN/Storage part of his responsibilities. As we used to say in World of Warcraft, "Can't heal stupid."
  - nitrogen 6y ago
    If you made a list of every mistake each person makes in 2-3 years, and omitted all other detail, pretty much everybody would look like a terrible person. Context, frequency, etc. all matter. If particular systems or people are seeing a high frequency of mistakes, maybe the system design is at fault, not just the person. Obviously it's hard to do in practice, but the ideal is to design systems that are mistake proof.
- jodrellblank 6y ago
  > "He had worked with our old vendor's SAN for years, then we got a new SAN." Great way to invalidate years of experience. Presumably from your telling of the story, he didn't cause problems with the old vendor's SAN? > "He "accidentally" shut down the entire SAN." So, was it an accident, or was it an "accident"? You can't have it being a mistake if you're also hinting it was deliberate and malicious.
  - greedo 6y ago
    He was trained and certified on the new SAN, and surely some of his prior experience on the legacy SAN would translate. Just as moving from AIX to RHEL/CentOS wouldn't invalidate all your skills and experience. It was a real accident when he shut down the SAN the first time. I don't know why I put it in scare quotes.
Lex-2008 6y ago
> These incidents usually sound simple at the surface level ("I clicked the button in the wrong window") but often hint at deeper, perhaps even organizational, issues. These words reminded me a story of similar/different "flaps" and "landing gear" controls on a plane - where crashed airplanes were also blamed on pilots first, before a trivial engineering/UI solution was implemented: https://www.endsight.net/blog/what-the-wwii-b17-bomber-can-teach-us-about-quality-control https://www.endsight.net/blog/what-the-wwii-b17-bomber-can-t...
_asummers 6y ago
There's a really good book describing this phenomenon called Behind Human Error. It speaks of "first stories" and "second stories" and how in analysis of incidents, it is all too common to stop at the first story and chalk it up to human error, when the system itself allowed it to take place.
Huggernaut 6y ago
Nickolas Means has an absolutely wonderful set of talks on themes like this. Particularly relevant here I think, is his talk: "Who Destroyed Three Mile Island?" - which goes through the events that occurred at the nuclear power plant, the systemic problems, and how to find the "second stories" of why failures occurred. https://www.youtube.com/watch?v=1xQeXOz0Ncs https://www.youtube.com/watch?v=1xQeXOz0Ncs

jacobsenscott 6y ago

Agree. There is never a single cause to this kind of error. It takes a village. Someone didn't name things properly, someone else didn't store backups properly, someone else gave everyone root access to production, etc. It was inevitable the database would be deleted - doesn't matter who actually did it. If your CTO scattered those landmines all over then "not stepping right" is not an error. It just sucks.

greedo 6y ago

Sometimes. And sometimes they make the same mistake over and over. We had an admin in charge of our storage. He had worked with our old vendor's SAN for years, then we got a new SAN. Trained him/certified him etc. He "accidentally" shut down the entire SAN. That brought down the entire company for over 9 hours. Fast forward two years later, he screwed up again and caused a storage outage affecting about 1100 VMs. Luckily not much data loss, but a painful outage. Then a month ago, he offlines part of the SAN. Some people never learn, and recognizing this early is usually better than letting someone continue to risk things.

mehrdadn 6y ago
3 mistakes in... >2 years? I feel like it's really hard to tell if the problem is really the person at that point. Have you had others perform the same job for a similar duration to see if they avoid the same mistakes?
- greedo 6y ago
  This is just the mistakes made in the SAN/Storage part of his responsibilities. As we used to say in World of Warcraft, "Can't heal stupid."
- nitrogen 6y ago
  If you made a list of every mistake each person makes in 2-3 years, and omitted all other detail, pretty much everybody would look like a terrible person. Context, frequency, etc. all matter. If particular systems or people are seeing a high frequency of mistakes, maybe the system design is at fault, not just the person. Obviously it's hard to do in practice, but the ideal is to design systems that are mistake proof.
jodrellblank 6y ago
> "He had worked with our old vendor's SAN for years, then we got a new SAN." Great way to invalidate years of experience. Presumably from your telling of the story, he didn't cause problems with the old vendor's SAN? > "He "accidentally" shut down the entire SAN." So, was it an accident, or was it an "accident"? You can't have it being a mistake if you're also hinting it was deliberate and malicious.
- greedo 6y ago
  He was trained and certified on the new SAN, and surely some of his prior experience on the legacy SAN would translate. Just as moving from AIX to RHEL/CentOS wouldn't invalidate all your skills and experience. It was a real accident when he shut down the SAN the first time. I don't know why I put it in scare quotes.

Lex-2008 6y ago

> These incidents usually sound simple at the surface level ("I clicked the button in the wrong window") but often hint at deeper, perhaps even organizational, issues. These words reminded me a story of similar/different "flaps" and "landing gear" controls on a plane - where crashed airplanes were also blamed on pilots first, before a trivial engineering/UI solution was implemented: https://www.endsight.net/blog/what-the-wwii-b17-bomber-can-teach-us-about-quality-control https://www.endsight.net/blog/what-the-wwii-b17-bomber-can-t...

_asummers 6y ago

There's a really good book describing this phenomenon called Behind Human Error. It speaks of "first stories" and "second stories" and how in analysis of incidents, it is all too common to stop at the first story and chalk it up to human error, when the system itself allowed it to take place.

Huggernaut 6y ago

Nickolas Means has an absolutely wonderful set of talks on themes like this. Particularly relevant here I think, is his talk: "Who Destroyed Three Mile Island?" - which goes through the events that occurred at the nuclear power plant, the systemic problems, and how to find the "second stories" of why failures occurred. https://www.youtube.com/watch?v=1xQeXOz0Ncs https://www.youtube.com/watch?v=1xQeXOz0Ncs