3 ms·
Sometimes. And sometimes they make the same mistake over and over. We had an admin in charge of our storage. He had worked with our old vendor's SAN for years,
by greedo 6y ago
Sometimes. And sometimes they make the same mistake over and over.
We had an admin in charge of our storage. He had worked with our old vendor's SAN for years, then we got a new SAN. Trained him/certified him etc. He "accidentally" shut down the entire SAN. That brought down the entire company for over 9 hours.
Fast forward two years later, he screwed up again and caused a storage outage affecting about 1100 VMs. Luckily not much data loss, but a painful outage.
Then a month ago, he offlines part of the SAN.
Some people never learn, and recognizing this early is usually better than letting someone continue to risk things.
- mehrdadn 6y ago3 mistakes in... >2 years? I feel like it's really hard to tell if the problem is really the person at that point. Have you had others perform the same job for a similar duration to see if they avoid the same mistakes?
- greedo 6y agoThis is just the mistakes made in the SAN/Storage part of his responsibilities. As we used to say in World of Warcraft, "Can't heal stupid."
- nitrogen 6y agoIf you made a list of every mistake each person makes in 2-3 years, and omitted all other detail, pretty much everybody would look like a terrible person. Context, frequency, etc. all matter. If particular systems or people are seeing a high frequency of mistakes, maybe the system design is at fault, not just the person. Obviously it's hard to do in practice, but the ideal is to design systems that are mistake proof.
- jodrellblank 6y ago> "He had worked with our old vendor's SAN for years, then we got a new SAN." Great way to invalidate years of experience. Presumably from your telling of the story, he didn't cause problems with the old vendor's SAN? > "He "accidentally" shut down the entire SAN." So, was it an accident, or was it an "accident"? You can't have it being a mistake if you're also hinting it was deliberate and malicious.
- greedo 6y agoHe was trained and certified on the new SAN, and surely some of his prior experience on the legacy SAN would translate. Just as moving from AIX to RHEL/CentOS wouldn't invalidate all your skills and experience. It was a real accident when he shut down the SAN the first time. I don't know why I put it in scare quotes.