3 ms·
Kudos on this. Coming off of the Symantec thing, I think we're all viewing businesses in the CA space a lot more skeptically. I worked in a hybrid Infrastruct
by mdip 9y ago
Kudos on this. Coming off of the Symantec thing, I think we're all viewing businesses in the CA space a lot more skeptically.
I worked in a hybrid Infrastructure / Development job for almost 17 years. One of the reasons I could never shake the infrastructure part of things[0] was that I knew the environment from more angles than most of the staff and I tended to be key in high severity issues. We were a highly geographically distributed team and coordinated almost all activity via conference calls.
We had two major rules when a critical failure was discovered. The first was that anyone asking "who caused this" was admonished and often ejected from the call. Often, the person responsible for the failure was on the calls and everyone had a pretty good understanding of who was at fault (if it was, indeed, a self-inflicted wound). The person at fault was also usually critical to the fix and putting them on the defensive caused evasiveness out of self-preservation. To help the first rule, the second rule was barring of anyone at the Director level or higher on these calls[1]. Generally, they were concerned with two things: (1) When's it going to work again and (2) why did it fail in the first place -- both result in delays. They were given updates, separately, by someone from the call at regular intervals. Keeping the politics low and focusing on the problem meant much shorter outages and calmer, more focused, brains of our technical staff.
And to your point, despite working at a company that laid off 10% or so of it's staff about every 9 months, I can't think of a case where a member of technical staff was specifically let go because of an (infrequent) failure -- even situations that would fall under gross negligence of the technician were forgiven provided they were one-off. People get distracted and fail to follow testing procedures/forget to read architecture documentation and make mistakes[2].
After the situation is resolved, a post-mortem/"root cause analysis" was always performed. It was often done via documents rather than meetings, to avoid the confrontation associated with a bunch of managers pointing fingers, and we always learned from our mistakes and got better as a result.
[0] I was a developer of infrastructure services, doing similar work that a DevOps might do, today but more focused on providing internal services and automating support. I was a full-time developer for 3/4 of that time with regular interruptions of sev0 incident response.
[1] This didn't always happen, but I can count on one hand the number of times when a director/VP/C-Level joined and actually made his/her presence known.
[2] I'll never forget the time early in my career when I was on-call and had to restore our most critical/largest file server that was destroyed as a result of a technician getting frustrated with a rack and dislodging a 60(?)-pin SCSI cable...partly...where it attempted to run for several hours in that condition. The best part was the maintenance he was doing was on the backup server that had required his attention for a few weeks and hadn't functioned since then. When I saw the condition of the cables, I was so pissed off that while restoring the weeks' old backups at 4:00 AM, I had the operators in that room show me how to review the (VHS) security tapes. Watching him kick the door several times to get it to latch, then peering in -- directly at the bent-up cable -- just before walking out the door was the best part. He should have been fired for that, and eventually a similar behavior had that result.