4 ms·

This isn't a post-mortem, it's a CYA report. A monthly standing meeting as a result of failure to communicate? Even if they had implemented the recommended pr

by pp19dd 8y ago

This isn't a post-mortem, it's a CYA report. A monthly standing meeting as a result of failure to communicate? Even if they had implemented the recommended procedure prior to the failure, this wouldn't have mattered one bit: "ensure that decisions not to install microcode bundle updates are documented and approved." That's not a fix. It wouldn't have affected the impending decision favorably, just made it documented and rubber-stamped. The approvals aren't done by a person with any technical knowledge, it's a standard-issue bureaucrat. Literally, "authorizing official" and "responsible directors" spelled out in the PDF consist of an actuary with an MBA and BA in economics, a single graduate in "Sceince" from '77 whose resume is littered with leadership platitudes and zero indication of hands-on work in technology, two-three underwriters, and the usual coterie of HR, procurement, inclusion professionals. This is how you get gridlocked bureaucracy and why the Federal retirement system is literally run out of a Pennsylvania cave. [1] https://www.washingtonpost.com/sf/national/2014/03/22/sinkhole-of-bureaucracy/ https://www.washingtonpost.com/sf/national/2014/03/22/sinkho...

jimktrains2 8y ago
> Even if they had implemented the recommended procedure prior to the failure, this wouldn't have mattered one bit: "ensure that decisions not to install microcode bundle updates are documented and approved." That's not a fix. It wouldn't have affected the impending decision favorably, just made it documented and rubber-stamped. I got the impression that it was more that the goal is a more in-depth discussion of the technical aspects of the updates and the risks associated with doing and not doing the update. "Formalize the monthly microcode bundle meetings with IBM and Unisys to include documenting meeting participants, detailed meeting minutes, and discussions of risks identified in the release notes for the current microcode updates." I'm not disagreeing with you about this being a CYA report, but at the same time, to me this feels like a problem with no solid solution. The only thing I would venture to say would have helped, based solely on the report, would have been notification of the issues the other IBM customer had in January to customers who had not upgraded; at the very least to disseminate what the condition looks like and the monitoring script much sooner.
- pp19dd 8y ago
  Just to distill that statement into specifics, the "goal" as you see it literally says (1) they had no written record before, and now they will and (2) two other vendors did the actual work. There's nothing technical about it but it implies intent behind the recommendations. IMO, that statement alone does illuminate a "solid solution", which is to reduce vendoring out some of these critical functions and gradually staff leadership with a mix of people who have significant hands-on experience with technology. Or more shortly, "care for properly."

by pp19dd 8y ago

jimktrains2 8y ago

> Even if they had implemented the recommended procedure prior to the failure, this wouldn't have mattered one bit: "ensure that decisions not to install microcode bundle updates are documented and approved." That's not a fix. It wouldn't have affected the impending decision favorably, just made it documented and rubber-stamped. I got the impression that it was more that the goal is a more in-depth discussion of the technical aspects of the updates and the risks associated with doing and not doing the update. "Formalize the monthly microcode bundle meetings with IBM and Unisys to include documenting meeting participants, detailed meeting minutes, and discussions of risks identified in the release notes for the current microcode updates." I'm not disagreeing with you about this being a CYA report, but at the same time, to me this feels like a problem with no solid solution. The only thing I would venture to say would have helped, based solely on the report, would have been notification of the issues the other IBM customer had in January to customers who had not upgraded; at the very least to disseminate what the condition looks like and the monitoring script much sooner.

pp19dd 8y ago
Just to distill that statement into specifics, the "goal" as you see it literally says (1) they had no written record before, and now they will and (2) two other vendors did the actual work. There's nothing technical about it but it implies intent behind the recommendations. IMO, that statement alone does illuminate a "solid solution", which is to reduce vendoring out some of these critical functions and gradually staff leadership with a mix of people who have significant hands-on experience with technology. Or more shortly, "care for properly."