4 ms·
No real insights here, just high level explanation and empty quotes from folks. > The root cause analysis (RCA) means that a CrowdStrike programmer(s) did not
by duxup 2y ago
No real insights here, just high level explanation and empty quotes from folks.
> The root cause analysis (RCA) means that a CrowdStrike programmer(s) did not check their inputs before pushing an update to the CrowdStrike Falcon Windows Sensor in production.
How is it this isn’t just automated and / or that update automatically run in a VM or something and when it crashed the rollout prevented?
It’s not a new concept…
- moffkalast 2y agoThese sort of things happen occasionally despite all the possible safeguards you can set up. It's just gonna be something else you didn't predict and frankly the wrong thing to focus on. The real problem this exposed is that somehow it's apparently the law for corporations to introduce a single point of failure into their entire IT system with absolutely zero fallback capacity or any workarounds at all. It's just entropy meeting stupidity. It's like it was mandated for planes to have a built in off switch that can be triggered remotely and people then blame the company for accidentally triggering it, crashing every single one and killing everyone, instead of asking why the fuck do all planes have an off switch?!
- duxup 2y agoMy understanding is this update crashed very consistently.
- Jtsummers 2y agoYep. Every Windows machine in our company that was online during the update period was brought down. The only ones to survive were offline for some reason (off, sleep mode, people traveling with them, on an air gapped network, etc.).
- moffkalast 2y agoYeah it probably could've been caught. Major disasters are always this way, someone does something they shouldn't have, two people that should've checked were on vacation and the third one was asleep, and a manager saying let's goooo because we're in a hurry and way behind! Come on, just skip the checklist this once, it'll be fine. It's always a perfect storm and it will always happen, we can just reduce the frequency with more and more safeguards. If something is so critical that people will die if it doesn't work, it needs backups and probably secondary backups, end of story. Hospitals that were brought down with this have no excuse.