7 ms·
> without even the most basic level of qualification That was my first thought too. Our company does firmware updates to hundreds of thousands of devices every
by bonestamp2 2y ago
> without even the most basic level of qualification
That was my first thought too. Our company does firmware updates to hundreds of thousands of devices every month and those updates always go through 3 rounds of internal testing, then to a couple dozen real world users who we have a close relationship with (and we supply them with spare hardware that is not on the early update path in case there is a problem with an early rollout). Then the update goes to a small subset of users who opt in to those updates, then they get rolled out in batches to the regular users in case we still somehow missed something along the way. Nothing has ever gotten past our two dozen real world users.
- rvnx 2y agoOr it could be made that Windows stops loading drivers that are crashing. Third-party driver/module crashed more than 3 times in a row -> Third-party driver/module is punished and has to be manually re-enabled.
- Lx1oG-AWb6h_ZG0 2y agoWouldn't this be an attack vector? Use some low-hanging bug to bring down an entire security module, allowing you to escalate?
- SAI_Peregrinus 2y agoIt's currently a DOS by the crashing component, so it's already broken the Availability part of Confidentiality/Integrity/Availability that defines the goals of security.
- hunter2_ 2y agoBut a loss of availability is so much more palatable than the others, plus the others often result in manually restricting availability anyway when discovered.
- jamie0 2y agoI think the wider societal impact from the loss of availability today - particularly for those in healthcare settings - might suggest this isn't always the case
- prng2021 2y agoAvailability of a system that can’t ensure data integrity seems equally bad though.
- azinman2 2y agoTell that to the millions of people whose flights were canceled, the surgeries not performed, etc etc.
- josephg 2y agoOr anyone who owns CrowdStrike shares.
- prng2021 2y agoWhat is the importance of data integrity? If important pre-op data/instructions are missing or gets saved on the wrong patient record which causes botched surgeries, if there are misprescribed post-op medications, if there is huge confusion and delays in critical follow-up surgeries because of a 100% available system that messed up patient data across hospitals nationwide, if there are malpractice lawsuits putting entire hospitals out of business etc etc, then is that fallout clearly worth having an available system in the first place?
- sudosysgen 2y agoIf you're planning around bugs in security modules, you're better off disabling them - malware routinely use bugs in drivers to escalate, so the bug you're allowing can make the escalation vector even more powerful as now it gets to Ring 0 early loading.
- cyanydeez 2y agoRequires state level social engineering. Might by why north Koreans are trying to get work from home jobs. https://www.businessinsider.com/woman-helped-north-korea-find-remote-jobs-in-us-prosecutors-2024-5?op=1 https://www.businessinsider.com/woman-helped-north-korea-fin...
- tomxor 2y ago> Wouldn't this be an attack vector? Isn't DoSing your own OS an attack vector? and a worse one when it's used in critical infrastructure where lives are at stake. There is a reasonable balance to strike, sometimes it's not a good idea to go to extreme measures to prevent unlikely intrusion vectors due to the non-monetary costs. See: The optimal amount of fraud is non-zero.
- Thorrez 2y agoIn the absence of a Crowdstrike bug, if an attacker is able to cause Crowdstrike to trigger a bluescreen, I assume the attacker would be able to trigger a bluescreen in some other way. So I don't think this is a good argument for removing the check.
- tomxor 2y agoThat assumes it's more likely than crowdstrike mass bricking all of these computers... this is the balance, it's not about possibility, it's about probability.
- Thorrez 2y agoI think we're in agreement. I now realize my previous comment replied to the wrong comment. I meant to reply to Lx1oG-AWb6h_ZG0. Sorry.
- tatersolid 2y agoBecause CrowdStrike is an EDR solution it likely has tamper-proofing features (scheduled tasks, watchdog services, etc.) that re-enables it. These features are designed to prevent malware or manual attackers from disabling it.
- Wheaties466 2y agoit does. several crowdstrike alerts popped when i was remediating systems of the broken driver.
- rkagerer 2y agoThese features drive me nuts because they prevent me, the computer owner/admin, from disabling. One person thought up techniques like "let's make a scheduled task that sledgehammers out the knobs these 'dumb' users keep turning' and then everyone else decided to copycat that awful practice.
- Thorrez 2y agoIf you're the admin, I would assume you have the ability to disable Crowdstrike. There must be some way to uninstall it, right?
- xena 2y agoNot if you want to keep the magic green compliance checkbox!
- Thorrez 2y agoAre you saying that the compliance rule requires the software to be uninstallable? Once it's installed it's impossible to uninstall? No one can uninstall it? I have a hard time believing it's impossible to remove the software. In the extreme case, you could reimage the machine and reinstall Windows without Crowdstrike. Or are you saying that it is possible to uninstall, but once you do that, you're not in compliance, so while it's technically possible to uninstall, you'll be breaking the rules if you do so?
- commandersaki 2y agoI use Explorer Patcher on a windows 11 machine. It had a history of crash loops with Explorer that they implemented this circuit breaker functionality.
- fortran77 2y agoIt does. CrowdStrike forced itself into boot process. Normal windows drivers will be disable automatically if they caused a crash
- yibg 2y agoThis was my first thought too. I'm not that familiar with the space, but I would think for something this sensitive the rollout would be staggered at least instead of what looks like globally all at the same time.
- dudeism_est_03 2y agoThis is the bit I am still trying to understand. On CrowdStrike you can define how many updates a host is behind. I.e. n (latest), n-1 (one behind) or n-2 etc. This update was applied to a 'latest' policy hosts and the n-2 hosts. To me it appears that there was more to this than just a corrupt update, otherwise how was this policy ignored? Unless it doesn't separate the update as deeply and maybe just a small policy aspect, which would also be very concerning. I guess we won't really know until they release the post mortem...
- Xamayon 2y agoYeah, my guess is that they roll out the updates to every client at the same time, and then have the client implement the n-1/2/whatever part locally. That worked great-ish until they pushed a corrupt (empty) update file which crashed the client when it tried to interpret the contents... Not ideal, and obviously there isn't enough internal testing before sending stuff out to actual clients.
- mihaaly 2y agoExactly this what I was missing in the story. Like why not to have a limited set of users have it before going live for the whole user base at a mission critical product like this is beyond comprehension of everyone ever came across software bugs (so billions of people). And then we already overcame the part of not testing internally well, or at all? Something clusteruck must have happened there which is still better than imagining that this is the normal way the organization operates. Which is a very scary vision. Serious rethinking of trusting this organization is due everywhere!
- nikau 2y agoBut that would require hiring staff to manage the process, and that is money taken away from sponsoring an F1 racing team.
- Rinzler89 2y agoThe funniest part was seeing Mercedes F1 team pit crew staring at BSODs at their workstations[1] while wearing CrowdStrike t-shirts. Some jokes just write themselves. Imagine if they loose the race because of their sponsor. But hey, at least they actually dogfood the products of their sponsors instead of just taking money to shill random stuff. [1] https://www.thedrive.com/news/crowdstrike-sponsored-mercedes-f1-has-recovered-from-the-blue-screens-of-death https://www.thedrive.com/news/crowdstrike-sponsored-mercedes...
- cush 2y agoIt's baffling how fast and wide the blast radius was for this Crowdstrike update. Quite impressive actually, if you think about it - updating billions of systems that quickly.
- rkagerer 2y agoThat is the right way to do it.
- throwaway7356 2y agoBut do you ever get free world-wide advertisement that everyone uses your product? Crowdstrike sure did and I'm sure they'll use that to sell it to more people.