4 ms·

So apparently "The issue has been identified, isolated and a fix has been deployed" https://x.com/George_Kurtz/status/1814235001745027317 https://x.com/George_K

So apparently "The issue has been identified, isolated and a fix has been deployed" https://x.com/George_Kurtz/status/1814235001745027317 https://x.com/George_Kurtz/status/1814235001745027317 Yet the chaos seems to continue. Could it be that this fix can't be rolled out automatically to affected machines because they crash during boot - before the Crowdstrike Updater runs?

CubsFan1060 2y ago
Correct. Many just end up in an endless loop and never actually boot. It's about as bad as it gets.
jwitthuhn 2y ago
Yeah, you need to manually fix each affected system by booting in safe mode. Not possible to do remotely.
- rudasn 2y ago
  Ouch!
- martyvis 2y ago
  And you will need your bitlocker recovery key to access your encrypted drive in safe mode. I luckily had mine available offline There's going be a lot of handholding to get end users through this.
  - fifteen1506 2y ago
    You can enable safemode for next boot without the recovery key and then you can delete the offending file on that next boot.
    - kiririn 2y ago
      That requires being able to boot in the first place
      - fifteen1506 2y ago
        You can do a minimal boot. I'm told.
martypitt 2y ago
That update is so tone-deaf and half-assed. There's no apology. If you go to the website, there's nothing on their front-page. The post on their blog (https://www.crowdstrike.com/blog/statement-on-windows-sensor-update/ https://www.crowdstrike.com/blog/statement-on-windows-sensor...) doesn't even link to the solution. There's no link to "Support Portal" anywhere to be seen on their front-page. So, you have to go digging to find the update. And the "Fix" that they've "Deployed" requires someone to go to Every. Single. Machine. Companies with fleets of 50k machines are on this HN thread - how are they supposed to visit every machine?!?!
- mycall 2y ago
  They won't apologize for legal reasons. Also, it will only make their stock fall further.
  - ibejoeb 2y ago
    The CEO actually did apologize: "We're deeply sorry for the impact that we've caused to customers, to travelers, to anyone affected by this..." https://www.reuters.com/technology/crowdstrike-ceo-apologizes-tech-outage-says-systems-should-be-recovering-nbc-2024-07-19/ https://www.reuters.com/technology/crowdstrike-ceo-apologize...
- _heimdall 2y ago
  Any response they make in the middle of a global outage will be half-assed. They have all available resources figuring out what the hell just happened and how to fix it. An apology this early is a lose-lose. If they do apologize they'll piss off people dealing with it and want a fix not an apology. If they do t apologize they're tone deaf and don't seem to care.
  - adamhp 2y ago
    Imagine being anywhere near the team that sent this...
senectus1 2y ago
lol sounds good, but how the hell do they deploy a fix to a machine that has crash and is looping BSOD with no internet or netwrok connectivity... You do what I've been doing for the last 10 hours or so. you walk to each and every desktop and manually type in the bitlocker key so you can remove the offending update. at least the virtual devices can be fixed sitting at a desk while suckling at a comfort coffee..

CubsFan1060 2y ago

Correct. Many just end up in an endless loop and never actually boot. It's about as bad as it gets.

jwitthuhn 2y ago

Yeah, you need to manually fix each affected system by booting in safe mode. Not possible to do remotely.

rudasn 2y ago
Ouch!
martyvis 2y ago
And you will need your bitlocker recovery key to access your encrypted drive in safe mode. I luckily had mine available offline There's going be a lot of handholding to get end users through this.
- fifteen1506 2y ago
  You can enable safemode for next boot without the recovery key and then you can delete the offending file on that next boot.
  - kiririn 2y ago
    That requires being able to boot in the first place
    - fifteen1506 2y ago
      You can do a minimal boot. I'm told.

martypitt 2y ago

That update is so tone-deaf and half-assed. There's no apology. If you go to the website, there's nothing on their front-page. The post on their blog (https://www.crowdstrike.com/blog/statement-on-windows-sensor-update/ https://www.crowdstrike.com/blog/statement-on-windows-sensor...) doesn't even link to the solution. There's no link to "Support Portal" anywhere to be seen on their front-page. So, you have to go digging to find the update. And the "Fix" that they've "Deployed" requires someone to go to Every. Single. Machine. Companies with fleets of 50k machines are on this HN thread - how are they supposed to visit every machine?!?!

mycall 2y ago
They won't apologize for legal reasons. Also, it will only make their stock fall further.
- ibejoeb 2y ago
  The CEO actually did apologize: "We're deeply sorry for the impact that we've caused to customers, to travelers, to anyone affected by this..." https://www.reuters.com/technology/crowdstrike-ceo-apologizes-tech-outage-says-systems-should-be-recovering-nbc-2024-07-19/ https://www.reuters.com/technology/crowdstrike-ceo-apologize...
_heimdall 2y ago
Any response they make in the middle of a global outage will be half-assed. They have all available resources figuring out what the hell just happened and how to fix it. An apology this early is a lose-lose. If they do apologize they'll piss off people dealing with it and want a fix not an apology. If they do t apologize they're tone deaf and don't seem to care.
- adamhp 2y ago
  Imagine being anywhere near the team that sent this...

senectus1 2y ago

lol sounds good, but how the hell do they deploy a fix to a machine that has crash and is looping BSOD with no internet or netwrok connectivity... You do what I've been doing for the last 10 hours or so. you walk to each and every desktop and manually type in the bitlocker key so you can remove the offending update. at least the virtual devices can be fixed sitting at a desk while suckling at a comfort coffee..