4 ms·

Workaround steps: https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/comment/ldvwkbn https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/commen...

ndsipa_pomu 2y ago
> CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes. > Workaround Steps: > Boot Windows into Safe Mode or the Windows Recovery Environment > Navigate to the C:\Windows\System32\drivers\CrowdStrike directory > Locate the file matching “C-00000291*.sys”, and delete it. > Boot the host normally.
- tjoff 2y ago
  Was thinking about a bootable usb-stick that would do that automagically. But I guess it is harder to boot from a usb-stick in these environments than the actual fix. I guess more feasible and even neater to do it if you have network boot or similar.
entuno 2y ago
There's potentially a huge issue here for people using BitLocker with on-prem AD, because they'll need the BitLocker recovery keys for each endpoint to go in an fix it. And if all those recovery keys are stored in AD (as they usually are), and the Domain Controllers all had Crowdstrike on them...
- mschuster91 2y ago
  Assuming that they also have a regular Bitlocker password, there's hope with a bit manual effort. https://news.ycombinator.com/item?id=41003893 https://news.ycombinator.com/item?id=41003893
  - entuno 2y ago
    Most of the large deployments I've seen don't use pre-boot PINs, because of the difficulty of managing them with users - they just use TPM and occasionally network unlock. So might save a few people, but I suspect not many.
    - mschuster91 2y ago
      Yeah but TPM-only Bitlocker shouldn't be affected anyway by this issue, these machines should start up just fine. Whoever only has AD-based Bitlocker encryption is straight up fucked. Man, and that on a Friday.
      - tjoff 2y ago
        That's the easy part? just do the domain controller first?
- hulitu 2y ago
  This is the best definition of "Single point of failure" i have ever seen.
- depr 2y ago
  Bitlocker keys are apparently not necessary: https://x.com/AttilaBubby/status/1814216589559861673 https://x.com/AttilaBubby/status/1814216589559861673
  - tamimio 2y ago
    It might work on some machines, but doubt to work on the rest. Worth the try.
- mejs 2y ago
  I got around BitLocker and booted into safe mode by setting automatic boot to safe mode via bcdedit https://blog.vladovince.com/mitigating-the-crowdstrike-outage-without-bitlocker-keys-how-to-get-into-windows-recovery-mode-via-secureboot-toggling/ https://blog.vladovince.com/mitigating-the-crowdstrike-outag...
commandersaki 2y ago
So booting into safe mode should do the trick right, even if Bitlocker is enabled?
higeorge13 2y ago
2000s vibes.
sschueller 2y ago
What if you have 50k workstations? Can you even do this remotely? The problem may be fixed but I can see some companies having a really shit weekend.

ndsipa_pomu 2y ago

> CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes. > Workaround Steps: > Boot Windows into Safe Mode or the Windows Recovery Environment > Navigate to the C:\Windows\System32\drivers\CrowdStrike directory > Locate the file matching “C-00000291*.sys”, and delete it. > Boot the host normally.

tjoff 2y ago
Was thinking about a bootable usb-stick that would do that automagically. But I guess it is harder to boot from a usb-stick in these environments than the actual fix. I guess more feasible and even neater to do it if you have network boot or similar.

entuno 2y ago

There's potentially a huge issue here for people using BitLocker with on-prem AD, because they'll need the BitLocker recovery keys for each endpoint to go in an fix it. And if all those recovery keys are stored in AD (as they usually are), and the Domain Controllers all had Crowdstrike on them...

mschuster91 2y ago
Assuming that they also have a regular Bitlocker password, there's hope with a bit manual effort. https://news.ycombinator.com/item?id=41003893 https://news.ycombinator.com/item?id=41003893
- entuno 2y ago
  Most of the large deployments I've seen don't use pre-boot PINs, because of the difficulty of managing them with users - they just use TPM and occasionally network unlock. So might save a few people, but I suspect not many.
  - mschuster91 2y ago
    Yeah but TPM-only Bitlocker shouldn't be affected anyway by this issue, these machines should start up just fine. Whoever only has AD-based Bitlocker encryption is straight up fucked. Man, and that on a Friday.
    - tjoff 2y ago
      That's the easy part? just do the domain controller first?
hulitu 2y ago
This is the best definition of "Single point of failure" i have ever seen.
depr 2y ago
Bitlocker keys are apparently not necessary: https://x.com/AttilaBubby/status/1814216589559861673 https://x.com/AttilaBubby/status/1814216589559861673
- tamimio 2y ago
  It might work on some machines, but doubt to work on the rest. Worth the try.
mejs 2y ago
I got around BitLocker and booted into safe mode by setting automatic boot to safe mode via bcdedit https://blog.vladovince.com/mitigating-the-crowdstrike-outage-without-bitlocker-keys-how-to-get-into-windows-recovery-mode-via-secureboot-toggling/ https://blog.vladovince.com/mitigating-the-crowdstrike-outag...

commandersaki 2y ago

So booting into safe mode should do the trick right, even if Bitlocker is enabled?

higeorge13 2y ago

2000s vibes.

sschueller 2y ago

What if you have 50k workstations? Can you even do this remotely? The problem may be fixed but I can see some companies having a really shit weekend.