5 ms·

Right after you enter the bit locker recovery key. You do have your bit locker recovery key, right? .....right?

by Neil44 2y ago

tamimio 2y ago
I doubt most of the clients who use CS know what BitLocker is, let alone how to back it up, assuming it wasn’t backed up automatically by Windows.
- Fire-Dragon-DoL 2y ago
  Doesn't that get backed up automatically to the Microsoft account?
  - dist-epoch 2y ago
    That's opt-in. In Enterprise setups the key should be backed somewhere in Active Directory.
    - mrhhaacckk 2y ago
      Yes you should be able to pull it from your domain controllers. Unless they're also down, which they're likely to be seeing as Tier 0 assets are most likely to have crowdstrike on them. So you're now in a catch 22.
      - mavhc 2y ago
        Log into hypervisor, rollback VM
        olyjohn 2y ago
        Microsoft shops gonna be running Hyper-V. Probably also got hosed.
        tremon 2y ago
        Rolling back an Active Directory server is a spectacularly bad idea. Better make doubly sure it's not connected to any network before you even attempt to do so.
    - ryanjshaw 2y ago
      In theory. I've seen it not happen twice. (The worst part is that you can hit the Bitlocker recovery somewhat randomly because of an irrelevant piece of hardware failing, and now you have to rebuild the OS because the recovery key is MIA.)
  - tamimio 2y ago
    I know it does for personal accounts once linked to your machine. Years ago, I used the enterprise version and it didn’t, probably because it was “assumed” that it should be done with group policies, but that was in 2017.
- acdha 2y ago
  Most people installed CrowdStrike because an audit said they needed it. I find it exceedingly unlikely that the same audit did not say they have to enable Bitlocker and backup its keys.
  - toomuchtodo 2y ago
    I can confirm this. EDR checkbox for CrowdStrike, BitLocker enabled for local disk encryption checkbox. BitLocker backups to Entra because we know reality happens, no checkbox for that.
paganel 2y ago
This was particularly interesting (from the reddit thread posted above): > A colleague is dealing with a particularly nasty case. The server storing the BitLocker recovery keys (for thousands of users) is itself BitLocker protected and running CrowdStrike (he says mandates state that all servers must have "encryption at rest"). > His team believes that the recovery key for that server is stored somewhere else, and they may be able to get it back up and running, but they can't access any of the documentation to do so, because everything is down.
- radiator 2y ago
  I find that hilarious
  - lukan 2y ago
    Me too, as I am also not affected. But I do pity those guys who now try to solve that deadlock.
- lordnacho 2y ago
  > but they can't access any of the documentation to do so, because everything is down. One of my biggest frustrations with learning networking was not being able to access the internet. Nowadays you probably have a phone with a browser, but back in the day if you were sitting in a data room and you'd configured stuff wrong, you had a problem.
- RobotToaster 2y ago
  Nobody, not one person, thought that documentation should be stored in hard copy?
  - whynotmaybe 2y ago
    I'm guessing someone somewhere said that "it must be stored in hard copy in a safe" and the answer was in the range of "we don't have a safe, we'll be fine". Or worse, if it's like where I worked in the past, they're still in the buying process for a safe (started 13 months ago) and the analysts are building up a general plan for the management of the safe combination. They still have to start the discussions with the union to see how they'll adapt the salary for the people that will have to remember the code for the safe and who's gonna be legally responsible for anything that happens to the safe. Last follow-up meeting summary is "everything's going well but we'll have to modify the schedule and postpone the delivery date of a few months, let's say 6 to be safe"
sebazzz 2y ago
BitLocker for Business stores the bitlocker key centrally. Still, it is a huge manual undertaking fixing every system.
- tgshaik 2y ago
  Absolutely correct. Unfortunately, there is no other solution to this issue. If the laptops were powered down overnight, there might be a stroke of luck. However, this will be one of the most challenging recoveries in IT history, making it a highly unpleasant experience.
- teeheelol 2y ago
  Yeah in context we have about 1000 remote workers down. We have to call them and talk through each machine because we can't fix them remotely because they are stuck boot looping. A large proportion of these users are non-technical.
  - chrisjj 2y ago
    How fortunate the phone system is not vulnerable to CrowdStrike...
  - CoastalCoder 2y ago
    Man, talk about a mass-phishing opportunity.
- immibis 2y ago
  I heard the central system was on Azure, running CrowdStrike.
mschuster91 2y ago
MS Windows Recovery screen (or the OS installer disk) might ask you for the recovery key only, but you can unlock the drive manually with the password as well! I had to do that a week ago after a disk clone gone wrong, so in case someone steps on the same issue (this here is tested with Win 10, but it should be just the same for W11 and Server): 1. Boot the affected machine from the Windows installer disk 2. Use "Repair options" 3. Click through to the option to spawn a shell 4. It will now ask you for unlocking the disk with a recovery key. SKIP THAT. 5. In the shell, type: "manage-bde -unlock C: -Password", enter the password 6. The drive is unlocked, now go and execute whatever recovery you have to do. Good luck.
- sam_lowry_ 2y ago
  On my corporate Windows 11 22H2 "manage-bde -unlock C: -Password" does not unlock the disk with the user key. I guess it needs recovery key as well.
- plonk 2y ago
  Don’t you need more options if the key is in a TPM, or there is a password but it’s only part of the key? Can you even get the secret from the TPM in recovery mode?
  - mschuster91 2y ago
    > Can you even get the secret from the TPM in recovery mode? Given that you can (relatively trivially) sniff the TPM communication to obtain the key [1], yes it should be possible. Can't verify it though as I've long ago switched to Mac for my primary driver and the old cheesegrater Mac I use as a gaming rig doesn't have a hardware TPM chip. [1] https://pulsesecurity.co.nz/articles/TPM-sniffing https://pulsesecurity.co.nz/articles/TPM-sniffing
    - toast0 2y ago
      TPMs embedded in the processor (fTPM) are pretty popular and it's a lot harder to sniff communications that stay inside the cpu.
fragmede 2y ago
help for if you have bitlocker turned on (because of course you do). https://x.com/attilabubby/status/1814216589559861673 https://x.com/attilabubby/status/1814216589559861673 https://x.com/nathanmcnulty/status/1785094215749476722?s=46 https://x.com/nathanmcnulty/status/1785094215749476722?s=46 GPO to fix: https://gist.github.com/whichbuffer/7830c73711589dcf9e7a5217797ca617 https://gist.github.com/whichbuffer/7830c73711589dcf9e7a5217...
downrightmike 2y ago
Saved to my desktop? How does that help? /s

Right after you enter the bit locker recovery key. You do have your bit locker recovery key, right? .....right?

by Neil44 2y ago

Right after you enter the bit locker recovery key. You do have your bit locker recovery key, right? .....right?

I doubt most of the clients who use CS know what BitLocker is, let alone how to back it up, assuming it wasn’t backed up automatically by Windows.

Fire-Dragon-DoL 2y ago
Doesn't that get backed up automatically to the Microsoft account?
- dist-epoch 2y ago
  That's opt-in. In Enterprise setups the key should be backed somewhere in Active Directory.
  - mrhhaacckk 2y ago
    Yes you should be able to pull it from your domain controllers. Unless they're also down, which they're likely to be seeing as Tier 0 assets are most likely to have crowdstrike on them. So you're now in a catch 22.
    - mavhc 2y ago
      Log into hypervisor, rollback VM
      - olyjohn 2y ago
        Microsoft shops gonna be running Hyper-V. Probably also got hosed.
      - tremon 2y ago
        Rolling back an Active Directory server is a spectacularly bad idea. Better make doubly sure it's not connected to any network before you even attempt to do so.
  - ryanjshaw 2y ago
    In theory. I've seen it not happen twice. (The worst part is that you can hit the Bitlocker recovery somewhat randomly because of an irrelevant piece of hardware failing, and now you have to rebuild the OS because the recovery key is MIA.)
- tamimio 2y ago
  I know it does for personal accounts once linked to your machine. Years ago, I used the enterprise version and it didn’t, probably because it was “assumed” that it should be done with group policies, but that was in 2017.
acdha 2y ago
Most people installed CrowdStrike because an audit said they needed it. I find it exceedingly unlikely that the same audit did not say they have to enable Bitlocker and backup its keys.
- toomuchtodo 2y ago
  I can confirm this. EDR checkbox for CrowdStrike, BitLocker enabled for local disk encryption checkbox. BitLocker backups to Entra because we know reality happens, no checkbox for that.

paganel 2y ago

This was particularly interesting (from the reddit thread posted above): > A colleague is dealing with a particularly nasty case. The server storing the BitLocker recovery keys (for thousands of users) is itself BitLocker protected and running CrowdStrike (he says mandates state that all servers must have "encryption at rest"). > His team believes that the recovery key for that server is stored somewhere else, and they may be able to get it back up and running, but they can't access any of the documentation to do so, because everything is down.

radiator 2y ago
I find that hilarious
- lukan 2y ago
  Me too, as I am also not affected. But I do pity those guys who now try to solve that deadlock.
lordnacho 2y ago
> but they can't access any of the documentation to do so, because everything is down. One of my biggest frustrations with learning networking was not being able to access the internet. Nowadays you probably have a phone with a browser, but back in the day if you were sitting in a data room and you'd configured stuff wrong, you had a problem.
RobotToaster 2y ago
Nobody, not one person, thought that documentation should be stored in hard copy?
- whynotmaybe 2y ago
  I'm guessing someone somewhere said that "it must be stored in hard copy in a safe" and the answer was in the range of "we don't have a safe, we'll be fine". Or worse, if it's like where I worked in the past, they're still in the buying process for a safe (started 13 months ago) and the analysts are building up a general plan for the management of the safe combination. They still have to start the discussions with the union to see how they'll adapt the salary for the people that will have to remember the code for the safe and who's gonna be legally responsible for anything that happens to the safe. Last follow-up meeting summary is "everything's going well but we'll have to modify the schedule and postpone the delivery date of a few months, let's say 6 to be safe"

sebazzz 2y ago

BitLocker for Business stores the bitlocker key centrally. Still, it is a huge manual undertaking fixing every system.

tgshaik 2y ago
Absolutely correct. Unfortunately, there is no other solution to this issue. If the laptops were powered down overnight, there might be a stroke of luck. However, this will be one of the most challenging recoveries in IT history, making it a highly unpleasant experience.
teeheelol 2y ago
Yeah in context we have about 1000 remote workers down. We have to call them and talk through each machine because we can't fix them remotely because they are stuck boot looping. A large proportion of these users are non-technical.
- chrisjj 2y ago
  How fortunate the phone system is not vulnerable to CrowdStrike...
- CoastalCoder 2y ago
  Man, talk about a mass-phishing opportunity.
immibis 2y ago
I heard the central system was on Azure, running CrowdStrike.

mschuster91 2y ago

MS Windows Recovery screen (or the OS installer disk) might ask you for the recovery key only, but you can unlock the drive manually with the password as well! I had to do that a week ago after a disk clone gone wrong, so in case someone steps on the same issue (this here is tested with Win 10, but it should be just the same for W11 and Server): 1. Boot the affected machine from the Windows installer disk 2. Use "Repair options" 3. Click through to the option to spawn a shell 4. It will now ask you for unlocking the disk with a recovery key. SKIP THAT. 5. In the shell, type: "manage-bde -unlock C: -Password", enter the password 6. The drive is unlocked, now go and execute whatever recovery you have to do. Good luck.

sam_lowry_ 2y ago
On my corporate Windows 11 22H2 "manage-bde -unlock C: -Password" does not unlock the disk with the user key. I guess it needs recovery key as well.
plonk 2y ago
Don’t you need more options if the key is in a TPM, or there is a password but it’s only part of the key? Can you even get the secret from the TPM in recovery mode?
- mschuster91 2y ago
  > Can you even get the secret from the TPM in recovery mode? Given that you can (relatively trivially) sniff the TPM communication to obtain the key [1], yes it should be possible. Can't verify it though as I've long ago switched to Mac for my primary driver and the old cheesegrater Mac I use as a gaming rig doesn't have a hardware TPM chip. [1] https://pulsesecurity.co.nz/articles/TPM-sniffing https://pulsesecurity.co.nz/articles/TPM-sniffing
  - toast0 2y ago
    TPMs embedded in the processor (fTPM) are pretty popular and it's a lot harder to sniff communications that stay inside the cpu.

fragmede 2y ago

help for if you have bitlocker turned on (because of course you do). https://x.com/attilabubby/status/1814216589559861673 https://x.com/attilabubby/status/1814216589559861673 https://x.com/nathanmcnulty/status/1785094215749476722?s=46 https://x.com/nathanmcnulty/status/1785094215749476722?s=46 GPO to fix: https://gist.github.com/whichbuffer/7830c73711589dcf9e7a5217797ca617 https://gist.github.com/whichbuffer/7830c73711589dcf9e7a5217...

downrightmike 2y ago

Saved to my desktop? How does that help? /s