6 ms·

Reminder that Microsoft could have programmed Windows to notice if a driver has caused a blue screen three times in a row, and prompt if you want to disable the

Reminder that Microsoft could have programmed Windows to notice if a driver has caused a blue screen three times in a row, and prompt if you want to disable the driver on boot. After all, Windows already collects how many times a driver causes a crash. This would have made recovery one click instead of heading into Safe Mode and needing BitLocker keys. But they didn’t. And Microsoft, I argue, also has blood on their hands for every hospital this hit. Giving users a prompt to disable the driver, after three successive failed boots, would have saved lives.

t-writescode 2y ago
How would that have helped the server farms that were experiencing the issue?
- gjsman-1000 2y ago
  Oh I don’t know, the servers down, you go and look as a technician, and you simply see a screen saying: “CSAgent.sys has caused a failure to boot three times in a row. Do you want to disable this driver? <Yes> <No>.” You click “Yes.” Server reboots with CloudStrike driver disabled. The day is saved in 5 minutes instead of building a custom ISO image or going on a BitLocker key recovery spree.
  - politelemon 2y ago
    It would still have required on site presence and interaction during which there is still downtime, so this accomplishes marginally small gains.
    - gjsman-1000 2y ago
      At the same time though, imagine you woke up and CloudStrike hit your organization. For most users, they’ll try clicking “Yes.” And then it’s back to work. After all, “No” just causes a blue screen again, might as well try the other path. This would have been the difference between the IT department handling 10,000+ calls or a few hundred (plus sending out a bulletin) in many, many organizations. It also could have saved billions at this point. Heck, it would have saved lives in hospitals.
      - jonathantf2 2y ago
        But then you have millions of endpoints booting without malware protection
      - echoangle 2y ago
        Can you cite some reports of deaths caused by the outage?
- deleted 2y ago
  [deleted]
- morkalork 2y ago
  Instead of prompting on the screen, disable the driver and boot directly into a recovery state that has networking enabled so sysadmins can push scripts and fixes? As long as it's not a network driver you'd be okay.
  - t-writescode 2y ago
    Disable the driver that is explicitly there to protect from malware and attacks? Wouldn’t malware just use that as an attack vector?
    - morkalork 2y ago
      >Nooo you don't understaaaand kernel code is special :'( actually BSOD was a desired feature because CrowdStrike is a Security (TM) application
danlitt 2y ago
[flagged]
- gjsman-1000 2y ago
  That’s very easily mitigated - write the security software so it can’t crash. Like, you know, drivers should be written. Malware can’t crash a well-written or memory-safe driver, so it will never be unloaded. Problem solved.
  - deleted 2y ago
    [deleted]
  - echoangle 2y ago
    Writing the driver so it can’t crash is the hard part, I think the developers knew that this was the goal.
  - jayd16 2y ago
    Just don't make mistakes is not actionable feedback.
    - stubish 2y ago
      With modern developer tooling, it kind of is. Pick the right language for the right job. You can catch the vast majority of crashes in your code at compile time, to the point where the potential of 'making a mistake' is a statistical improbability. But so many companies just stick with C, and even ignore compiler and static code analysis warnings, because the machismo of their devs protects them much better. Software by companies failing basic risk analysis.
      - jayd16 2y ago
        Why are we talking about language safety when it was a malformed config? It shouldn't boot with a malformed config, by design. The issues seem to be operational not technical.
Uvix 2y ago
Those hospitals chose to deploy software that didn't support testing. The blood is on their own hands.
- goosejuice 2y ago
  This is how I feel. If you're blindly installing software system wide, that has kernel access no less, and not accounting for failure of that software in your risk analysis then you are to blame more so than the vendor. Certainly I expect some SLA is in place but that's only of monetary benefit and irrelevant to keeping critical infra online.
galangalalgol 2y ago
I think sueing MS for the behavior that ensued when people installed a rootkit directly into the kernel and opened all the ports on their network to let that rootkit get used, is... excessive. Both MS and CS should have had a fail to previous good kernel ability, but the negligence here is clearly with CS for not even trying a blank data file in the automated tests for a piece of safety critical software, and then not using canary deployments before pushing to millions of devices.
crazygringo 2y ago
Do I like your idea for that? Yes, absolutely. It's a clever idea. But do I think Microsoft was negligent in not building that? No, I think that's going too far. Windows already has Safe Mode -- as you note -- to allow for manual recovery, which is what people are using. I don't think it makes sense for it to be Microsoft's legal responsibility to protect its users from software with a critical bug that wasn't written by Microsoft. Otherwise, where would it end? If a third-party program tries to delete all your user data, is it Microsoft's legal responsibility to check whenever a process is deleting a lot of data, and intervene with a confirmation dialog? Is it Microsoft's responsibility to protect you from all malware and ransomware, no matter how cleverly written? Is it Microsoft's responsibility to constantly cache program state on disk so that when a third-party program crashes, you don't lose your data since your last save? I think that's going too far, in terms of legal obligation.
- grumpyprole 2y ago
  Microsoft may be negligent in selling a product unsuitable for these applications. Windows is unsuitable precisely because it can be brought down by third party updates, such that it cannot recover without manual intervention by technical experts. Third party vendors are forced into writing unsafe kernel drivers because Microsoft does not provide sufficient user mode APIs. Windows has a dated design and a security model no longer fit for purpose. As for your other example, it could be protecting users from malicious programs that may delete data, simply by having a better security model, like Android and iOS.
  - crazygringo 2y ago
    I don't think Microsoft can be negligent here, because Windows isn't being brought down by Microsoft updates. Somebody bought Windows, and bought CrowdStrike. CrowdStrike is negligent, and possibly also the person/org who chose to rely on Windows+CrowdStrike without a backup plan if that resulted in further damages to others. Third party vendors are absolutely not "forced into writing unsafe kernel drivers". They can properly test things to write safer code (which CrowdStrike infamously didn't). And kernel mode is fundamentally required for security software like this, as far as I understand. And using app-based mobile OS's is not necessarily a useful comparison point. They are limited in all sorts of ways that desktop OS's are not -- and don't you hear people here on HN constantly complaining about that? A better comparison point is macOS and Linux. CrowdStrike also crashed Linux, and macOS still lets you bypass SIP if you want to.
Khaine 2y ago
AFAIK Windows does do that, except for drivers that are marked as required for boot. CrowdStrike's drivers are marked as required for boot.
ziml77 2y ago
Imagine I've installed CrowdStrike under the assumption that it makes my system more secure. Why would I want the OS to allow the system to boot up in a less secure state by providing a prompt for that? Most users will just click whichever option gets them back up and running and IT will have no control over that.
nerdjon 2y ago
This is very much a “easier said than done” situation that I would think Hacker News of all places would be better about when it comes to “just” doing something in code. First Windows already does something similar. After 3 it is supposed to boot into WindowsRE which gives you options to revert to a previous version, uninstall updates, and I believe also reverts configurations like recent driver installations. The problem here though, CrowdStrike itself didn’t update. It updated a definition file (last I saw at least) and that likely would not have been caught by Windows as a new version. Also frankly, not super thrilled at the idea of Windows just deciding to disable/uninstall something except for rolling back (so a previously working config) due to how things could interact. This situation could have been far worse and harder to recover from. In this case maybe Windows could have noticed that the configuration update is what was causing it and rolled that back, but it’s possible it would have just re-downloaded the file when it started back up anyways. Regarding saved lives, do we actually know that anyone’s lives were lost due to this? My local hospitals were still performing emergency surgery.
sudosysgen 2y ago
Windows does do that by default. If it fails to boot, it will start an "Automatic Repair" screen and it will offer to disable drivers (ie: Safe Mode), or sometime just disable the driver itself. The problem is that CrowdStrike doesn't want to let you start the computer without it running. It's the reason why it's an ELAM driver - it's marked as required for boot, so Windows won't try to boot without it, much like it won't if you remove crucial hardware drivers. I guess what they are trying to avoid is malware crashing the kernel driver which then gets disabled letting the malware roam free, without realizing that the cure is worse than the disease.
phendrenad2 2y ago
So attackers just have to find a way to trick the drivers into crashing 3 times and they have full access to the system without the pesky security systems in place? Nice!

t-writescode 2y ago

How would that have helped the server farms that were experiencing the issue?

gjsman-1000 2y ago
Oh I don’t know, the servers down, you go and look as a technician, and you simply see a screen saying: “CSAgent.sys has caused a failure to boot three times in a row. Do you want to disable this driver? <Yes> <No>.” You click “Yes.” Server reboots with CloudStrike driver disabled. The day is saved in 5 minutes instead of building a custom ISO image or going on a BitLocker key recovery spree.
- politelemon 2y ago
  It would still have required on site presence and interaction during which there is still downtime, so this accomplishes marginally small gains.
  - gjsman-1000 2y ago
    At the same time though, imagine you woke up and CloudStrike hit your organization. For most users, they’ll try clicking “Yes.” And then it’s back to work. After all, “No” just causes a blue screen again, might as well try the other path. This would have been the difference between the IT department handling 10,000+ calls or a few hundred (plus sending out a bulletin) in many, many organizations. It also could have saved billions at this point. Heck, it would have saved lives in hospitals.
    - jonathantf2 2y ago
      But then you have millions of endpoints booting without malware protection
    - echoangle 2y ago
      Can you cite some reports of deaths caused by the outage?
deleted 2y ago
[deleted]
morkalork 2y ago
Instead of prompting on the screen, disable the driver and boot directly into a recovery state that has networking enabled so sysadmins can push scripts and fixes? As long as it's not a network driver you'd be okay.
- t-writescode 2y ago
  Disable the driver that is explicitly there to protect from malware and attacks? Wouldn’t malware just use that as an attack vector?
  - morkalork 2y ago
    >Nooo you don't understaaaand kernel code is special :'( actually BSOD was a desired feature because CrowdStrike is a Security (TM) application

danlitt 2y ago

[flagged]

gjsman-1000 2y ago
That’s very easily mitigated - write the security software so it can’t crash. Like, you know, drivers should be written. Malware can’t crash a well-written or memory-safe driver, so it will never be unloaded. Problem solved.
- deleted 2y ago
  [deleted]
- echoangle 2y ago
  Writing the driver so it can’t crash is the hard part, I think the developers knew that this was the goal.
- jayd16 2y ago
  Just don't make mistakes is not actionable feedback.
  - stubish 2y ago
    With modern developer tooling, it kind of is. Pick the right language for the right job. You can catch the vast majority of crashes in your code at compile time, to the point where the potential of 'making a mistake' is a statistical improbability. But so many companies just stick with C, and even ignore compiler and static code analysis warnings, because the machismo of their devs protects them much better. Software by companies failing basic risk analysis.
    - jayd16 2y ago
      Why are we talking about language safety when it was a malformed config? It shouldn't boot with a malformed config, by design. The issues seem to be operational not technical.

Uvix 2y ago

Those hospitals chose to deploy software that didn't support testing. The blood is on their own hands.

goosejuice 2y ago
This is how I feel. If you're blindly installing software system wide, that has kernel access no less, and not accounting for failure of that software in your risk analysis then you are to blame more so than the vendor. Certainly I expect some SLA is in place but that's only of monetary benefit and irrelevant to keeping critical infra online.

galangalalgol 2y ago

I think sueing MS for the behavior that ensued when people installed a rootkit directly into the kernel and opened all the ports on their network to let that rootkit get used, is... excessive. Both MS and CS should have had a fail to previous good kernel ability, but the negligence here is clearly with CS for not even trying a blank data file in the automated tests for a piece of safety critical software, and then not using canary deployments before pushing to millions of devices.

crazygringo 2y ago

Do I like your idea for that? Yes, absolutely. It's a clever idea. But do I think Microsoft was negligent in not building that? No, I think that's going too far. Windows already has Safe Mode -- as you note -- to allow for manual recovery, which is what people are using. I don't think it makes sense for it to be Microsoft's legal responsibility to protect its users from software with a critical bug that wasn't written by Microsoft. Otherwise, where would it end? If a third-party program tries to delete all your user data, is it Microsoft's legal responsibility to check whenever a process is deleting a lot of data, and intervene with a confirmation dialog? Is it Microsoft's responsibility to protect you from all malware and ransomware, no matter how cleverly written? Is it Microsoft's responsibility to constantly cache program state on disk so that when a third-party program crashes, you don't lose your data since your last save? I think that's going too far, in terms of legal obligation.

grumpyprole 2y ago
Microsoft may be negligent in selling a product unsuitable for these applications. Windows is unsuitable precisely because it can be brought down by third party updates, such that it cannot recover without manual intervention by technical experts. Third party vendors are forced into writing unsafe kernel drivers because Microsoft does not provide sufficient user mode APIs. Windows has a dated design and a security model no longer fit for purpose. As for your other example, it could be protecting users from malicious programs that may delete data, simply by having a better security model, like Android and iOS.
- crazygringo 2y ago
  I don't think Microsoft can be negligent here, because Windows isn't being brought down by Microsoft updates. Somebody bought Windows, and bought CrowdStrike. CrowdStrike is negligent, and possibly also the person/org who chose to rely on Windows+CrowdStrike without a backup plan if that resulted in further damages to others. Third party vendors are absolutely not "forced into writing unsafe kernel drivers". They can properly test things to write safer code (which CrowdStrike infamously didn't). And kernel mode is fundamentally required for security software like this, as far as I understand. And using app-based mobile OS's is not necessarily a useful comparison point. They are limited in all sorts of ways that desktop OS's are not -- and don't you hear people here on HN constantly complaining about that? A better comparison point is macOS and Linux. CrowdStrike also crashed Linux, and macOS still lets you bypass SIP if you want to.

Khaine 2y ago

AFAIK Windows does do that, except for drivers that are marked as required for boot. CrowdStrike's drivers are marked as required for boot.

ziml77 2y ago

Imagine I've installed CrowdStrike under the assumption that it makes my system more secure. Why would I want the OS to allow the system to boot up in a less secure state by providing a prompt for that? Most users will just click whichever option gets them back up and running and IT will have no control over that.

nerdjon 2y ago

This is very much a “easier said than done” situation that I would think Hacker News of all places would be better about when it comes to “just” doing something in code. First Windows already does something similar. After 3 it is supposed to boot into WindowsRE which gives you options to revert to a previous version, uninstall updates, and I believe also reverts configurations like recent driver installations. The problem here though, CrowdStrike itself didn’t update. It updated a definition file (last I saw at least) and that likely would not have been caught by Windows as a new version. Also frankly, not super thrilled at the idea of Windows just deciding to disable/uninstall something except for rolling back (so a previously working config) due to how things could interact. This situation could have been far worse and harder to recover from. In this case maybe Windows could have noticed that the configuration update is what was causing it and rolled that back, but it’s possible it would have just re-downloaded the file when it started back up anyways. Regarding saved lives, do we actually know that anyone’s lives were lost due to this? My local hospitals were still performing emergency surgery.

sudosysgen 2y ago

Windows does do that by default. If it fails to boot, it will start an "Automatic Repair" screen and it will offer to disable drivers (ie: Safe Mode), or sometime just disable the driver itself. The problem is that CrowdStrike doesn't want to let you start the computer without it running. It's the reason why it's an ELAM driver - it's marked as required for boot, so Windows won't try to boot without it, much like it won't if you remove crucial hardware drivers. I guess what they are trying to avoid is malware crashing the kernel driver which then gets disabled letting the malware roam free, without realizing that the cure is worse than the disease.

phendrenad2 2y ago

So attackers just have to find a way to trick the drivers into crashing 3 times and they have full access to the system without the pesky security systems in place? Nice!