8 ms·

Microsoft should revoke the CrowdStrike driver signature and should do an internal check as to why CrowdStrike's driver was approved when it can execute arbitra

Microsoft should revoke the CrowdStrike driver signature and should do an internal check as to why CrowdStrike's driver was approved when it can execute arbitrary code on the kernel level without any checks. If your "driver" requires this feature MS should require CrowdStrike to submit the entire source and they should have to pay MS to do a review of the code. What is the point of driver signing if a vendor can basically build in a back door and Microsoft doesn't validate that this back door is at least somewhat reasonable

_flux 2y ago
Do you think Microsoft customers using CrowdStrike would then be happier, being unable to run the software at all, due to an action Microsoft took? Backdoors of all kinds can be installed to most any operating system without vendor co-operation. That is the nature of general-purpose operating systems.
- sigseg1v 2y ago
  I'm a customer that is forced to use CrowdStrike via IT policies and I would be giddy with delight if something came along and caused the removal of it from my systems. I don't need programs sitting on my computer preventing me from installing code that I've literally just compiled, preventing me from deleting or modifying folders on my machine, and causing extreme lag for many basic system operations even when it does work. At this point, the time in lost productivity (via normal operation) and downtime (via their recent bug) has easily exceeded a thousand times over the aggregate sum of all benefits that CrowdStrike will ever have provided from threat detection and prevention. It's time to remove the malware.
  - _flux 2y ago
    You are not the customer, though, your employer is the customer. Perhaps you should push this change up in the food chain, then, and if the company is good the request will be taken seriously. As I understand it, while CrowdStrike is the biggest name in EDR, it's far from the only one, if that's what your company requires to pass some checkboxes in certifications.
    - hello_moto 2y ago
      Vendors are competing with one and another to win contracts. CIO/CISO don't select vendors lightly. There seems to be a typical/classical Engineer's mindset of "make a claim first, ask later" around the subject lately. "My boss plays golf with Sales Rep" might need more proof because if they selected the lesser capable vendors and they got hit with ransomware, bet my ass your boss will no longer play Golf with any Sales Rep ever.
      - CRConrad 2y ago
        > Vendors are competing with one and another to win contracts. Sure, in a well-functioning market economy without any distortions. But there are lots of those at play, so competition is severely hampered (by network effects, regulatory capture, and on and on... Up to and including, I suspect, mere ephemeral fashion). What we actually have in many areas of the "tech market" are oligopolies and near-monopolies, not perfect competition. > CIO/CISO don't select vendors lightly. Muahaha. Seems rather more like they're at least as naïve as any Web-surfing consumer on their sofa, easily bamboozled by trendy buzzwords and slick marketing campaigns.
  - deleted 2y ago
    [deleted]
  - hello_moto 2y ago
    Sounds like your IT (sec team, specifically) doesn't setup the software correctly. I've worked for a company that installs Falcon on all its fleet and I never run into issues like yours.
- mbreese 2y ago
  At this point… yes. It would be one thing Microsoft could do to focus 100% of the attention/blame away from Windows and onto CloudStrike. And customers will want their pound of flesh from somewhere. Really, this should serve as a wake up call w/in Microsoft to start to harden the kernel against such vulnerabilities. Was the crash the fault of Windows? No. But did a Windows design decision make this possible? yes. I’m sure the design decision made sense at the time (at least business sense). Keeping the kernel more open for others to add drivers to makes it easier to write/add drivers, but makes the system more vulnerable. This a good opportunity within Microsoft to get support for changing that.
  - _flux 2y ago
    Ultimately this would have been almost a non-issue if there had been better deployment strategies in place for also the data file updates. If by changing the system you mean adding some kind of in-kernel isolation to it, then I don't think it would be worth the effort to make that kind of major change to the way operating systems work just to give arguably a minor risk reduction to systems—in particular if CrowdStrike and other vendors take some learnings from this event. Microsoft might improve their system rollback mechanism to also include files that are not strictly integrated to the system, merely used by the parts that are (the channel files loaded by the driver). Actually I think we can just be happy that the incident was a mistake, not an attack. Had this kind of "first ever" situation been an attack, it could be extremely difficult to recover from it. I wonder how well EDRs deal with "attacks from within".. CrowdStrike pulled off the update within 1.5 hours. I wonder if they actually use Falcon themselves? But then somehow missed the problem? Doesn't seem like they eat their own dog food :). (Or at least their own channel files.)
    - mbreese 2y ago
      If many things had gone differently, this could have been avoided. But I’m looking at this from the Microsoft perspective. No matter how much people scream high and loud that it was a CloudStrike issue and not Microsoft’s fault, Microsoft is still getting blamed. It’s a Windows BSOD. I talked to my dad (retired enterprise operations/IT) this weekend and he was telling me that the next computer he buys will probably be a Mac, largely because he doesn’t want to deal with the possibility of a crash like this. Does he run CloudStrike? Not at all. Does he know who they are? Nope. (He’s been retired for a while) What he does know (well, thinks) is that Windows now has an unstable kernel. And Microsoft has no control over distribution policies for other vendors. How those vendors distribute updates is up to them. Even if a sane deployment strategy could have avoided the larger global problems, Microsoft can’t control that. So, if you have Microsoft dealing with negative publicity and public sentiment, with no way to control errors like this in the future, what can you do? To me, the best they can do is kneecap CloudStrike, put the full blame on them, and use this as an excuse to change the kernel/driver model to one where they can have more control over the stability of the OS.
- CaptainZapp 2y ago
  > Backdoors of all kinds can be installed to most any operating system without vendor co-operation Not on Kernel level. Not without active support by the vendor.
  - _flux 2y ago
    How much does it really help you if your complete user-space can still be messed up by an offending Windows SYSTEM process? As I understand it, they are able to hurt the system e.g. by killing processes, uninstalling applications, replacing binaries, allocating memory, starting too many processes, .. Actually I could easily see a buggy remote system management update could just decide to uninstall everything and nuke the system, because it thinks it's stolen. And it would be designed functionality for it.
roblabla 2y ago
Unless you have a source, you should really avoid spreading misinfo here. CrowdStrike doesn't have kernel-level ACE. It has a buggy configuration parser, and they pushed a corrupted config that triggered those buggy codepath in the parser.
- sschueller 2y ago
  My source: https://www.youtube.com/watch?v=wAzEJxOo1ts https://www.youtube.com/watch?v=wAzEJxOo1ts From my understanding the CS driver lives in the kernel space and parses configs/applications downloaded in the user space. Hence the system even does a BSOD. "CrowdStrike doesn't have kernel-level ACE" please provide your source.
  - roblabla 2y ago
    > From my understanding the CS driver lives in the kernel space and parses configs/applications downloaded in the user space. Hence the system even does a BSOD. That's my understanding as well, but not quite the same as > execute arbitrary code on the kernel level without any checks At least for me, when we talk about kernel-level ACE, it's something like libcapcom[0], which allowed executing arbitrary unsigned code in the kernel. Here, the driver can only execute the code present within itself, which was signed by Microsoft. The configuration itself isn't signed by microsoft, but the config isn't code (at least, as far as I can tell - I see some people claiming the CS-0000.sys files are essentially bytecode, but have yet to see conclusive proof of this). Now, we could argue that it's weird that Microsoft signed a buggy driver, and MS should do better qualification of third-party drivers. But in practice, MS doesn't really vet driver quality. From what I can tell, the driver signing is mostly there so they can easily attribute provenance of drivers, and revoke the certs if it ends up in the hands of malicious actors. [0]: https://github.com/notscimmy/libcapcom https://github.com/notscimmy/libcapcom
- dathinab 2y ago
  there are some places claiming that this "config" language is so flexible that it's basically a interpreted scripting language but AFIK no sources I trust have yet claimed it but it's probably where the idea comes from
AshamedCaptain 2y ago
> when it can execute arbitrary code on the kernel level without any checks That would be grounds for blacklisting indeed (out of experience). However, that's not the case here, no matter how you put it.
cookiengineer 2y ago
> What is the point of driver signing if a vendor can basically build in a back door and Microsoft doesn't validate that Before you downvote that comment, I'd like to remind everyone that this was already happening. Realtek's driver cert was leaked, and a lot of malware used this cert to sign their drivers for _a decade_ until anything happened about it. Microsoft's driver signing workflow is utterly pointless and it doesn't mean anything. Any vendor that takes their security serious should never trust those driver signatures.
SoftTalker 2y ago
I never assumed that driver signing was any kind of indicator of quality. It simply says "this is the Crowdstrike driver, it has not been modified" Maybe I'm wrong and Microsoft does some QA on drivers before they are signed?
rramadass 2y ago
Finally! You hit the nail right on the head!

_flux 2y ago

Do you think Microsoft customers using CrowdStrike would then be happier, being unable to run the software at all, due to an action Microsoft took? Backdoors of all kinds can be installed to most any operating system without vendor co-operation. That is the nature of general-purpose operating systems.

sigseg1v 2y ago
I'm a customer that is forced to use CrowdStrike via IT policies and I would be giddy with delight if something came along and caused the removal of it from my systems. I don't need programs sitting on my computer preventing me from installing code that I've literally just compiled, preventing me from deleting or modifying folders on my machine, and causing extreme lag for many basic system operations even when it does work. At this point, the time in lost productivity (via normal operation) and downtime (via their recent bug) has easily exceeded a thousand times over the aggregate sum of all benefits that CrowdStrike will ever have provided from threat detection and prevention. It's time to remove the malware.
- _flux 2y ago
  You are not the customer, though, your employer is the customer. Perhaps you should push this change up in the food chain, then, and if the company is good the request will be taken seriously. As I understand it, while CrowdStrike is the biggest name in EDR, it's far from the only one, if that's what your company requires to pass some checkboxes in certifications.
  - hello_moto 2y ago
    Vendors are competing with one and another to win contracts. CIO/CISO don't select vendors lightly. There seems to be a typical/classical Engineer's mindset of "make a claim first, ask later" around the subject lately. "My boss plays golf with Sales Rep" might need more proof because if they selected the lesser capable vendors and they got hit with ransomware, bet my ass your boss will no longer play Golf with any Sales Rep ever.
    - CRConrad 2y ago
      > Vendors are competing with one and another to win contracts. Sure, in a well-functioning market economy without any distortions. But there are lots of those at play, so competition is severely hampered (by network effects, regulatory capture, and on and on... Up to and including, I suspect, mere ephemeral fashion). What we actually have in many areas of the "tech market" are oligopolies and near-monopolies, not perfect competition. > CIO/CISO don't select vendors lightly. Muahaha. Seems rather more like they're at least as naïve as any Web-surfing consumer on their sofa, easily bamboozled by trendy buzzwords and slick marketing campaigns.
- deleted 2y ago
  [deleted]
- hello_moto 2y ago
  Sounds like your IT (sec team, specifically) doesn't setup the software correctly. I've worked for a company that installs Falcon on all its fleet and I never run into issues like yours.
mbreese 2y ago
At this point… yes. It would be one thing Microsoft could do to focus 100% of the attention/blame away from Windows and onto CloudStrike. And customers will want their pound of flesh from somewhere. Really, this should serve as a wake up call w/in Microsoft to start to harden the kernel against such vulnerabilities. Was the crash the fault of Windows? No. But did a Windows design decision make this possible? yes. I’m sure the design decision made sense at the time (at least business sense). Keeping the kernel more open for others to add drivers to makes it easier to write/add drivers, but makes the system more vulnerable. This a good opportunity within Microsoft to get support for changing that.
- _flux 2y ago
  Ultimately this would have been almost a non-issue if there had been better deployment strategies in place for also the data file updates. If by changing the system you mean adding some kind of in-kernel isolation to it, then I don't think it would be worth the effort to make that kind of major change to the way operating systems work just to give arguably a minor risk reduction to systems—in particular if CrowdStrike and other vendors take some learnings from this event. Microsoft might improve their system rollback mechanism to also include files that are not strictly integrated to the system, merely used by the parts that are (the channel files loaded by the driver). Actually I think we can just be happy that the incident was a mistake, not an attack. Had this kind of "first ever" situation been an attack, it could be extremely difficult to recover from it. I wonder how well EDRs deal with "attacks from within".. CrowdStrike pulled off the update within 1.5 hours. I wonder if they actually use Falcon themselves? But then somehow missed the problem? Doesn't seem like they eat their own dog food :). (Or at least their own channel files.)
  - mbreese 2y ago
    If many things had gone differently, this could have been avoided. But I’m looking at this from the Microsoft perspective. No matter how much people scream high and loud that it was a CloudStrike issue and not Microsoft’s fault, Microsoft is still getting blamed. It’s a Windows BSOD. I talked to my dad (retired enterprise operations/IT) this weekend and he was telling me that the next computer he buys will probably be a Mac, largely because he doesn’t want to deal with the possibility of a crash like this. Does he run CloudStrike? Not at all. Does he know who they are? Nope. (He’s been retired for a while) What he does know (well, thinks) is that Windows now has an unstable kernel. And Microsoft has no control over distribution policies for other vendors. How those vendors distribute updates is up to them. Even if a sane deployment strategy could have avoided the larger global problems, Microsoft can’t control that. So, if you have Microsoft dealing with negative publicity and public sentiment, with no way to control errors like this in the future, what can you do? To me, the best they can do is kneecap CloudStrike, put the full blame on them, and use this as an excuse to change the kernel/driver model to one where they can have more control over the stability of the OS.
CaptainZapp 2y ago
> Backdoors of all kinds can be installed to most any operating system without vendor co-operation Not on Kernel level. Not without active support by the vendor.
- _flux 2y ago
  How much does it really help you if your complete user-space can still be messed up by an offending Windows SYSTEM process? As I understand it, they are able to hurt the system e.g. by killing processes, uninstalling applications, replacing binaries, allocating memory, starting too many processes, .. Actually I could easily see a buggy remote system management update could just decide to uninstall everything and nuke the system, because it thinks it's stolen. And it would be designed functionality for it.

roblabla 2y ago

Unless you have a source, you should really avoid spreading misinfo here. CrowdStrike doesn't have kernel-level ACE. It has a buggy configuration parser, and they pushed a corrupted config that triggered those buggy codepath in the parser.

sschueller 2y ago
My source: https://www.youtube.com/watch?v=wAzEJxOo1ts https://www.youtube.com/watch?v=wAzEJxOo1ts From my understanding the CS driver lives in the kernel space and parses configs/applications downloaded in the user space. Hence the system even does a BSOD. "CrowdStrike doesn't have kernel-level ACE" please provide your source.
- roblabla 2y ago
  > From my understanding the CS driver lives in the kernel space and parses configs/applications downloaded in the user space. Hence the system even does a BSOD. That's my understanding as well, but not quite the same as > execute arbitrary code on the kernel level without any checks At least for me, when we talk about kernel-level ACE, it's something like libcapcom[0], which allowed executing arbitrary unsigned code in the kernel. Here, the driver can only execute the code present within itself, which was signed by Microsoft. The configuration itself isn't signed by microsoft, but the config isn't code (at least, as far as I can tell - I see some people claiming the CS-0000.sys files are essentially bytecode, but have yet to see conclusive proof of this). Now, we could argue that it's weird that Microsoft signed a buggy driver, and MS should do better qualification of third-party drivers. But in practice, MS doesn't really vet driver quality. From what I can tell, the driver signing is mostly there so they can easily attribute provenance of drivers, and revoke the certs if it ends up in the hands of malicious actors. [0]: https://github.com/notscimmy/libcapcom https://github.com/notscimmy/libcapcom
dathinab 2y ago
there are some places claiming that this "config" language is so flexible that it's basically a interpreted scripting language but AFIK no sources I trust have yet claimed it but it's probably where the idea comes from

AshamedCaptain 2y ago

> when it can execute arbitrary code on the kernel level without any checks That would be grounds for blacklisting indeed (out of experience). However, that's not the case here, no matter how you put it.

cookiengineer 2y ago

> What is the point of driver signing if a vendor can basically build in a back door and Microsoft doesn't validate that Before you downvote that comment, I'd like to remind everyone that this was already happening. Realtek's driver cert was leaked, and a lot of malware used this cert to sign their drivers for _a decade_ until anything happened about it. Microsoft's driver signing workflow is utterly pointless and it doesn't mean anything. Any vendor that takes their security serious should never trust those driver signatures.

SoftTalker 2y ago

I never assumed that driver signing was any kind of indicator of quality. It simply says "this is the Crowdstrike driver, it has not been modified" Maybe I'm wrong and Microsoft does some QA on drivers before they are signed?

rramadass 2y ago

Finally! You hit the nail right on the head!