19 ms·

> just like blaming microsoft for the crowdstrike bsod is stupid Wait, how is this stupid? Unless I'm missing something, wasn't the patch part of a Microsoft p

by mbesto 2y ago

> just like blaming microsoft for the crowdstrike bsod is stupid Wait, how is this stupid? Unless I'm missing something, wasn't the patch part of a Microsoft payload that included an update to Crowdstrike? Surely Crowdstrike is culpable, but that doesn't completely absolve Microsoft of any responsibility, as its their payload.

roblabla 2y ago
Do you have a source for this? It's the first time I hear of this. From what I've understood (perhaps wrongly), the error came from the CrowdStrike driver (csagent.sys) having bugs in their configuration parser that could cause it to BSOD. CrowdStrike pushed a corrupted configuration (the CS-000whatever.sys we're told to delete) that hit that bug. I'm not sure how Microsoft fits in this story.
- mbesto 2y ago
  Just read more into it. You're correct. I think it would be dumb to solely blame MS, but I don't think you can completely absolve them. this comment right here sums it up: > Sure, but Windows shares some portion of the blame for allowing third-party security vendors to “shit in the kernel”. https://news.ycombinator.com/item?id=41006176 https://news.ycombinator.com/item?id=41006176
  - roblabla 2y ago
    Yeah, the fact that Windows requires kernel-level access to be able to do EDR stuff is really unfortunate. MacOS has been very successful with their userspace EndpointSecurity Framework for this purpose. On the other hand, Linux is similarly crippled: eBPF LSM are fairly recent and don't work everywhere (I'm looking at you Ubuntu[0]), and the only real alternative if you want to be able to block processes is a kernel module. Which comes with the same dangers as Windows. [0]: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2054810 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2054810
    - sgift 2y ago
      Well, Microsoft says that's because of the EU commission: https://news.ycombinator.com/item?id=41029590 https://news.ycombinator.com/item?id=41029590 I have my doubts, but that's at least what they give as the reason for the kernel-level access of EDR tools.
angulardragon03 2y ago
You’re missing something. The Crowdstrike issue was caused by a channel update (basically a definitions update) that they pushed that broke their own sensor. Microsoft wasn’t involved in the delivery of that update.
ninepoints 2y ago
You're missing something. Many somethings.
GuB-42 2y ago
What I understand is that some Azure VMs are running CrowdStrike, and like any other computer running CrowdStrike on Windows, they crashed. Totally not Microsoft's fault, CrowdStrike messed with the kernel, the only thing we can blame Microsoft on is to allow such a software to exist. Where Microsoft is to blame however is the unrelated Azure outage in the Central US region that happened (and was fixed) just before the CrowdStrike faulty update.
sschueller 2y ago
Microsoft should revoke the CrowdStrike driver signature and should do an internal check as to why CrowdStrike's driver was approved when it can execute arbitrary code on the kernel level without any checks. If your "driver" requires this feature MS should require CrowdStrike to submit the entire source and they should have to pay MS to do a review of the code. What is the point of driver signing if a vendor can basically build in a back door and Microsoft doesn't validate that this back door is at least somewhat reasonable
- _flux 2y ago
  Do you think Microsoft customers using CrowdStrike would then be happier, being unable to run the software at all, due to an action Microsoft took? Backdoors of all kinds can be installed to most any operating system without vendor co-operation. That is the nature of general-purpose operating systems.
  - sigseg1v 2y ago
    I'm a customer that is forced to use CrowdStrike via IT policies and I would be giddy with delight if something came along and caused the removal of it from my systems. I don't need programs sitting on my computer preventing me from installing code that I've literally just compiled, preventing me from deleting or modifying folders on my machine, and causing extreme lag for many basic system operations even when it does work. At this point, the time in lost productivity (via normal operation) and downtime (via their recent bug) has easily exceeded a thousand times over the aggregate sum of all benefits that CrowdStrike will ever have provided from threat detection and prevention. It's time to remove the malware.
    - _flux 2y ago
      You are not the customer, though, your employer is the customer. Perhaps you should push this change up in the food chain, then, and if the company is good the request will be taken seriously. As I understand it, while CrowdStrike is the biggest name in EDR, it's far from the only one, if that's what your company requires to pass some checkboxes in certifications.
PedroBatista 2y ago
The "only" thing Microsoft should be blame is for signing a driver updates itself by taking "configurations ( code )" from userspace and turning a blind eye to all these loopholes because they also know it's not practical for them to sign all the driver code that goes into the kernel. Maybe most of these "drivers" shouldn't be in Ring0 to begin with? This is a general problem and the norm, Windows is just another OS that allows this this way.
- dathinab 2y ago
  it's not that simple user-land drivers are a thing, heck they are the standard for modern micro kernel architectures and even with hybrid kernels pushing part of the code out of the kernel into something like "user land co-processes" is more then doable, now it's now trivial to retrofit in a performant way and flexible way but possible Mac has somewhat done that (but I don't know the details). On Linux it's also possible, through with BPF a bit of a in-between hybrid (leaving some parts of the drivers in kernel, but as BPF programs which are much less likely to cause such issues compared to "normal" drivers). A good example for that is how graphic drivers have developed on Linux, with most code now being in a) the user-land part of the driver and b) on the GPU itself leaving the in kernel part to be mostly just memory management. And the thing is Windows has not enforced such direction, or even pushed hard for it AFIK, and that is something you can very well blame then for. You in general should not have a complicated config file parser in a kernel driver, that's just a terrible idea, some would say negligent and Windows shouldn't have certified drivers like that. (But then given that CrowdStrike insists that it _must_ be loaded on start (outside of recovery mode) I guess it would still have hung all systems even if the parsing would have been outsourced because it can't start if it can't parse it's config).
  - traject_ 2y ago
    > And the thing is Windows has not enforced such direction, or even pushed hard for it AFIK, and that is something you can very well blame then for. Even here it's pretty hard to blame them due to antitrust concerns. Just google the word Patchguard.
beAbU 2y ago
Crowdstrike pushed out a world-wide update to their client software, which auto-updated itself. This update was buggy and it caused the host machine to go into a BSOD boot loop. The fact that the host machine happened to be running Windows has very little to nothing to do with it. It's like blaming a pothole for your car going exploding. Yes there was a pothole, yes it shouldn't have been there, yes it could have been avoided, but the fact that your car self immolated because of mere pothole points to possibly other underlying issues with your car.

by mbesto 2y ago

roblabla 2y ago

Do you have a source for this? It's the first time I hear of this. From what I've understood (perhaps wrongly), the error came from the CrowdStrike driver (csagent.sys) having bugs in their configuration parser that could cause it to BSOD. CrowdStrike pushed a corrupted configuration (the CS-000whatever.sys we're told to delete) that hit that bug. I'm not sure how Microsoft fits in this story.

mbesto 2y ago
Just read more into it. You're correct. I think it would be dumb to solely blame MS, but I don't think you can completely absolve them. this comment right here sums it up: > Sure, but Windows shares some portion of the blame for allowing third-party security vendors to “shit in the kernel”. https://news.ycombinator.com/item?id=41006176 https://news.ycombinator.com/item?id=41006176
- roblabla 2y ago
  Yeah, the fact that Windows requires kernel-level access to be able to do EDR stuff is really unfortunate. MacOS has been very successful with their userspace EndpointSecurity Framework for this purpose. On the other hand, Linux is similarly crippled: eBPF LSM are fairly recent and don't work everywhere (I'm looking at you Ubuntu[0]), and the only real alternative if you want to be able to block processes is a kernel module. Which comes with the same dangers as Windows. [0]: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2054810 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2054810
  - sgift 2y ago
    Well, Microsoft says that's because of the EU commission: https://news.ycombinator.com/item?id=41029590 https://news.ycombinator.com/item?id=41029590 I have my doubts, but that's at least what they give as the reason for the kernel-level access of EDR tools.

angulardragon03 2y ago

You’re missing something. The Crowdstrike issue was caused by a channel update (basically a definitions update) that they pushed that broke their own sensor. Microsoft wasn’t involved in the delivery of that update.

ninepoints 2y ago

You're missing something. Many somethings.

GuB-42 2y ago

What I understand is that some Azure VMs are running CrowdStrike, and like any other computer running CrowdStrike on Windows, they crashed. Totally not Microsoft's fault, CrowdStrike messed with the kernel, the only thing we can blame Microsoft on is to allow such a software to exist. Where Microsoft is to blame however is the unrelated Azure outage in the Central US region that happened (and was fixed) just before the CrowdStrike faulty update.

sschueller 2y ago

Microsoft should revoke the CrowdStrike driver signature and should do an internal check as to why CrowdStrike's driver was approved when it can execute arbitrary code on the kernel level without any checks. If your "driver" requires this feature MS should require CrowdStrike to submit the entire source and they should have to pay MS to do a review of the code. What is the point of driver signing if a vendor can basically build in a back door and Microsoft doesn't validate that this back door is at least somewhat reasonable

_flux 2y ago
Do you think Microsoft customers using CrowdStrike would then be happier, being unable to run the software at all, due to an action Microsoft took? Backdoors of all kinds can be installed to most any operating system without vendor co-operation. That is the nature of general-purpose operating systems.
- sigseg1v 2y ago
  I'm a customer that is forced to use CrowdStrike via IT policies and I would be giddy with delight if something came along and caused the removal of it from my systems. I don't need programs sitting on my computer preventing me from installing code that I've literally just compiled, preventing me from deleting or modifying folders on my machine, and causing extreme lag for many basic system operations even when it does work. At this point, the time in lost productivity (via normal operation) and downtime (via their recent bug) has easily exceeded a thousand times over the aggregate sum of all benefits that CrowdStrike will ever have provided from threat detection and prevention. It's time to remove the malware.
  - _flux 2y ago
    You are not the customer, though, your employer is the customer. Perhaps you should push this change up in the food chain, then, and if the company is good the request will be taken seriously. As I understand it, while CrowdStrike is the biggest name in EDR, it's far from the only one, if that's what your company requires to pass some checkboxes in certifications.

PedroBatista 2y ago

The "only" thing Microsoft should be blame is for signing a driver updates itself by taking "configurations ( code )" from userspace and turning a blind eye to all these loopholes because they also know it's not practical for them to sign all the driver code that goes into the kernel. Maybe most of these "drivers" shouldn't be in Ring0 to begin with? This is a general problem and the norm, Windows is just another OS that allows this this way.

dathinab 2y ago
it's not that simple user-land drivers are a thing, heck they are the standard for modern micro kernel architectures and even with hybrid kernels pushing part of the code out of the kernel into something like "user land co-processes" is more then doable, now it's now trivial to retrofit in a performant way and flexible way but possible Mac has somewhat done that (but I don't know the details). On Linux it's also possible, through with BPF a bit of a in-between hybrid (leaving some parts of the drivers in kernel, but as BPF programs which are much less likely to cause such issues compared to "normal" drivers). A good example for that is how graphic drivers have developed on Linux, with most code now being in a) the user-land part of the driver and b) on the GPU itself leaving the in kernel part to be mostly just memory management. And the thing is Windows has not enforced such direction, or even pushed hard for it AFIK, and that is something you can very well blame then for. You in general should not have a complicated config file parser in a kernel driver, that's just a terrible idea, some would say negligent and Windows shouldn't have certified drivers like that. (But then given that CrowdStrike insists that it _must_ be loaded on start (outside of recovery mode) I guess it would still have hung all systems even if the parsing would have been outsourced because it can't start if it can't parse it's config).
- traject_ 2y ago
  > And the thing is Windows has not enforced such direction, or even pushed hard for it AFIK, and that is something you can very well blame then for. Even here it's pretty hard to blame them due to antitrust concerns. Just google the word Patchguard.

beAbU 2y ago

Crowdstrike pushed out a world-wide update to their client software, which auto-updated itself. This update was buggy and it caused the host machine to go into a BSOD boot loop. The fact that the host machine happened to be running Windows has very little to nothing to do with it. It's like blaming a pothole for your car going exploding. Yes there was a pothole, yes it shouldn't have been there, yes it could have been avoided, but the fact that your car self immolated because of mere pothole points to possibly other underlying issues with your car.