17 ms·

Microsoft technical breakdown of CrowdStrike incident

415 points by nar001 2y ago

smcleod 2y ago
[flagged]
- frde_me 2y ago
  Depends on the frame of reference. Versus IOS it definitely is. Versus linux, probably not so.
  - pcwalton 2y ago
    It depends on what you mean by "open and flexible". Obviously, Windows NT isn't open source (though it has been heavily reverse engineered, and Microsoft publishes kernel symbols, which is more open than most proprietary software). But Windows does have a stable driver API and a license that permits publishing proprietary kernel-mode drivers, neither of which Linux has. Linux's position is that if you want to develop proprietary drivers, either develop them in user space or sandbox them with eBPF. This is arguably less flexible than Windows's policies, though the CrowdStrike incident certainly highlights the reliability benefits of Linux's approach.
- ethbr1 2y ago
  Microsoft needed some PR pushback against "Microsoft Windows outage" headlines. Which, I can't fault them for trying, considering Crowdstrike isn't their product...
  - dmattia 2y ago
    Agreed, I definitely understand their reasoning. But I also found it a bit funny that the word "CrowdStrike" appears 27 times in a fairly short blog
- ko_pivot 2y ago
  Yeah the insistence that Windows is insecure by default on purpose is a bit silly, but overall I thought this post was informative.
- aoopa 2y ago
  [flagged]
- graeme 2y ago
  That's actually the core of the issue. Windows gives users the capacity to run programs with Kernel access. The affected companies gave Crowdstrike kernel access. Crowdstrike crashed in Kernel, taking out the machines. This kind of issue would not be possible in Linux/Macos unless the user configures them to allow the same kind of direct kernel access. At least for macos this is off by default. Not certain about linux but I think the default is more locked down than Windows. Microsoft attempted to lock kernel access in 2009, the EU complained, so Microsoft kept it open in order to allow "competition in security vendors", and this is the result.
  - Rinzler89 2y ago
    >Microsoft attempted to lock kernel access in 2009, the EU complained And the EU complained only because the likes of McAfee complained. Microsoft here is "dammed if you do dammed if you don't" since so many of these security vendors make a living by being able to act like malware themselves in order to sell security. If Microsoft would lock them out of the kernel they'd all scream "anti-competitive" to the regulators.
    - deleted 2y ago
      [deleted]
ldjkfkdsjnv 2y ago
The true story is that I bet some major divisions of Crowdstrike are ran by non technical people that got there through non meritocratic means. Theres generally been no repercussions for their underperformance, much like boeing. Crowdstrike business is built on relationships, not technical supremacy. And bada bing bada boom, we have a complete failure of basic technical competency (no rigourous role out process).
- Paianni 2y ago
  All business are built on relationships, technical competency can but doesn't have to be a means to that end.
  - Wytwwww 2y ago
    > technical competency In a more fair world (that also valued economic productivity/growth more) companies which completely ignore that wouldn't survive, though.
- wiseowise 2y ago
  [flagged]
  - ldjkfkdsjnv 2y ago
    I try not to, catch me on the wrong day and it slips out
jacobgorm 2y ago
I used to work on Control Flow Integrity (CFI/XFI) research at places like MSR Silicon Valley and VMware, as far back as 2006. Back then, sandboxing a kernel module like ramdisk.sys was doable with a lot of binary rewriting magic, and later with custom LLVM passes, but nowadays it should be a simple matter of compiling the code with clang and the appropriate flags, to completely rule out this type of memory safety error, turning a BSOD into a polite log message and disabling the faulty driver.
- pcwalton 2y ago
  I mean, this is basically what eBPF accomplishes in Linux.
  - gclawes 2y ago
    There is eBPF for Windows: https://github.com/microsoft/ebpf-for-windows https://github.com/microsoft/ebpf-for-windows I'd hope security products in the future leverage this more than custom kernel-mode sensors.
    - capitainenemo 2y ago
      Was discussed on HN last week. Top comment notes the Windows support is still very limited. https://news.ycombinator.com/item?id=41033579 https://news.ycombinator.com/item?id=41033579
      - capitainenemo 2y ago
        ... oh. and the article by Brendan Gregg in the HN link above had the telling phrase: "Once Microsoft's eBPF support for Windows becomes production-ready"
- torginus 2y ago
  from what I understand, CrowdStrike has essentially put a Turing-complete interpreter for their scripting language into the kernel. I doubt you can do much when something is that general purpose.
janice1999 2y ago
At least they're not blaming the European Union in this breakdown (as they did earlier).
- zh3 2y ago
  Even this is written after multiple reviews by corporate lawyers.
- whimsicalism 2y ago
  they’re right though…
  - DarkNova6 2y ago
    Yes. Only Microsoft should be allowed to crash their operating system. Like back in the good old days when only MS could use their secret high-performance APIs.
    - graeme 2y ago
      Why exactly should security vendors have the ability to crash the operating system?
      - dmattia 2y ago
        They shouldn't. Microsoft should have APIs that enable security vendors to work in userspace. The EU didn't say that Microsoft couldn't kick vendors out of the kernel, just that they couldn't do so without having the APIs available that would let security vendors operate outside the kernel. Mac and Linux have such APIs, so CrowdStrike operates in user-mode on those platforms, so those platforms do not give security vendors the ability to crash the operating system.
- deleted 2y ago
  [deleted]
- strombofulous 2y ago
  Would this still have happened if the EU had not ruled against Microsoft?
rdtsc 2y ago
> We plan to work with the anti-malware ecosystem to take advantage of these integrated features to modernize their approach, helping to support and even increase security along with reliability. > Providing safe rollout guidance, best practices, and technologies to make it safer to perform updates to security products. > Reducing the need for kernel drivers to access important security data. They are being as diplomatic as they can, but it's definitely a slap to CS. Read as "they don't know how to roll things out, they need guidance on basic QA practices, we'll happily teach them...". Then, they list a set of facilities running in user-mode to avoid needing to run as many things in kernel mode. I would be interested what the water cooler discussion about CS was like inside Microsoft. Especially in teams needed to respond to customers about "Your windows OS is broken, our hospital patients are suffering...".
- notepad0x90 2y ago
  I must disagree with that take, your last quoted sentence is in response to all the supposed self-proclaimed experts asking "why does it need kernel access", the ones before that is to limit their own liability. What I've heard from people in the industry is not this silly "oh no, crowdstrike is so incompetent" b.s. that is being spread on sites like HN and reddit but more of an empathic "it could have been us" sentiment. In this write up as well, Microsoft knows they have caused their share of outages, it is a technical write-up but in part, it is to cover their bases for government investigations and lawsuits that will arise from this incident. And in part, they are also responsible for recovering from third-party driver errors and repeated boot failures caused by faulty drivers.
  - retrochameleon 2y ago
    CrowdStrike blamed their test software, but in the same breath revealed that they haven't been using any canary deployments. The bug that caused all this was present in their kernel driver for a long time. For being such a large cybersecurity player and deploying updates to 8.5 million devices, their quality control practices are embarrasingly lacking.
    - 2y ago
dmattia 2y ago
I suppose I was expecting something more authoritative here. They confirm that there was an attempted read-out-of-bounds, as CrowdStrike said, but that's not really new information at this point. I suppose we'll need to wait for more detailed analysis from CrowdStrike at some point. This post explains why security software has historically run in kernel-mode, and really seems to be pushing new technology that Microsoft has that would push security vendors into user-mode (with APIs that attempt to assist with many of the reasons why they have historically used kernel-mode). Crowdstrike already runs in user-mode on both Mac and Linux (from what I can tell), and it seems like running in user-mode on Windows would significantly lessen the risk of catastrophic failures like a blue-screen-of-death. I know the bulk of the failures here belong to CrowdStrike, but I can't help but think about the fact that Apple kicked security vendors out of kernel-mode a ways back, and that if Windows had done similarly, an issue like this probably wouldn't have been possible. By even offering kernel-mode options to external vendors, I believe Microsoft is creating risk for themselves.
- Rinzler89 2y ago
  > I can't help but think about the fact that Apple kicked security vendors out of kernel-mode a ways back, and that if Windows had done similarly, an issue like this probably wouldn't have been possible Like others already said, Microsoft already tried to do that with PatchGuard in 2006 with the launch of Windows Vista and the likes of Symantec and McAfee complained to the EU about this would harm the sales of their products, so the EU told Microsoft to not do it in 2009[1]. Apple has the luxury of a small market share on the desktop PC space to not attract the attention of the regulators, plus a user base that's used to Apple constantly rewriting the OS, deprecating APIs, switching CPU architectures, etc. without giving a fuck about breaking backwards compatibility or cutting off developers access to OS features their products use and getting away with it, luxuries that Microsoft doesn't have. IMHO, sticking with Window's default security and not using third party anit-malware has made Windows vastly more secure and rulabile than it was in the days when you'd be looking on installing the likes of Symantec or McAfee for your "protection" which ended up acting like malware after a while throwing dark patterns at you to milk more subsection fees, so as much as it hurts their sales, it's important for the regulators to understand that security is far more important than the regulations they put on Windows for Internet Explorer and Media Player and just like Apple's apps-store, it's sometimes better to let the original product maker handle security and not leave the product open at all points just so some of these bandits can make a living selling security for it. It's like foxes complaining to regulators how chicken wire is a threat to their existence. [1] https://stratechery.com/2024/crashes-and-competition/ https://stratechery.com/2024/crashes-and-competition/
akira2501 2y ago
> where security and availability are non-negotiable. Yep. You just have to pretend that everyone who deployed Windows had an actual competitive choice available to them. > A second benefit of loading into kernel mode is tamper resistance. I guess availability is negotiable after all.
- qsdf38100 2y ago
  > Yep. You just have to pretend that everyone who deployed Windows had an actual competitive choice available to them. Could you elaborate? How is that related to security and availability being non negotiable?
  - akira2501 2y ago
    Microsoft's statement implies that people choose Windows because of it's security and availability. Whereas most people end up with Windows because the software they want to run only operates on that single platform. The security and availability, to the extent they even exist, are clearly not part of the market's decision making process.
    - jojobas 2y ago
      The critical infrastructure that people actually cared about (ATC for example) had all the choice in the world. So did people designing bespoke POS systems. To rephrase the old IBM trope, "nobody got fired for choosing Windows".
      - akira2501 2y ago
        > critical infrastructure that people actually cared about Did hospitals have a choice in which OS their MRI machine runs? Are those not "critical?" Or should we just not "actually care?" > ATC for example Were they impacted by this outage? Isn't the reason flights were canceled is because the _carriers_ systems are the ones that had issues? > So did people designing bespoke POS systems. Really? I would assume most of that is down to the hardware, like cash drawers, credit card readers and order printers, which is likely third party and proprietary, and is only available and supported on Windows. Do you have evidence otherwise? > "nobody got fired for choosing Windows". You've recognized the same outcome I have but have gone to great lengths in an attempt to obscure the reasons for it happening. Why?
gjsman-1000 2y ago
Reminder that Microsoft could have programmed Windows to notice if a driver has caused a blue screen three times in a row, and prompt if you want to disable the driver on boot. After all, Windows already collects how many times a driver causes a crash. This would have made recovery one click instead of heading into Safe Mode and needing BitLocker keys. But they didn’t. And Microsoft, I argue, also has blood on their hands for every hospital this hit. Giving users a prompt to disable the driver, after three successive failed boots, would have saved lives.
- t-writescode 2y ago
  How would that have helped the server farms that were experiencing the issue?
  - gjsman-1000 2y ago
    Oh I don’t know, the servers down, you go and look as a technician, and you simply see a screen saying: “CSAgent.sys has caused a failure to boot three times in a row. Do you want to disable this driver? <Yes> <No>.” You click “Yes.” Server reboots with CloudStrike driver disabled. The day is saved in 5 minutes instead of building a custom ISO image or going on a BitLocker key recovery spree.
    - politelemon 2y ago
      It would still have required on site presence and interaction during which there is still downtime, so this accomplishes marginally small gains.
      - gjsman-1000 2y ago
        At the same time though, imagine you woke up and CloudStrike hit your organization. For most users, they’ll try clicking “Yes.” And then it’s back to work. After all, “No” just causes a blue screen again, might as well try the other path. This would have been the difference between the IT department handling 10,000+ calls or a few hundred (plus sending out a bulletin) in many, many organizations. It also could have saved billions at this point. Heck, it would have saved lives in hospitals.
zh3 2y ago
I do have to wonder how many agonising layers of review this went through with the marketing and legal departments as part of shifting the blame. If you want to decide which OS/distros to avoid for critical stuff, look to see who's learning from the incident (even if not bitten by it) compared to those saying "it wasn't our fault" (and that's not just MS).
EasyMark 2y ago
Oh I like this breakdown a lot. Fairly technical, links to resources used, flow of debug process, didn’t get lost in a the weeds of details and how clever they were. I wish more debug retrospectives were like this. It seems like you end up with 100 pages of analysis or a couple of vague paragraphs.
userbinator 2y ago
I'm going to be the controversial one here and say that, as bad as CrowdStrike was, the alternative of having only Microsoft be able to decide what people can do is far worse. I've already seen many others trying to use this incident to advocate for digital totalitarianism.
- scarface_74 2y ago
  Microsoft as the OS vendor will always be a potential source of updates that crash computers. Now with a third party, you’re adding another level of risk.
superposeur 2y ago
I’m surprised no one has yet noted that Microsoft itself is a chief CrowdStrike competitor.
- tonymet 2y ago
  i thought crowdstrike provided features that go beyond windows defender. is there another MS product that competes?
  - abhinavk 2y ago
    There is a paid version called Microsoft Defender for Endpoint.
  - superposeur 2y ago
    FWIW, here is CrowdStrike’s own comparison of features: https://www.crowdstrike.com/compare/crowdstrike-vs-microsoft-defender/ https://www.crowdstrike.com/compare/crowdstrike-vs-microsoft...
    - ryukoposting 2y ago
      Interesting that they bill Defender as "requiring frequent OS updates," alleging that their solution is somehow better in that regard. Are the suggesting that by installing CrowdStrike, you don't need to update Windows anymore? It really reads like that.
      - deleted 2y ago
        [deleted]
      - aaronmdjones 2y ago
        That's hilarious considering that it was a frequent CrowdStrike update that resulted in this chaos.
tonymet 2y ago
Did either release from MS or Crowdstrike explain how this crash bypassed QC? I'm still baffled that a 100% repro crash even made it anywhere near the later stages of QC. This is something easily caught by the earliest CI phases , at the developer and at least first build automation phase, let alone human QC.
- magicalhippo 2y ago
  From what I read in the previous thread, their test environment didn't actually test what was deployed. That is, there was a post-test pre-distribution packaging stage, and that's where the distributed file(s) got f'ed up. If true that would explain how it got past their testing, but would also be an incredible lack of competence IMHO. But yeah, curious if there's been some more concrete details there.
  - tonymet 2y ago
    I heard something similar. that they deploy content separately from code, but they don't test all of the combinations of code + content. This crash was from "stable" code in the driver mixed with a corrupt or incomplete content file (config, etc) , triggering the null-ptr exception . Sounds like one of those companies where you get hired and are shocked by the sausage factory you just stepped into
    - rvnx 2y ago
      In February they added new code that allows to spy/block named pipes. Named pipes are pipes of communication that processes can use to talk to each other, as an alternative to sockets. For example Chrome uses them between the user interface and the actual page renderer. In March they tested it in staging, said it was fine, pushed to prod with few rules in April, still looked fine. In July they added a new rule, which was deployed to 100% immediately, as from their perspective, a new entry in a database definition doesn't need testing nor canary deploy (which is still irresponsible, because bad rules could cause damage as well like any security/antivirus software, even if the parser didn't crash, but it could have blocked legitimate actions or files)
sammyteee 2y ago
I stopped reading after "Windows is an open and flexible platform"
DeathMetal3000 2y ago
“Windows has announced a commitment around the Rust programming language as part of Microsoft’s Secure Future Initiative (SFI) and has recently expanded the Windows kernel to support Rust.”
jarule 2y ago
[flagged]
squirrel 2y ago
Telling that there’s no mention of eBPF, which is standard on Linux and available on Windows, but hasn’t been brought into the main Windows OS. Static analysis might or might not have caught the Blue Friday bug, but it certainly increases the protection level over the current do-as-you-wish model for kernel modules.
eqvinox 2y ago
> Move tool-tip APIs from kernel to user mode ?!?!
Animats 2y ago
So how did this kernel level driver get through WHQL verification? The Static Driver Verifier should have caught this.[1] Do some security vendors get to bypass that? Microsoft is very quiet about that. That's the sort of thing a negligence lawyer focuses on. Partner at Brown Rudrick: "The most likely legal theory will be one of negligence. [Congress] will drag the guy over the coals, they'll maybe implicate him and his company and put in place a negligence action. There'll maybe be a couple of plaintiffs lawyers who dig up some exceptional theory on negligence, and get some class action lawsuits going. Again, we still don't know all the facts in this case, and there are other dimensions which have not yet been fully explored, including how CrowdStrike had access to kernel level updates on the Microsoft operating system? How come Microsoft didn't have any control over these updates being pushed on their kernel?" The first two class actions are already starting. [1] https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/static-driver-verifier https://learn.microsoft.com/en-us/windows-hardware/drivers/d... [2] https://www.channele2e.com/analysis/crowdstrike-legal-and-liability-implications-as-recovery-progresses https://www.channele2e.com/analysis/crowdstrike-legal-and-li...
- deleted 2y ago
  [deleted]
- meowkit 2y ago
  Because it wasn't an updated driver, it was a malformed blob config. https://www.youtube.com/watch?v=ZHrayP-Y71Q https://www.youtube.com/watch?v=ZHrayP-Y71Q https://www.youtube.com/watch?v=wAzEJxOo1ts https://www.youtube.com/watch?v=wAzEJxOo1ts That verification is for interactions with the OS. Its not going to catch driver specific exceptions.
  - Animats 2y ago
    If the driver can dereference nil, it shouldn't pass the Static Driver Verifier.[1] [1] https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/nullcheck https://learn.microsoft.com/en-us/windows-hardware/drivers/d...
    - 2y ago
someonehere 2y ago
Unless actually required by your org, choose the N -1 policy in CS to avoid snafus like this in the future. It’s in the console so use it.
- Maxious 2y ago
  The N-1 policy never applied and still does not apply to "Rapid Response Content" as outlined in the Preliminary Post Incident Review https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio...
waterTanuki 2y ago
I am still to this day gobsmacked how a company the size of Microsoft doesn't do all of it's security in-house like Apple, which locked down kernel access to macos some time ago. The blame is mostly on CrowdStrike, but Microsoft does share responsibility in allowing third-parties to pepper the kernel with whatever code they want to.
- breadwinner 2y ago
  Agree... but Microsoft is pointing the finger at EU (the same people that made you click "Accept cookies" on every website). https://www.neowin.net/news/microsoft-points-finger-at-the-eu-for-not-being-able-to-lock-down-windows/ https://www.neowin.net/news/microsoft-points-finger-at-the-e...
  - yawaramin 2y ago
    The EU didn't make you click 'Accept cookies' on every website, websites decided to interpret EU regulations as 'Let's make it super annoying for users to not accept, so they accept out of laziness'.
    - breadwinner 2y ago
      > websites decided to interpret EU regulations.. Does it matter? Ultimately, the impact to the citizenry is what matters.
      - ParetoOptimal 2y ago
        It does matter because you spin things to make it sound like the EU is anti-consumer when they are frequently the ones improving consumer protections and privacy.
- motohagiography 2y ago
  the trade off is either you make your own hardware, or make exceptions to integrate with other OEMs, which causes the fragmentation problem that security vendors exist to provide solutions for. Apple doesn't have that fragmentation problem, but they also don't have as rich an enterprise ecosystem. Android is the definition of a fragmentation problem, where instead of resolving it, Google manages it and calls it an ecosystem, and they're right. Fragmentation makes consistent governance/security impossible, but the heterogeneity also limits the scope of incidents. Apple balances its greater monoculture risk with deep control of the underlying hardware, to where as a user or developer you're only ever interacting with a very, very high level abstraction.
WalterBright 2y ago
What I heard is that CrowdStrike normally rate limits pushing a fix. This is so that if the fix is bad, the damage is limited. But for some reason, the rate limiter was turned off and the update went out to everyone.
- self_awareness 2y ago
  > What I heard is that CrowdStrike normally rate limits pushing a fix. This is so that if the fix is bad, the damage is limited. But for some reason, the rate limiter was turned off and the update went out to everyone. Is this true though? They've released a Post Incident article: https://www.crowdstrike.com/blog/falcon-content-update-preliminary-post-incident-report/ https://www.crowdstrike.com/blog/falcon-content-update-preli... in which they state: > How Do We Prevent This From Happening Again? Implement a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base, starting with a canary deployment. So it seems, if I understand this correctly, they've just implemented the rate limiter as a response to this incident.
  - WalterBright 2y ago
    Either version might be true!
aurelien 2y ago
You use a distribution made with foot for secretary and gamers and you blindly try to explain where the problem is. You are the clown's of the world, that's all ... xD
rldjbpin 2y ago
one thing from this whole fiasco that i wished bring to conversation was the fact that (crucial/market-dominant) digital/IT services don't have the same level of liability as mundane, physical goods. a simple plastic covering of your new dyson has more legal scrutiny and action (see the "children may choke" warnings they all need to come with) than software that we otherwise block in the name of "national security". given how much overvalued tech companies are in this region, i believe it is high time to start legally recognizing the real-life impact of digital tech. to hell with the "but muh innovation" argument.
magwa101 2y ago
[dead]

415 points by nar001 2y ago

smcleod 2y ago

[flagged]

frde_me 2y ago
Depends on the frame of reference. Versus IOS it definitely is. Versus linux, probably not so.
- pcwalton 2y ago
  It depends on what you mean by "open and flexible". Obviously, Windows NT isn't open source (though it has been heavily reverse engineered, and Microsoft publishes kernel symbols, which is more open than most proprietary software). But Windows does have a stable driver API and a license that permits publishing proprietary kernel-mode drivers, neither of which Linux has. Linux's position is that if you want to develop proprietary drivers, either develop them in user space or sandbox them with eBPF. This is arguably less flexible than Windows's policies, though the CrowdStrike incident certainly highlights the reliability benefits of Linux's approach.
ethbr1 2y ago
Microsoft needed some PR pushback against "Microsoft Windows outage" headlines. Which, I can't fault them for trying, considering Crowdstrike isn't their product...
- dmattia 2y ago
  Agreed, I definitely understand their reasoning. But I also found it a bit funny that the word "CrowdStrike" appears 27 times in a fairly short blog
ko_pivot 2y ago
Yeah the insistence that Windows is insecure by default on purpose is a bit silly, but overall I thought this post was informative.
aoopa 2y ago
[flagged]
graeme 2y ago
That's actually the core of the issue. Windows gives users the capacity to run programs with Kernel access. The affected companies gave Crowdstrike kernel access. Crowdstrike crashed in Kernel, taking out the machines. This kind of issue would not be possible in Linux/Macos unless the user configures them to allow the same kind of direct kernel access. At least for macos this is off by default. Not certain about linux but I think the default is more locked down than Windows. Microsoft attempted to lock kernel access in 2009, the EU complained, so Microsoft kept it open in order to allow "competition in security vendors", and this is the result.
- Rinzler89 2y ago
  >Microsoft attempted to lock kernel access in 2009, the EU complained And the EU complained only because the likes of McAfee complained. Microsoft here is "dammed if you do dammed if you don't" since so many of these security vendors make a living by being able to act like malware themselves in order to sell security. If Microsoft would lock them out of the kernel they'd all scream "anti-competitive" to the regulators.
  - deleted 2y ago
    [deleted]

ldjkfkdsjnv 2y ago

The true story is that I bet some major divisions of Crowdstrike are ran by non technical people that got there through non meritocratic means. Theres generally been no repercussions for their underperformance, much like boeing. Crowdstrike business is built on relationships, not technical supremacy. And bada bing bada boom, we have a complete failure of basic technical competency (no rigourous role out process).

Paianni 2y ago
All business are built on relationships, technical competency can but doesn't have to be a means to that end.
- Wytwwww 2y ago
  > technical competency In a more fair world (that also valued economic productivity/growth more) companies which completely ignore that wouldn't survive, though.
wiseowise 2y ago
[flagged]
- ldjkfkdsjnv 2y ago
  I try not to, catch me on the wrong day and it slips out

jacobgorm 2y ago

I used to work on Control Flow Integrity (CFI/XFI) research at places like MSR Silicon Valley and VMware, as far back as 2006. Back then, sandboxing a kernel module like ramdisk.sys was doable with a lot of binary rewriting magic, and later with custom LLVM passes, but nowadays it should be a simple matter of compiling the code with clang and the appropriate flags, to completely rule out this type of memory safety error, turning a BSOD into a polite log message and disabling the faulty driver.

pcwalton 2y ago
I mean, this is basically what eBPF accomplishes in Linux.
- gclawes 2y ago
  There is eBPF for Windows: https://github.com/microsoft/ebpf-for-windows https://github.com/microsoft/ebpf-for-windows I'd hope security products in the future leverage this more than custom kernel-mode sensors.
  - capitainenemo 2y ago
    Was discussed on HN last week. Top comment notes the Windows support is still very limited. https://news.ycombinator.com/item?id=41033579 https://news.ycombinator.com/item?id=41033579
    - capitainenemo 2y ago
      ... oh. and the article by Brendan Gregg in the HN link above had the telling phrase: "Once Microsoft's eBPF support for Windows becomes production-ready"
torginus 2y ago
from what I understand, CrowdStrike has essentially put a Turing-complete interpreter for their scripting language into the kernel. I doubt you can do much when something is that general purpose.

janice1999 2y ago

At least they're not blaming the European Union in this breakdown (as they did earlier).

zh3 2y ago
Even this is written after multiple reviews by corporate lawyers.
whimsicalism 2y ago
they’re right though…
- DarkNova6 2y ago
  Yes. Only Microsoft should be allowed to crash their operating system. Like back in the good old days when only MS could use their secret high-performance APIs.
  - graeme 2y ago
    Why exactly should security vendors have the ability to crash the operating system?
    - dmattia 2y ago
      They shouldn't. Microsoft should have APIs that enable security vendors to work in userspace. The EU didn't say that Microsoft couldn't kick vendors out of the kernel, just that they couldn't do so without having the APIs available that would let security vendors operate outside the kernel. Mac and Linux have such APIs, so CrowdStrike operates in user-mode on those platforms, so those platforms do not give security vendors the ability to crash the operating system.
deleted 2y ago
[deleted]
strombofulous 2y ago
Would this still have happened if the EU had not ruled against Microsoft?

rdtsc 2y ago

> We plan to work with the anti-malware ecosystem to take advantage of these integrated features to modernize their approach, helping to support and even increase security along with reliability. > Providing safe rollout guidance, best practices, and technologies to make it safer to perform updates to security products. > Reducing the need for kernel drivers to access important security data. They are being as diplomatic as they can, but it's definitely a slap to CS. Read as "they don't know how to roll things out, they need guidance on basic QA practices, we'll happily teach them...". Then, they list a set of facilities running in user-mode to avoid needing to run as many things in kernel mode. I would be interested what the water cooler discussion about CS was like inside Microsoft. Especially in teams needed to respond to customers about "Your windows OS is broken, our hospital patients are suffering...".

notepad0x90 2y ago
I must disagree with that take, your last quoted sentence is in response to all the supposed self-proclaimed experts asking "why does it need kernel access", the ones before that is to limit their own liability. What I've heard from people in the industry is not this silly "oh no, crowdstrike is so incompetent" b.s. that is being spread on sites like HN and reddit but more of an empathic "it could have been us" sentiment. In this write up as well, Microsoft knows they have caused their share of outages, it is a technical write-up but in part, it is to cover their bases for government investigations and lawsuits that will arise from this incident. And in part, they are also responsible for recovering from third-party driver errors and repeated boot failures caused by faulty drivers.
- retrochameleon 2y ago
  CrowdStrike blamed their test software, but in the same breath revealed that they haven't been using any canary deployments. The bug that caused all this was present in their kernel driver for a long time. For being such a large cybersecurity player and deploying updates to 8.5 million devices, their quality control practices are embarrasingly lacking.
  - 2y ago

dmattia 2y ago

I suppose I was expecting something more authoritative here. They confirm that there was an attempted read-out-of-bounds, as CrowdStrike said, but that's not really new information at this point. I suppose we'll need to wait for more detailed analysis from CrowdStrike at some point. This post explains why security software has historically run in kernel-mode, and really seems to be pushing new technology that Microsoft has that would push security vendors into user-mode (with APIs that attempt to assist with many of the reasons why they have historically used kernel-mode). Crowdstrike already runs in user-mode on both Mac and Linux (from what I can tell), and it seems like running in user-mode on Windows would significantly lessen the risk of catastrophic failures like a blue-screen-of-death. I know the bulk of the failures here belong to CrowdStrike, but I can't help but think about the fact that Apple kicked security vendors out of kernel-mode a ways back, and that if Windows had done similarly, an issue like this probably wouldn't have been possible. By even offering kernel-mode options to external vendors, I believe Microsoft is creating risk for themselves.

Rinzler89 2y ago
> I can't help but think about the fact that Apple kicked security vendors out of kernel-mode a ways back, and that if Windows had done similarly, an issue like this probably wouldn't have been possible Like others already said, Microsoft already tried to do that with PatchGuard in 2006 with the launch of Windows Vista and the likes of Symantec and McAfee complained to the EU about this would harm the sales of their products, so the EU told Microsoft to not do it in 2009[1]. Apple has the luxury of a small market share on the desktop PC space to not attract the attention of the regulators, plus a user base that's used to Apple constantly rewriting the OS, deprecating APIs, switching CPU architectures, etc. without giving a fuck about breaking backwards compatibility or cutting off developers access to OS features their products use and getting away with it, luxuries that Microsoft doesn't have. IMHO, sticking with Window's default security and not using third party anit-malware has made Windows vastly more secure and rulabile than it was in the days when you'd be looking on installing the likes of Symantec or McAfee for your "protection" which ended up acting like malware after a while throwing dark patterns at you to milk more subsection fees, so as much as it hurts their sales, it's important for the regulators to understand that security is far more important than the regulations they put on Windows for Internet Explorer and Media Player and just like Apple's apps-store, it's sometimes better to let the original product maker handle security and not leave the product open at all points just so some of these bandits can make a living selling security for it. It's like foxes complaining to regulators how chicken wire is a threat to their existence. [1] https://stratechery.com/2024/crashes-and-competition/ https://stratechery.com/2024/crashes-and-competition/

akira2501 2y ago

> where security and availability are non-negotiable. Yep. You just have to pretend that everyone who deployed Windows had an actual competitive choice available to them. > A second benefit of loading into kernel mode is tamper resistance. I guess availability is negotiable after all.

qsdf38100 2y ago
> Yep. You just have to pretend that everyone who deployed Windows had an actual competitive choice available to them. Could you elaborate? How is that related to security and availability being non negotiable?
- akira2501 2y ago
  Microsoft's statement implies that people choose Windows because of it's security and availability. Whereas most people end up with Windows because the software they want to run only operates on that single platform. The security and availability, to the extent they even exist, are clearly not part of the market's decision making process.
  - jojobas 2y ago
    The critical infrastructure that people actually cared about (ATC for example) had all the choice in the world. So did people designing bespoke POS systems. To rephrase the old IBM trope, "nobody got fired for choosing Windows".
    - akira2501 2y ago
      > critical infrastructure that people actually cared about Did hospitals have a choice in which OS their MRI machine runs? Are those not "critical?" Or should we just not "actually care?" > ATC for example Were they impacted by this outage? Isn't the reason flights were canceled is because the _carriers_ systems are the ones that had issues? > So did people designing bespoke POS systems. Really? I would assume most of that is down to the hardware, like cash drawers, credit card readers and order printers, which is likely third party and proprietary, and is only available and supported on Windows. Do you have evidence otherwise? > "nobody got fired for choosing Windows". You've recognized the same outcome I have but have gone to great lengths in an attempt to obscure the reasons for it happening. Why?

gjsman-1000 2y ago

Reminder that Microsoft could have programmed Windows to notice if a driver has caused a blue screen three times in a row, and prompt if you want to disable the driver on boot. After all, Windows already collects how many times a driver causes a crash. This would have made recovery one click instead of heading into Safe Mode and needing BitLocker keys. But they didn’t. And Microsoft, I argue, also has blood on their hands for every hospital this hit. Giving users a prompt to disable the driver, after three successive failed boots, would have saved lives.

t-writescode 2y ago
How would that have helped the server farms that were experiencing the issue?
- gjsman-1000 2y ago
  Oh I don’t know, the servers down, you go and look as a technician, and you simply see a screen saying: “CSAgent.sys has caused a failure to boot three times in a row. Do you want to disable this driver? <Yes> <No>.” You click “Yes.” Server reboots with CloudStrike driver disabled. The day is saved in 5 minutes instead of building a custom ISO image or going on a BitLocker key recovery spree.
  - politelemon 2y ago
    It would still have required on site presence and interaction during which there is still downtime, so this accomplishes marginally small gains.
    - gjsman-1000 2y ago
      At the same time though, imagine you woke up and CloudStrike hit your organization. For most users, they’ll try clicking “Yes.” And then it’s back to work. After all, “No” just causes a blue screen again, might as well try the other path. This would have been the difference between the IT department handling 10,000+ calls or a few hundred (plus sending out a bulletin) in many, many organizations. It also could have saved billions at this point. Heck, it would have saved lives in hospitals.

zh3 2y ago

I do have to wonder how many agonising layers of review this went through with the marketing and legal departments as part of shifting the blame. If you want to decide which OS/distros to avoid for critical stuff, look to see who's learning from the incident (even if not bitten by it) compared to those saying "it wasn't our fault" (and that's not just MS).

EasyMark 2y ago

Oh I like this breakdown a lot. Fairly technical, links to resources used, flow of debug process, didn’t get lost in a the weeds of details and how clever they were. I wish more debug retrospectives were like this. It seems like you end up with 100 pages of analysis or a couple of vague paragraphs.

userbinator 2y ago

I'm going to be the controversial one here and say that, as bad as CrowdStrike was, the alternative of having only Microsoft be able to decide what people can do is far worse. I've already seen many others trying to use this incident to advocate for digital totalitarianism.

scarface_74 2y ago
Microsoft as the OS vendor will always be a potential source of updates that crash computers. Now with a third party, you’re adding another level of risk.

superposeur 2y ago

I’m surprised no one has yet noted that Microsoft itself is a chief CrowdStrike competitor.

tonymet 2y ago
i thought crowdstrike provided features that go beyond windows defender. is there another MS product that competes?
- abhinavk 2y ago
  There is a paid version called Microsoft Defender for Endpoint.
- superposeur 2y ago
  FWIW, here is CrowdStrike’s own comparison of features: https://www.crowdstrike.com/compare/crowdstrike-vs-microsoft-defender/ https://www.crowdstrike.com/compare/crowdstrike-vs-microsoft...
  - ryukoposting 2y ago
    Interesting that they bill Defender as "requiring frequent OS updates," alleging that their solution is somehow better in that regard. Are the suggesting that by installing CrowdStrike, you don't need to update Windows anymore? It really reads like that.
    - deleted 2y ago
      [deleted]
    - aaronmdjones 2y ago
      That's hilarious considering that it was a frequent CrowdStrike update that resulted in this chaos.

tonymet 2y ago

Did either release from MS or Crowdstrike explain how this crash bypassed QC? I'm still baffled that a 100% repro crash even made it anywhere near the later stages of QC. This is something easily caught by the earliest CI phases , at the developer and at least first build automation phase, let alone human QC.

magicalhippo 2y ago
From what I read in the previous thread, their test environment didn't actually test what was deployed. That is, there was a post-test pre-distribution packaging stage, and that's where the distributed file(s) got f'ed up. If true that would explain how it got past their testing, but would also be an incredible lack of competence IMHO. But yeah, curious if there's been some more concrete details there.
- tonymet 2y ago
  I heard something similar. that they deploy content separately from code, but they don't test all of the combinations of code + content. This crash was from "stable" code in the driver mixed with a corrupt or incomplete content file (config, etc) , triggering the null-ptr exception . Sounds like one of those companies where you get hired and are shocked by the sausage factory you just stepped into
  - rvnx 2y ago
    In February they added new code that allows to spy/block named pipes. Named pipes are pipes of communication that processes can use to talk to each other, as an alternative to sockets. For example Chrome uses them between the user interface and the actual page renderer. In March they tested it in staging, said it was fine, pushed to prod with few rules in April, still looked fine. In July they added a new rule, which was deployed to 100% immediately, as from their perspective, a new entry in a database definition doesn't need testing nor canary deploy (which is still irresponsible, because bad rules could cause damage as well like any security/antivirus software, even if the parser didn't crash, but it could have blocked legitimate actions or files)

sammyteee 2y ago

I stopped reading after "Windows is an open and flexible platform"

DeathMetal3000 2y ago

“Windows has announced a commitment around the Rust programming language as part of Microsoft’s Secure Future Initiative (SFI) and has recently expanded the Windows kernel to support Rust.”

jarule 2y ago

[flagged]

squirrel 2y ago

Telling that there’s no mention of eBPF, which is standard on Linux and available on Windows, but hasn’t been brought into the main Windows OS. Static analysis might or might not have caught the Blue Friday bug, but it certainly increases the protection level over the current do-as-you-wish model for kernel modules.

eqvinox 2y ago

> Move tool-tip APIs from kernel to user mode ?!?!

Animats 2y ago

So how did this kernel level driver get through WHQL verification? The Static Driver Verifier should have caught this.[1] Do some security vendors get to bypass that? Microsoft is very quiet about that. That's the sort of thing a negligence lawyer focuses on. Partner at Brown Rudrick: "The most likely legal theory will be one of negligence. [Congress] will drag the guy over the coals, they'll maybe implicate him and his company and put in place a negligence action. There'll maybe be a couple of plaintiffs lawyers who dig up some exceptional theory on negligence, and get some class action lawsuits going. Again, we still don't know all the facts in this case, and there are other dimensions which have not yet been fully explored, including how CrowdStrike had access to kernel level updates on the Microsoft operating system? How come Microsoft didn't have any control over these updates being pushed on their kernel?" The first two class actions are already starting. [1] https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/static-driver-verifier https://learn.microsoft.com/en-us/windows-hardware/drivers/d... [2] https://www.channele2e.com/analysis/crowdstrike-legal-and-liability-implications-as-recovery-progresses https://www.channele2e.com/analysis/crowdstrike-legal-and-li...

deleted 2y ago
[deleted]
meowkit 2y ago
Because it wasn't an updated driver, it was a malformed blob config. https://www.youtube.com/watch?v=ZHrayP-Y71Q https://www.youtube.com/watch?v=ZHrayP-Y71Q https://www.youtube.com/watch?v=wAzEJxOo1ts https://www.youtube.com/watch?v=wAzEJxOo1ts That verification is for interactions with the OS. Its not going to catch driver specific exceptions.
- Animats 2y ago
  If the driver can dereference nil, it shouldn't pass the Static Driver Verifier.[1] [1] https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/nullcheck https://learn.microsoft.com/en-us/windows-hardware/drivers/d...
  - 2y ago

someonehere 2y ago

Unless actually required by your org, choose the N -1 policy in CS to avoid snafus like this in the future. It’s in the console so use it.

Maxious 2y ago
The N-1 policy never applied and still does not apply to "Rapid Response Content" as outlined in the Preliminary Post Incident Review https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio...

waterTanuki 2y ago

I am still to this day gobsmacked how a company the size of Microsoft doesn't do all of it's security in-house like Apple, which locked down kernel access to macos some time ago. The blame is mostly on CrowdStrike, but Microsoft does share responsibility in allowing third-parties to pepper the kernel with whatever code they want to.

breadwinner 2y ago
Agree... but Microsoft is pointing the finger at EU (the same people that made you click "Accept cookies" on every website). https://www.neowin.net/news/microsoft-points-finger-at-the-eu-for-not-being-able-to-lock-down-windows/ https://www.neowin.net/news/microsoft-points-finger-at-the-e...
- yawaramin 2y ago
  The EU didn't make you click 'Accept cookies' on every website, websites decided to interpret EU regulations as 'Let's make it super annoying for users to not accept, so they accept out of laziness'.
  - breadwinner 2y ago
    > websites decided to interpret EU regulations.. Does it matter? Ultimately, the impact to the citizenry is what matters.
    - ParetoOptimal 2y ago
      It does matter because you spin things to make it sound like the EU is anti-consumer when they are frequently the ones improving consumer protections and privacy.
motohagiography 2y ago
the trade off is either you make your own hardware, or make exceptions to integrate with other OEMs, which causes the fragmentation problem that security vendors exist to provide solutions for. Apple doesn't have that fragmentation problem, but they also don't have as rich an enterprise ecosystem. Android is the definition of a fragmentation problem, where instead of resolving it, Google manages it and calls it an ecosystem, and they're right. Fragmentation makes consistent governance/security impossible, but the heterogeneity also limits the scope of incidents. Apple balances its greater monoculture risk with deep control of the underlying hardware, to where as a user or developer you're only ever interacting with a very, very high level abstraction.

WalterBright 2y ago

What I heard is that CrowdStrike normally rate limits pushing a fix. This is so that if the fix is bad, the damage is limited. But for some reason, the rate limiter was turned off and the update went out to everyone.

self_awareness 2y ago
> What I heard is that CrowdStrike normally rate limits pushing a fix. This is so that if the fix is bad, the damage is limited. But for some reason, the rate limiter was turned off and the update went out to everyone. Is this true though? They've released a Post Incident article: https://www.crowdstrike.com/blog/falcon-content-update-preliminary-post-incident-report/ https://www.crowdstrike.com/blog/falcon-content-update-preli... in which they state: > How Do We Prevent This From Happening Again? Implement a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base, starting with a canary deployment. So it seems, if I understand this correctly, they've just implemented the rate limiter as a response to this incident.
- WalterBright 2y ago
  Either version might be true!

aurelien 2y ago

You use a distribution made with foot for secretary and gamers and you blindly try to explain where the problem is. You are the clown's of the world, that's all ... xD

rldjbpin 2y ago

one thing from this whole fiasco that i wished bring to conversation was the fact that (crucial/market-dominant) digital/IT services don't have the same level of liability as mundane, physical goods. a simple plastic covering of your new dyson has more legal scrutiny and action (see the "children may choke" warnings they all need to come with) than software that we otherwise block in the name of "national security". given how much overvalued tech companies are in this region, i believe it is high time to start legally recognizing the real-life impact of digital tech. to hell with the "but muh innovation" argument.

magwa101 2y ago

[dead]