8 ms·

CrowdStrike's outage should not have happened

49 points by b-man 2y ago

echelon 2y ago
There is no reason to not use Rust for these systems anymore. It's why the US Government is pushing so much for Rust adoption. We're going to keep seeing these horror stories until C/C++ go away.
- victor106 2y ago
  > It's why the US Government is pushing so much for Rust adoption. Can you provide any references for this?
  - timschmidt 2y ago
    He's likely referring to this: https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/press-release-technical-report/ https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/pre...
  - RockRobotRock 2y ago
    https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/press-release-technical-report/ https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/pre... https://media.defense.gov/2022/Nov/10/2003112742/-1/-1/0/CSI_SOFTWARE_MEMORY_SAFETY.PDF https://media.defense.gov/2022/Nov/10/2003112742/-1/-1/0/CSI...
  - kevindamm 2y ago
    They may be referring to DARPA's research project TRACTOR: https://www.darpa.mil/program/translating-all-c-to-rust https://www.darpa.mil/program/translating-all-c-to-rust
  - AlotOfReading 2y ago
    There's been a number of reports released by the government that recommend adopting memory safe languages as opposed to C/C++. The White House put out a memo [0] for example, and NIST has this page [1]. Since there are no other systems languages with widespread developer mindshare, some people see these as primarily recommending Rust. [0] https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/memory-safety-statements-of-support/ https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/mem... [1] https://www.nist.gov/itl/ssd/software-quality-group/safer-languages https://www.nist.gov/itl/ssd/software-quality-group/safer-la...
    - jimbob45 2y ago
      What about MISRA C and C++ with static analysis tools? Also all GC languages would pretty easily solve the problem including Java and C#.
      - AlotOfReading 2y ago
        It's perfectly possible to write the crowdstrike issue in C/C++ code that will pass any MISRA checker, because MISRA checkers are unable to verify conformance with the standard. It basically says "don't do this" and leaves it up to the developer to ensure. Static analysis tools (broadly) are not powerful enough to ensure memory safety for nontrivial programs without a high developer burden in the form of annotations or restricted language subsets. GC languages are generally not appropriate for kernel level systems programming like crowdstrike was doing. Java has some history in this space, but generally without niceties like garbage collection that typical developers are used to.
    - echelon 2y ago
      > some people see these as primarily recommending Rust. That's not the reason. The DoD really is focused on Rust [1, 2]. [1] https://www.darpa.mil/program/translating-all-c-to-rust https://www.darpa.mil/program/translating-all-c-to-rust [2] https://www.darpa.mil/news-events/2024-07-31a https://www.darpa.mil/news-events/2024-07-31a
      - AlotOfReading 2y ago
        The federal government is a lot bigger than even the DoD, and the other agencies (e.g. NIST as linked above) have been very careful to make sure they aren't only recommending Rust, even if it's clearly a major alternative.
  - notarealllama 2y ago
    If you follow CISA vulns and guidelines, it's all industrial controllers getting hacked and pushing for memory safety for like a year straight. (On a side note, pleasantly surprised to see a Linux kernel android bug today)
  - keithalewis 2y ago
    We're from the Government and we're here to help. You need a license to operate a backhoe, but not to write software.
- AdieuToLogic 2y ago
  > There is no reason to not use Rust for these systems anymore. While Rust is a fine language, there are others which may be better suited for systems which have a global impact such as CrowdStrike does. Ada: https://en.wikipedia.org/wiki/Ada_(programming_language) https://en.wikipedia.org/wiki/Ada_(programming_language) SPARK: https://en.wikipedia.org/wiki/SPARK_(programming_language) https://en.wikipedia.org/wiki/SPARK_(programming_language) I am an advocate for neither and only reference their applicability for this problem domain based on the languages' design intent.
- MaulingMonkey 2y ago
  >> According to the RCA, the essence of what happened was an index out of bounds While Rust's behavior of panicing is saner than C++'s behavior of undefined behavior for a typical index out of bounds bug (`slice[index]`), I would imagine it'd still result in BSODs and outages in a kernel context. Admittedly, safe bounds checking (`slice.get(index) → Option<T>`) might be easier in Rust than in C++, but it's not a given that a RIIR would've avoided this bug.
- MBCook 2y ago
  Oh who cares about Rust. This could have been written in many different languages safely, including C/C++. Would it have been easier in Rust? Maybe. But it's also a legacy codebase. So we shouldn't be surprised it wasn't in Rust. There are SO MANY things that went wrong. They don't even seem to care about half of them. Ok, it wasn't tested. It wasn't staged. It wasn't a slow roll-out over hours. It wasn't verified from the deployment servers. How about why is it parsed in kernel space at all? I didn't see that listed. Why didn't you have a mechanism to catch faulty updates at runtime and flag them so that you'd be able to survive the next boot? Mark somewhere on the FS "I'm loading #19583". If you boot up and find that, it means you failed, so skip it next boot. Succeed? Remove the mark. Tada. Resiliency. Why are you even doing this stuff yourself? There are Microsoft APIs for a bunch of this stuff that would have helped reduce risk. Oh, because your code is deployed on XP and up and you don't want to have 2 or 3 versions? That went well didn't it. I bet if you were using those you wouldn't have crashed the kernel space. Do you fuzz? We know you don't. You broke Windows. You broke Macs. You broke Linux. You keep doing this. You're not good at being resilient. You had to face-plant in front of the entire world before you decided to do something. I didn't see fuzzing mentioned. Those mitigations you propose are not enough. They'll help, but you're still vulnerable. You're supposed to be a _security company_, I'd expect a better list of how to secure software from a security company. By all means, use Rust or Swift or (insert other memory safe language) if you want. That's smart. But if their existing kernel module had no UB and buffer overruns, it would still be doing things a kernel module shouldn't. The language involved is just a very tiny part.
  - aeyes 2y ago
    > Oh, because your code is deployed on XP and up Not even, it only runs on 7 / 2008 Server and up.
  - jiggawatts 2y ago
    > ...it means you failed, so skip it next boot. Succeed? Remove the mark. Tada. Resiliency. Better yet, it's high time that PCs booted like (some) mobile phones do: there's two copies of the OS. The running copy is read-only. The other copy can only be replaced with a monolithic "image" that has is validated with a HS256-based digitally signed Merkle hash. Either the entire image is bit-for-bit valid, or it is rejected. If booting to the new image fails twice, automatically roll back to the previous known-good image. This isn't "hard", it isn't even that much work. Heck, Windows can already boot from WIM images and VHDX disk images! The code is already in there, it just needs to be turned on.
- jki275 2y ago
  Rust isn't a panacea, and the US Government really isn't pushing for Rust adoption.
beardedwizard 2y ago
This outage says more to me about the state of software engineering in 2024 than it does about crowdstrike. Starting with the fact that software like crowdstrike exists to compensate for even poorer software rife with exploitable vulnerabilities. It is certainly hard to defend crowdstrike, but is even harder to hear so many hot takes when the engineer emperor has no clothes. Critical software engineering is a race to the bottom across many domains. Healthcare, banking, flight systems, etc.
- raggles 2y ago
  Yep, I'm an Electrical Engineer, and I also write code. I don't like the term Software Engineer, because there is none of the of the regulated safety and quality mechanisms required for software that are normally associated with professional engineering.
  - theideaofcoffee 2y ago
    Same, which is why I bristle at my title containing "engineer" as I don't have a PE. If most software engineers want to legitimately call themselves engineers, the field should be formalized as an engineering discipline, including coursework, certification/licensure and, better yet, apprenticeship-like experiences required for "real" engineers working toward their Professional Engineer license. Edit: I'd add this goes double when working on safety-critical code, or anything touching protected health data, or payment/financial data. It's just too toxic and valuable to leave to a chance change.
    - pxc 2y ago
      > If most software engineers want to legitimately call themselves engineers, the field should be formalized as an engineering discipline, including coursework, certification/licensure and, better yet, apprenticeship-like experiences required for "real" engineers working toward their Professional Engineer license. I agree, although in reality it's not chiefly developers themselves who are responsible for quick, lazy approaches, is it? Developers are typically the parties most pained by technical debt. If the discipline of software development is to become software engineering in earnest, there will have to be some pressure all the way up the management chain— pressure strong enough to outweigh software's low cost of iteration. I imagine this is really rare outside of highly regulated industries and very specific applications, and even with a formalized software engineering discipline, many companies will prefer sloppy software development and many competitive markets will 'select for' such companies.
ethbr1 2y ago
>> No staged deployment {changing to} Add staged deployment That's the thing that amazed me. How do you regularly YOLO patches worldwide to something that runs with enough permissions to crash a system? I don't care if this was a configuration update vs a new sensor capability -- universal rollout should never have been allowed by CrowdStrike's release team.
- dataflow 2y ago
  My speculation is they probably did stage the rollout to some extent, but didn't have a viable or fast enough feedback mechanism to let them know there was a kernel crash. That seems much more plausible than the engineers being incompetent enough to not have staged rollouts at all. Or they might have had it, but only for code rather than data.
  - aeyes 2y ago
    Crowdstrike in their own RCA wrote that they had no staged rollout for these types of updates: https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf https://www.crowdstrike.com/wp-content/uploads/2024/08/Chann... And I believe it because for administrators there is no configuration to delay the rollout of these "content updates", you can only delay the sensor updates.
  - ibiza 2y ago
    I think that is being too charitable. The problem is 100% reproducible. The machine blue-screens at boot up. What kind of staging environment allows this to go through?
    - ethbr1 2y ago
      Staged rollout in this case would be releasing it to a small portion of your customer base, then additional customers, etc.
- enragedcacti 2y ago
  My outsider's guess is that whoever was making decisions at that level was high on their own supply and decided that a staged rollout would result in time where they weren't protecting all those other machines from imminent and certain catastrophe. I'm still awestruck that any engineer would be willing to ship code in that setup, though I guess its also possible that they were being misled about how much testing was going on at QA.
zamadatix 2y ago
What is whooshing over my head about "Figure 1"?
- hipadev23 2y ago
  All his blogposts have a church image for some reason. This particular one I found amusing and now wonder if it’s intentional: * Saint Nedelya (church in the blog) * Satya Nadella (microsoft ceo)
Dwedit 2y ago
So you have an "Index Out Of Bounds" problem. It could either directly lead to reading out-of-bounds memory and generating an Access Violation exception, or you could see the out-of-bounds array access and throw an exception. Either way, you've got a kernel-mode exception that isn't being caught, and that's a BSOD.
- deathanatos 2y ago
  You wrap the particular extension such that if it crashes, there's a sensible fallback option, such as restarting it, etc. that doesn't take out the entire system. Many of the ideas of Tanenbaum would seem applicable towards preventing such a thing.
halayli 2y ago
> Multiple engineers identified the issue via analysis of stack dumps as being triggered by a null pointer bug in the C++ the Crowdstrike update was written in; it appears to have tried to call an invalid region of memory that results in a process getting immediately killed by Windows, but that take looked increasingly controversial and Crowdstrike itself said that the incident was not due to "null bytes contained within Channel File 291 [the update that triggered the crashes] or any other Channel File."
eugenekolo 2y ago
Sentiment might hold some merit, but this article is 80% copy pasting from an RCA report and 2 sentences saying nothing more than "This shouldn't happen" while offering no alternative or deep thought into improvement...
- b-man 2y ago
  I did provide an alternative. Formally verify their software, making implausible for such error to occur. I have linked to a peer reviewed article that goes in-depth about such.
dwheeler 2y ago
The Crowdstrike report explains why it crashed, but not how it passed final end-to-end testing. There appears to have been many tests of piece parts (unit testing), but that's not the same as testing the full system. I would think all the end-to-end tests of the full system would have been instantly detected the problem and prevented it, because it would have failed all the end-to-end tests. Did I miss something? Did they never test the complete system as deployed? Looks like it, but maybe I misunderstood something.
- tadfisher 2y ago
  From what I understand, this was a failure in parsing a configuration update file, and while they do end-to-end testing for releases of the Falcon code with the latest configuration, they don't (or didn't) do so against all configuration updates in the interim.
  - dwheeler 2y ago
    In other words, they failed to do end-to-end testing in some cases. They tested pieces, but not the full system they released. If you don't test each config update, by definition you didn't test it. Rust has many nice properties, but it can panic too. You should still test the whole system before each release if it's critical.
insane_dreamer 2y ago
That's quite a list of problems with that update; wasn't just a single bug that slipped through the cracks.
satisfice 2y ago
I’m not concerned about the technical solutions. Any technical solution has to be implemented by people. The thing not mentioned in CrowdStrike’s report is anything about people— especially management. Bad management and understaffed teams will defeat any technical solution, any day.
greenthrow 2y ago
Extremely low quality post by the submitter. Yes these shouldn't happen, but software engineers -- so far -- are all human. It's more useful to talk about the ways this could be mitigated than to just post a few sentences repeating that it shouldn't happen.
- megablast 2y ago
  It should never be up to the dev team. There should be QA and testing to stop this stuff as well. And rolling updates to catch the system in time.
  - ThrowawayR2 2y ago
    The industry doesn't want to pay what it costs to support the type of SDETs capable of doing QA on kernel drivers. Anyone that talented is going to do the math and switch to being a developer for a much bigger paycheck instead.
  - cangencer 2y ago
    The article is extremely shallow beyond saying "Formal verification a la SPARK" should have been used, while not offering how this could actually work in the real world - I don't think the author has any experience working on any similar piece of software either. While such techniques are available, would they be really applicable in a very dynamic environment such as with millions of PCs running various windows versions, needing continuous / real-time updates. And yes, we of course know that QA and testing magically removes all possible failure modes/bugs.
    - b-man 2y ago
      > While such techniques are available, would they be really applicable in a very dynamic environment such as with millions of PCs running various windows versions, needing continuous / real-time updates. I don't see much difference in complexity between the affected software and the several existing formally verified software. At the very least the parser/interpreter could very much be formally verified. But my point is, have they tried? They don't seem to be even aware of such.
luxuryballs 2y ago
I still think it was intentional, someone activated the CrowdStrike feature that was purchased by the DoD. Maybe people with inside knowledge of recent events were trying to make an exit so they had to smash the glass and hit the red button to stop air travel so they could snag them in time? Making it a perfect update failure is clever enough, but the name of the product is the best part. Imagine a system that can stop breaches even after they occur ;)
siliconc0w 2y ago
What commonly happens in these organizations is they have a software delivery path that has a lot of these best practices but soon people figure out that it is too slow so they invent a new, faster, path. From what I can tell Crowdstrike had a lot of the usual best practices like canary rollouts on their binary but they didn't on this configuration file despite it having the same consequences of a bad binary push. This wasn't even an edge case, it reliably BSOD'd every windows machine that got this update. One strategy Google SRE uses is that the team ensuring reliability has a different reporting path than the product team - so there is always a check and balance when things like rollout policies get worked around by clever product teams. It's a shame because I hear it's actually a pretty good product.
999900000999 2y ago
CrowdStrike outsourced their SDET positions to save a buck. This is what happens. Stop skimping on QA.
randerson 2y ago
Nation-state hackers of the world must love the idea of a supply chain that pushes out immediate untested updates to half the US Fortune 500, to be processed by a C++ kernel driver. If CrowdStrike's goal is to secure companies at scale, they could easily be doing the opposite.
jokoon 2y ago
I'm not a fan of rust, but if microsoft required that those sort of critical software be written in rust, it would be a good thing. Anything that is doing something sensitive or critical that can crash the system should be written in rust. If not, insurance companies would be mandated by law to run static analysis on such C++ code.

49 points by b-man 2y ago

There is no reason to not use Rust for these systems anymore. It's why the US Government is pushing so much for Rust adoption. We're going to keep seeing these horror stories until C/C++ go away.

victor106 2y ago
> It's why the US Government is pushing so much for Rust adoption. Can you provide any references for this?
- timschmidt 2y ago
  He's likely referring to this: https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/press-release-technical-report/ https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/pre...
- RockRobotRock 2y ago
  https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/press-release-technical-report/ https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/pre... https://media.defense.gov/2022/Nov/10/2003112742/-1/-1/0/CSI_SOFTWARE_MEMORY_SAFETY.PDF https://media.defense.gov/2022/Nov/10/2003112742/-1/-1/0/CSI...
- kevindamm 2y ago
  They may be referring to DARPA's research project TRACTOR: https://www.darpa.mil/program/translating-all-c-to-rust https://www.darpa.mil/program/translating-all-c-to-rust
- AlotOfReading 2y ago
  There's been a number of reports released by the government that recommend adopting memory safe languages as opposed to C/C++. The White House put out a memo [0] for example, and NIST has this page [1]. Since there are no other systems languages with widespread developer mindshare, some people see these as primarily recommending Rust. [0] https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/memory-safety-statements-of-support/ https://www.whitehouse.gov/oncd/briefing-room/2024/02/26/mem... [1] https://www.nist.gov/itl/ssd/software-quality-group/safer-languages https://www.nist.gov/itl/ssd/software-quality-group/safer-la...
  - jimbob45 2y ago
    What about MISRA C and C++ with static analysis tools? Also all GC languages would pretty easily solve the problem including Java and C#.
    - AlotOfReading 2y ago
      It's perfectly possible to write the crowdstrike issue in C/C++ code that will pass any MISRA checker, because MISRA checkers are unable to verify conformance with the standard. It basically says "don't do this" and leaves it up to the developer to ensure. Static analysis tools (broadly) are not powerful enough to ensure memory safety for nontrivial programs without a high developer burden in the form of annotations or restricted language subsets. GC languages are generally not appropriate for kernel level systems programming like crowdstrike was doing. Java has some history in this space, but generally without niceties like garbage collection that typical developers are used to.
  - echelon 2y ago
    > some people see these as primarily recommending Rust. That's not the reason. The DoD really is focused on Rust [1, 2]. [1] https://www.darpa.mil/program/translating-all-c-to-rust https://www.darpa.mil/program/translating-all-c-to-rust [2] https://www.darpa.mil/news-events/2024-07-31a https://www.darpa.mil/news-events/2024-07-31a
    - AlotOfReading 2y ago
      The federal government is a lot bigger than even the DoD, and the other agencies (e.g. NIST as linked above) have been very careful to make sure they aren't only recommending Rust, even if it's clearly a major alternative.
- notarealllama 2y ago
  If you follow CISA vulns and guidelines, it's all industrial controllers getting hacked and pushing for memory safety for like a year straight. (On a side note, pleasantly surprised to see a Linux kernel android bug today)
- keithalewis 2y ago
  We're from the Government and we're here to help. You need a license to operate a backhoe, but not to write software.
AdieuToLogic 2y ago
> There is no reason to not use Rust for these systems anymore. While Rust is a fine language, there are others which may be better suited for systems which have a global impact such as CrowdStrike does. Ada: https://en.wikipedia.org/wiki/Ada_(programming_language) https://en.wikipedia.org/wiki/Ada_(programming_language) SPARK: https://en.wikipedia.org/wiki/SPARK_(programming_language) https://en.wikipedia.org/wiki/SPARK_(programming_language) I am an advocate for neither and only reference their applicability for this problem domain based on the languages' design intent.
MaulingMonkey 2y ago
>> According to the RCA, the essence of what happened was an index out of bounds While Rust's behavior of panicing is saner than C++'s behavior of undefined behavior for a typical index out of bounds bug (`slice[index]`), I would imagine it'd still result in BSODs and outages in a kernel context. Admittedly, safe bounds checking (`slice.get(index) → Option<T>`) might be easier in Rust than in C++, but it's not a given that a RIIR would've avoided this bug.
MBCook 2y ago
Oh who cares about Rust. This could have been written in many different languages safely, including C/C++. Would it have been easier in Rust? Maybe. But it's also a legacy codebase. So we shouldn't be surprised it wasn't in Rust. There are SO MANY things that went wrong. They don't even seem to care about half of them. Ok, it wasn't tested. It wasn't staged. It wasn't a slow roll-out over hours. It wasn't verified from the deployment servers. How about why is it parsed in kernel space at all? I didn't see that listed. Why didn't you have a mechanism to catch faulty updates at runtime and flag them so that you'd be able to survive the next boot? Mark somewhere on the FS "I'm loading #19583". If you boot up and find that, it means you failed, so skip it next boot. Succeed? Remove the mark. Tada. Resiliency. Why are you even doing this stuff yourself? There are Microsoft APIs for a bunch of this stuff that would have helped reduce risk. Oh, because your code is deployed on XP and up and you don't want to have 2 or 3 versions? That went well didn't it. I bet if you were using those you wouldn't have crashed the kernel space. Do you fuzz? We know you don't. You broke Windows. You broke Macs. You broke Linux. You keep doing this. You're not good at being resilient. You had to face-plant in front of the entire world before you decided to do something. I didn't see fuzzing mentioned. Those mitigations you propose are not enough. They'll help, but you're still vulnerable. You're supposed to be a _security company_, I'd expect a better list of how to secure software from a security company. By all means, use Rust or Swift or (insert other memory safe language) if you want. That's smart. But if their existing kernel module had no UB and buffer overruns, it would still be doing things a kernel module shouldn't. The language involved is just a very tiny part.
- aeyes 2y ago
  > Oh, because your code is deployed on XP and up Not even, it only runs on 7 / 2008 Server and up.
- jiggawatts 2y ago
  > ...it means you failed, so skip it next boot. Succeed? Remove the mark. Tada. Resiliency. Better yet, it's high time that PCs booted like (some) mobile phones do: there's two copies of the OS. The running copy is read-only. The other copy can only be replaced with a monolithic "image" that has is validated with a HS256-based digitally signed Merkle hash. Either the entire image is bit-for-bit valid, or it is rejected. If booting to the new image fails twice, automatically roll back to the previous known-good image. This isn't "hard", it isn't even that much work. Heck, Windows can already boot from WIM images and VHDX disk images! The code is already in there, it just needs to be turned on.
jki275 2y ago
Rust isn't a panacea, and the US Government really isn't pushing for Rust adoption.

beardedwizard 2y ago

This outage says more to me about the state of software engineering in 2024 than it does about crowdstrike. Starting with the fact that software like crowdstrike exists to compensate for even poorer software rife with exploitable vulnerabilities. It is certainly hard to defend crowdstrike, but is even harder to hear so many hot takes when the engineer emperor has no clothes. Critical software engineering is a race to the bottom across many domains. Healthcare, banking, flight systems, etc.

raggles 2y ago
Yep, I'm an Electrical Engineer, and I also write code. I don't like the term Software Engineer, because there is none of the of the regulated safety and quality mechanisms required for software that are normally associated with professional engineering.
- theideaofcoffee 2y ago
  Same, which is why I bristle at my title containing "engineer" as I don't have a PE. If most software engineers want to legitimately call themselves engineers, the field should be formalized as an engineering discipline, including coursework, certification/licensure and, better yet, apprenticeship-like experiences required for "real" engineers working toward their Professional Engineer license. Edit: I'd add this goes double when working on safety-critical code, or anything touching protected health data, or payment/financial data. It's just too toxic and valuable to leave to a chance change.
  - pxc 2y ago
    > If most software engineers want to legitimately call themselves engineers, the field should be formalized as an engineering discipline, including coursework, certification/licensure and, better yet, apprenticeship-like experiences required for "real" engineers working toward their Professional Engineer license. I agree, although in reality it's not chiefly developers themselves who are responsible for quick, lazy approaches, is it? Developers are typically the parties most pained by technical debt. If the discipline of software development is to become software engineering in earnest, there will have to be some pressure all the way up the management chain— pressure strong enough to outweigh software's low cost of iteration. I imagine this is really rare outside of highly regulated industries and very specific applications, and even with a formalized software engineering discipline, many companies will prefer sloppy software development and many competitive markets will 'select for' such companies.

ethbr1 2y ago

>> No staged deployment {changing to} Add staged deployment That's the thing that amazed me. How do you regularly YOLO patches worldwide to something that runs with enough permissions to crash a system? I don't care if this was a configuration update vs a new sensor capability -- universal rollout should never have been allowed by CrowdStrike's release team.

dataflow 2y ago
My speculation is they probably did stage the rollout to some extent, but didn't have a viable or fast enough feedback mechanism to let them know there was a kernel crash. That seems much more plausible than the engineers being incompetent enough to not have staged rollouts at all. Or they might have had it, but only for code rather than data.
- aeyes 2y ago
  Crowdstrike in their own RCA wrote that they had no staged rollout for these types of updates: https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf https://www.crowdstrike.com/wp-content/uploads/2024/08/Chann... And I believe it because for administrators there is no configuration to delay the rollout of these "content updates", you can only delay the sensor updates.
- ibiza 2y ago
  I think that is being too charitable. The problem is 100% reproducible. The machine blue-screens at boot up. What kind of staging environment allows this to go through?
  - ethbr1 2y ago
    Staged rollout in this case would be releasing it to a small portion of your customer base, then additional customers, etc.
enragedcacti 2y ago
My outsider's guess is that whoever was making decisions at that level was high on their own supply and decided that a staged rollout would result in time where they weren't protecting all those other machines from imminent and certain catastrophe. I'm still awestruck that any engineer would be willing to ship code in that setup, though I guess its also possible that they were being misled about how much testing was going on at QA.

zamadatix 2y ago

What is whooshing over my head about "Figure 1"?

hipadev23 2y ago
All his blogposts have a church image for some reason. This particular one I found amusing and now wonder if it’s intentional: * Saint Nedelya (church in the blog) * Satya Nadella (microsoft ceo)

Dwedit 2y ago

So you have an "Index Out Of Bounds" problem. It could either directly lead to reading out-of-bounds memory and generating an Access Violation exception, or you could see the out-of-bounds array access and throw an exception. Either way, you've got a kernel-mode exception that isn't being caught, and that's a BSOD.

deathanatos 2y ago
You wrap the particular extension such that if it crashes, there's a sensible fallback option, such as restarting it, etc. that doesn't take out the entire system. Many of the ideas of Tanenbaum would seem applicable towards preventing such a thing.

halayli 2y ago

> Multiple engineers identified the issue via analysis of stack dumps as being triggered by a null pointer bug in the C++ the Crowdstrike update was written in; it appears to have tried to call an invalid region of memory that results in a process getting immediately killed by Windows, but that take looked increasingly controversial and Crowdstrike itself said that the incident was not due to "null bytes contained within Channel File 291 [the update that triggered the crashes] or any other Channel File."

eugenekolo 2y ago

Sentiment might hold some merit, but this article is 80% copy pasting from an RCA report and 2 sentences saying nothing more than "This shouldn't happen" while offering no alternative or deep thought into improvement...

b-man 2y ago
I did provide an alternative. Formally verify their software, making implausible for such error to occur. I have linked to a peer reviewed article that goes in-depth about such.

dwheeler 2y ago

The Crowdstrike report explains why it crashed, but not how it passed final end-to-end testing. There appears to have been many tests of piece parts (unit testing), but that's not the same as testing the full system. I would think all the end-to-end tests of the full system would have been instantly detected the problem and prevented it, because it would have failed all the end-to-end tests. Did I miss something? Did they never test the complete system as deployed? Looks like it, but maybe I misunderstood something.

tadfisher 2y ago
From what I understand, this was a failure in parsing a configuration update file, and while they do end-to-end testing for releases of the Falcon code with the latest configuration, they don't (or didn't) do so against all configuration updates in the interim.
- dwheeler 2y ago
  In other words, they failed to do end-to-end testing in some cases. They tested pieces, but not the full system they released. If you don't test each config update, by definition you didn't test it. Rust has many nice properties, but it can panic too. You should still test the whole system before each release if it's critical.

insane_dreamer 2y ago

That's quite a list of problems with that update; wasn't just a single bug that slipped through the cracks.

satisfice 2y ago

I’m not concerned about the technical solutions. Any technical solution has to be implemented by people. The thing not mentioned in CrowdStrike’s report is anything about people— especially management. Bad management and understaffed teams will defeat any technical solution, any day.

greenthrow 2y ago

Extremely low quality post by the submitter. Yes these shouldn't happen, but software engineers -- so far -- are all human. It's more useful to talk about the ways this could be mitigated than to just post a few sentences repeating that it shouldn't happen.

megablast 2y ago
It should never be up to the dev team. There should be QA and testing to stop this stuff as well. And rolling updates to catch the system in time.
- ThrowawayR2 2y ago
  The industry doesn't want to pay what it costs to support the type of SDETs capable of doing QA on kernel drivers. Anyone that talented is going to do the math and switch to being a developer for a much bigger paycheck instead.
- cangencer 2y ago
  The article is extremely shallow beyond saying "Formal verification a la SPARK" should have been used, while not offering how this could actually work in the real world - I don't think the author has any experience working on any similar piece of software either. While such techniques are available, would they be really applicable in a very dynamic environment such as with millions of PCs running various windows versions, needing continuous / real-time updates. And yes, we of course know that QA and testing magically removes all possible failure modes/bugs.
  - b-man 2y ago
    > While such techniques are available, would they be really applicable in a very dynamic environment such as with millions of PCs running various windows versions, needing continuous / real-time updates. I don't see much difference in complexity between the affected software and the several existing formally verified software. At the very least the parser/interpreter could very much be formally verified. But my point is, have they tried? They don't seem to be even aware of such.

luxuryballs 2y ago

I still think it was intentional, someone activated the CrowdStrike feature that was purchased by the DoD. Maybe people with inside knowledge of recent events were trying to make an exit so they had to smash the glass and hit the red button to stop air travel so they could snag them in time? Making it a perfect update failure is clever enough, but the name of the product is the best part. Imagine a system that can stop breaches even after they occur ;)

siliconc0w 2y ago

What commonly happens in these organizations is they have a software delivery path that has a lot of these best practices but soon people figure out that it is too slow so they invent a new, faster, path. From what I can tell Crowdstrike had a lot of the usual best practices like canary rollouts on their binary but they didn't on this configuration file despite it having the same consequences of a bad binary push. This wasn't even an edge case, it reliably BSOD'd every windows machine that got this update. One strategy Google SRE uses is that the team ensuring reliability has a different reporting path than the product team - so there is always a check and balance when things like rollout policies get worked around by clever product teams. It's a shame because I hear it's actually a pretty good product.

999900000999 2y ago

CrowdStrike outsourced their SDET positions to save a buck. This is what happens. Stop skimping on QA.

randerson 2y ago

Nation-state hackers of the world must love the idea of a supply chain that pushes out immediate untested updates to half the US Fortune 500, to be processed by a C++ kernel driver. If CrowdStrike's goal is to secure companies at scale, they could easily be doing the opposite.

jokoon 2y ago

I'm not a fan of rust, but if microsoft required that those sort of critical software be written in rust, it would be a good thing. Anything that is doing something sensitive or critical that can crash the system should be written in rust. If not, insurance companies would be mandated by law to run static analysis on such C++ code.