6 ms·

CrowdStrike blamed their test software, but in the same breath revealed that they haven't been using any canary deployments. The bug that caused all this was pr

CrowdStrike blamed their test software, but in the same breath revealed that they haven't been using any canary deployments. The bug that caused all this was present in their kernel driver for a long time. For being such a large cybersecurity player and deploying updates to 8.5 million devices, their quality control practices are embarrasingly lacking.

rvnx 2y ago
Clearly incompetence to deploy from 0 to 8 million devices without any gradual rollout. That goes even further, because apparently they were fully blind and didn't have crash metrics. "Ok we push the update, and pray".
- galangalalgol 2y ago
  I think it is past incompetence, and on into negligence. Given the stories we have heard here about emergency service failures it is likely that people died. When people die due to negligence isn't that usually criminal?
  - rvnx 2y ago
    Can't agree more, you found the right words.
  - binkHN 2y ago
    And this is how the lawsuits will start.
  - SoftTalker 2y ago
    Who is negligent though? Crowdstrike, or the emergency services that are using an OS that requires third party endpoint security right out of the box in order to be safely used, or the company that makes and sells that OS?
    - crazygringo 2y ago
      Why not both? Crowdstrike, for negligently not rolling out updates gradually. And emergency services, if they don't have robust fallback procedures/systems for when their IT system goes down. I mean it's totally fine if regular doctor's visits get postponed, but 911 should never go down just because their computers down. Just like aircraft have redundant systems, so too should 911. (The company that makes and sells the OS -- I don't see any negligence there, in this case. If security software fundamentally requires running at the kernel level and Microsoft allows that, I don't see how Microsoft can be at fault.)
      - jmb99 2y ago
        Yeah, I don’t see how one can blame Microsoft in this scenario. If you choose to run buggy kernel-level code, that’s on you, not the publisher of the kernel/OS. Especially when the code you’re running is a replacement for functionality already provided by the OS. It’s hard to argue that MS could be negligent for “not having a good enough AV/endpoint protection solution” or “allowing customers to run kernel-level code.”
    - Aeolun 2y ago
      It’s hard for people to understand that these massive ‘security’ enterprises are often connected by a large amount of bodies instead of competence.
  - notepad0x90 2y ago
    https://www.techradar.com/news/google-chrome-not-working-microsoft-defender-may-be-to-blame https://www.techradar.com/news/google-chrome-not-working-mic... ,not an unusual practice and they were not first av company to cause outages. and again, it was not a software update, the buggy software was deployed after testing back in march. Details matter! How about we let the lawyers figure out who had what liability, just like with the av/edr industry, we should know when the subject matter is outside our area of knowledge and expertise.
    - galangalalgol 2y ago
      What isn't an unusual practice? A lack of canary deployments? For av definition files, where crashing the scanner doesn't bring down the kernel I can agree. For a file read by a kernel module/driver that is primarily used by security sensitive infrastructure, I really hope it is unusual, and if it isn't lets make an example. If failing to do unit tests to get full coverage (line and branch) on software that gets loaded on security sensitive infrastructure isn't unusual then things have sure changed in the decade since I was involved. We had to get that coverage at the assembly level and then peer review the tests to make sure they were meaningful and correct. Every time the code changed. Code that failed to properly sanitize input files without panicking should not pass normal safety critical code pipelines. I do wonder if windows already has some disclaimer about being unfit for such use cases, in which case yeah, lawyers will have some fun, but even with that sort of protection, CS is clearly marketing to critical infrastructure, so it has no moral defense whatever lawyers and the law says.
- notepad0x90 2y ago
  I shared with a sibling commenter: https://www.techradar.com/news/google-chrome-not-working-microsoft-defender-may-be-to-blame https://www.techradar.com/news/google-chrome-not-working-mic... Did Microsoft do a staged or canary roll out with that? This is not a software update, if you're making such comments then you're speaking about something outside of your field of expertise.
mort96 2y ago
Every company I've ever been at rolls out updates slowly. Rolling out a change to 8.5 million computers at the same time seems ridiculous. Even the most cash strapped start-ups with every incentive to cut corners tends to get staged roll-outs more or less right. It's crazy.
- binkHN 2y ago
  Beyond crazy. I even have a small app that never makes it to production before being rolled out to internal and open testing first. And, even then, it's slowly rolled out to a percentage at each stage before being fully deployed. One would think a major company with kernel level access would do this at minimum.
- geon 2y ago
  I had a fleet of only maybe 200 computers I updated remotely. I did canary staged roll outs.
  - doubled112 2y ago
    When I managed ~ 15 developer’s Arch Linux workstations, I found it very beneficial to be the canary, and then rollout to a couple of the more capable of troubleshooting devs, and then the rest. I can always fix my own box. 8.5M all at once feels insane.
  - notepad0x90 2y ago
    not a software update!
    - mort96 2y ago
      Not relevant!
      - notepad0x90 2y ago
        details are always relevant in a technical discussion. look at my other comments where i pointed out microsoft performing similar immediate av signature updates and causing chaos.
duskwuff 2y ago
> CrowdStrike blamed their test software, but in the same breath revealed that they haven't been using any canary deployments. Their post-incident report [1] also stated that they intend to improve testing by "using testing types such as: local developer testing". One has to wonder what, if any, testing they were doing beforehand. [1]: https://www.crowdstrike.com/blog/falcon-content-update-preliminary-post-incident-report/ https://www.crowdstrike.com/blog/falcon-content-update-preli...
- MBCook 2y ago
  Well we know what the testing is, don’t we? The update literally crashed the system it was used on. There’s no way they couldn’t know that unless they never ran it. Right? Is this one of those things that only happened to 10% of users? Because I haven’t seen that reported anywhere.
  - duskwuff 2y ago
    > Is this one of those things that only happened to 10% of users? Because I haven’t seen that reported anywhere. As far as I'm aware, it affected all systems using Crowdstrike.
lupusreal 2y ago
Unless their developers had room temperature IQs or were actual psychopaths, I really wonder how they even managed to find developers who had the nerves to deploy to the whole world all at once like that. If it were me I'd be scared shitless, covered in sweat and probably shaking too hard to even type. Were CrowdStrike developers too stupid to even realize the magnitude of what they were doing? Or did they have cooler nerves than an open-heart surgeon? It's shocking to me that they could have done this so casually.
- Aeolun 2y ago
  > Were CrowdStrike developers too stupid to even realize the magnitude of what they were doing? More likely they were following a playbook to the letter, and were therefore 100% of success.

rvnx 2y ago

Clearly incompetence to deploy from 0 to 8 million devices without any gradual rollout. That goes even further, because apparently they were fully blind and didn't have crash metrics. "Ok we push the update, and pray".

galangalalgol 2y ago
I think it is past incompetence, and on into negligence. Given the stories we have heard here about emergency service failures it is likely that people died. When people die due to negligence isn't that usually criminal?
- rvnx 2y ago
  Can't agree more, you found the right words.
- binkHN 2y ago
  And this is how the lawsuits will start.
- SoftTalker 2y ago
  Who is negligent though? Crowdstrike, or the emergency services that are using an OS that requires third party endpoint security right out of the box in order to be safely used, or the company that makes and sells that OS?
  - crazygringo 2y ago
    Why not both? Crowdstrike, for negligently not rolling out updates gradually. And emergency services, if they don't have robust fallback procedures/systems for when their IT system goes down. I mean it's totally fine if regular doctor's visits get postponed, but 911 should never go down just because their computers down. Just like aircraft have redundant systems, so too should 911. (The company that makes and sells the OS -- I don't see any negligence there, in this case. If security software fundamentally requires running at the kernel level and Microsoft allows that, I don't see how Microsoft can be at fault.)
    - jmb99 2y ago
      Yeah, I don’t see how one can blame Microsoft in this scenario. If you choose to run buggy kernel-level code, that’s on you, not the publisher of the kernel/OS. Especially when the code you’re running is a replacement for functionality already provided by the OS. It’s hard to argue that MS could be negligent for “not having a good enough AV/endpoint protection solution” or “allowing customers to run kernel-level code.”
  - Aeolun 2y ago
    It’s hard for people to understand that these massive ‘security’ enterprises are often connected by a large amount of bodies instead of competence.
- notepad0x90 2y ago
  https://www.techradar.com/news/google-chrome-not-working-microsoft-defender-may-be-to-blame https://www.techradar.com/news/google-chrome-not-working-mic... ,not an unusual practice and they were not first av company to cause outages. and again, it was not a software update, the buggy software was deployed after testing back in march. Details matter! How about we let the lawyers figure out who had what liability, just like with the av/edr industry, we should know when the subject matter is outside our area of knowledge and expertise.
  - galangalalgol 2y ago
    What isn't an unusual practice? A lack of canary deployments? For av definition files, where crashing the scanner doesn't bring down the kernel I can agree. For a file read by a kernel module/driver that is primarily used by security sensitive infrastructure, I really hope it is unusual, and if it isn't lets make an example. If failing to do unit tests to get full coverage (line and branch) on software that gets loaded on security sensitive infrastructure isn't unusual then things have sure changed in the decade since I was involved. We had to get that coverage at the assembly level and then peer review the tests to make sure they were meaningful and correct. Every time the code changed. Code that failed to properly sanitize input files without panicking should not pass normal safety critical code pipelines. I do wonder if windows already has some disclaimer about being unfit for such use cases, in which case yeah, lawyers will have some fun, but even with that sort of protection, CS is clearly marketing to critical infrastructure, so it has no moral defense whatever lawyers and the law says.
notepad0x90 2y ago
I shared with a sibling commenter: https://www.techradar.com/news/google-chrome-not-working-microsoft-defender-may-be-to-blame https://www.techradar.com/news/google-chrome-not-working-mic... Did Microsoft do a staged or canary roll out with that? This is not a software update, if you're making such comments then you're speaking about something outside of your field of expertise.

mort96 2y ago

Every company I've ever been at rolls out updates slowly. Rolling out a change to 8.5 million computers at the same time seems ridiculous. Even the most cash strapped start-ups with every incentive to cut corners tends to get staged roll-outs more or less right. It's crazy.

binkHN 2y ago
Beyond crazy. I even have a small app that never makes it to production before being rolled out to internal and open testing first. And, even then, it's slowly rolled out to a percentage at each stage before being fully deployed. One would think a major company with kernel level access would do this at minimum.
geon 2y ago
I had a fleet of only maybe 200 computers I updated remotely. I did canary staged roll outs.
- doubled112 2y ago
  When I managed ~ 15 developer’s Arch Linux workstations, I found it very beneficial to be the canary, and then rollout to a couple of the more capable of troubleshooting devs, and then the rest. I can always fix my own box. 8.5M all at once feels insane.
- notepad0x90 2y ago
  not a software update!
  - mort96 2y ago
    Not relevant!
    - notepad0x90 2y ago
      details are always relevant in a technical discussion. look at my other comments where i pointed out microsoft performing similar immediate av signature updates and causing chaos.

duskwuff 2y ago

> CrowdStrike blamed their test software, but in the same breath revealed that they haven't been using any canary deployments. Their post-incident report [1] also stated that they intend to improve testing by "using testing types such as: local developer testing". One has to wonder what, if any, testing they were doing beforehand. [1]: https://www.crowdstrike.com/blog/falcon-content-update-preliminary-post-incident-report/ https://www.crowdstrike.com/blog/falcon-content-update-preli...

MBCook 2y ago
Well we know what the testing is, don’t we? The update literally crashed the system it was used on. There’s no way they couldn’t know that unless they never ran it. Right? Is this one of those things that only happened to 10% of users? Because I haven’t seen that reported anywhere.
- duskwuff 2y ago
  > Is this one of those things that only happened to 10% of users? Because I haven’t seen that reported anywhere. As far as I'm aware, it affected all systems using Crowdstrike.

lupusreal 2y ago

Unless their developers had room temperature IQs or were actual psychopaths, I really wonder how they even managed to find developers who had the nerves to deploy to the whole world all at once like that. If it were me I'd be scared shitless, covered in sweat and probably shaking too hard to even type. Were CrowdStrike developers too stupid to even realize the magnitude of what they were doing? Or did they have cooler nerves than an open-heart surgeon? It's shocking to me that they could have done this so casually.

Aeolun 2y ago
> Were CrowdStrike developers too stupid to even realize the magnitude of what they were doing? More likely they were following a playbook to the letter, and were therefore 100% of success.