7 ms·

Number 4 continues to be the most surprising bit to me. I could not fathom having a process that involves deploying to 8.5 million remote machines simultaneousl

Number 4 continues to be the most surprising bit to me. I could not fathom having a process that involves deploying to 8.5 million remote machines simultaneously. Bugs in code I can almost always understand and forgive, even the ones that seem like they’d be obvious with hindsight. But this is just an egregious lack of the most basic rollout standards.

gitfan86 2y ago
They probably don't get to claim agile story points until the ticket is in finished state. And they probably have a culture where vanity Metrics like "velocity" are prioritized
- nmg 2y ago
  This would answer the question that i've not heard anyone asking: what incentivized the bad decisions that led to this catastrophic failure?
  - phs318u 2y ago
    My understanding is that the culture (as reported by some customers) is quite aggressive and pushy. They are quite vocal when customers don’t turn in automatic updates. It makes sense in a way - given their fast growth strategy (from nowhere to top 3) and desire to “do things differently” - the iconoclast upstarts that redefine the industry. Or to summarise - hubris.
    - hello_moto 2y ago
      To catch 0day quickly, EDR needs to know "how". The "how" here is AV definition or a way to identify the attack. In CS-speak: content. Catching 0day quickly results in good reputation that your EDR works well. If people turn off their AV definition auto-update, they are at-risk. Why use EDR if folks don't want to stop attack quickly?
      - LtWorf 2y ago
        In theory you're correct. In practice it seems that crowdstrike has crashed systems with their updates much more often than 0day attacks.
        hello_moto 2y ago
        How many times? This is one bsod on windows 10. I saw another kernel panic on specific Linux distro. What else? One thing that is funny is that quite a few of their competitors are taking this opportunity to shit on them via Twitter and by marketing themselves as better than CrowdStrike. Twitter, with all its issues, apparently has a feature to prevent fake news and that feature will show crowd source sentiment to debunk fake news, in this case, Twitter users showed how many times Crowdstrike competitor BSOD windows
        phs318u 2y ago
        Bottom line is this: there is absolutely no good reason for not doing rolling updates. Do a few and make sure they are ok. Keep rolling out in groups. This single approach alone would've meant that this event was of marginal impact to most of the public, as sysadmins would've had the opportunity to halt further updates and work on remediating their first group (typically non-critical servers). Rolling out to everything all at once is just bad practice, period.
        hello_moto 2y ago
        Put yourself in the customer shoes: who wants to sign up to be rolled first? There's best practice and then there's customer
        RoyalHenOil 2y ago
        Give those customers a discount. The solution to customer reluctance to being the guinea pig is not to force every customer to be a guinea pig.
        2y ago
    - 77pt77 2y ago
      > They are quite vocal when customers don’t turn in automatic updates. I'm sorry but this is the customer's fault. If I'm using your services you work for me and you don't get to bully me into doing whatever you think needs to be done. People that chose this solution need to be penalized, but they won't.
      - mbreese 2y ago
        Customers don’t always have a choice here. They could be restricted by compliance programs (PCI, et al) and be required under those terms to have auto updates on. Compliance also has to share some of the blame here, if best practices (local testing) aren’t allowed to be followed in the name of “security”.
        nerdjon 2y ago
        This needs to keep being repeated anytime someone wants to blame the company. Many don’t have a choice, a lot of compliance is doing x to satisfy a checkbox and you don’t have a lot of flexibility in that or you may not be able to things like process credit cards which is kinda unacceptable depending on your company. (Note: I didn’t say all) CrowdStrike automatic update happens to satisfy some of those checkboxes.
- cruffle_duffle 2y ago
  Oh the games I have to play with story points that have personal performance metrics attached to them. Splitting tickets to span sprints so there aren’t holes in some dudes “effort” because they didn’t compete some task they committed to. I never thought such stories were real until I encountered them…
  - dboreham 2y ago
    Save yourself while you still can.
thundershart 2y ago
Surely, CrowdStrike's safety posture for update rollouts is in serious need of improvement. No argument there. But is there any responsibility for the clients consuming the data to have verified these updates prior to taking them in production? I haven't worn the sysadmin hat in a while now, but back when I was responsible for the upkeep of many thousands of machines, we'd never have blindly consumed updates without at least a basic smoke test in a production-adjacent UAT type environment. Core OS updates, firmware updates, third party software, whatever -- all of it would get at least some cursory smoke testing before allowing it to hit production. On the other hand, given EDR's real-world purpose and the speed at which novel attacks propagate, there's probably a compelling argument for always taking the latest definition/signature updates as soon as they're available, even in your production environments. I'm certainly not saying that CrowdStrike did nothing wrong here, that's clearly not the case. But if conventional wisdom says that you should kick the tires on the latest batch of OS updates from Microsoft in a test environment, maybe that same rationale should apply to EDR agents?
- stoolpigeon 2y ago
  I think point 3 of the grand parent indicates admins were not given an opportunity to test this. My company had a lot of Azure vms impacted by this and I'm not sure who the admin was who should have tested it. Microsoft? I don't think we have anything to do with crowdstrike software on our vms. ( I think - I'm sure I'll find out this week.) Edit: I just learned the Azure central region failure wasn't related to the larger event - and we weren't impacted by the crowd strike issue - I didn't know it was two different things. So my second part of the comment is irrelevant.
  - thundershart 2y ago
    Oh, I'd missed point #3 somehow. If individual consumers weren't even given the opportunity to test this, whether by policy or by bug, then ... yeesh. Even worse than I'd thought. Exactly which team owns the testing is probably left up to each individual company to determine. But ultimately, if you have a team of admins supporting the production deployment of the machines that enable your business, then someone's responsible for ensuring the availability of those machines. Given how impactful this CrowdStrike incident was, maybe these kinds of third-party auto-update postures need to be reviewed and potentially brought back into the fold of admin-reviewed updates.
mbreese 2y ago
For me, number 1 is the worst of the bunch. You should always expect that there will be bugs in processes, input files, etc… the fact that their code wasn’t robust enough to recognize a corrupted file and not crash is inexcusable. Especially in kernel code that is so widely deployed. If any one of the five points above hadn’t happened, this event would have been avoided. However, if number 1 had been addressed - any of the others could have happened (or all at the same time) and it would have been fine. I understand that we should assume that bugs will be present anywhere, which is why staggered deployments are also important. If there had been staggered deployments, the. The damage would have happened, but it would have been localized. I think security people would argue against a staged deployment though, as if it were discovered what the new definitions protected against, an exploit could be developed quickly to put those servers that aren’t in the “canary” group at risk. (At least in theory — I can’t see how staggering deployment over a 6-12 hour window would have been that risky).
- timmytokyo 2y ago
  They're all terrible, but I agree #1 is particularly egregious for a company ostensibly dedicated to security. A simple fuzz tester would have caught this type of bug, so they clearly don't perform even a minimal amount of testing on their code.
  - nsguy 2y ago
    Totally agree. Not only would a coverage guided fuzzer catch this they should also be adding every single file they send out to the corpus of that automated fuzz testing so they can get somewhat increased coverage on their parser. There may not be out of the box fuzzers that test device drivers so you hoist all the parser code, build it into a stand-alone application, and fuzz that. Likely this is a form of technical debt since I can understand not doing all of this day #1 when you have 5 customers but at some point as you scale up you need to change the way you look at risk.
  - fluidcruft 2y ago
    Seems like it would be easy enough to add a new checkbox for this to audits.
layer8 2y ago
Malware signature updates are supposed to be deployed ASAP, because every minute may count when a new attack is spreading. The mistake may have been to apply that policy indiscriminately.
mrbombastic 2y ago
And here I thought shipping a new version on the app store was scary. Is there anything we can take from other professions/tradecraft/unions/legislation to ensure shops can’t skip the basic best practices we are aware of in the industry like staged rollouts? How do we set incentives to prevent this? Seriously the App Store was raking in $$ from us for years with no support for staged rollouts and no other options.
avree 2y ago
A lot of snarky replies to this comment, but the reality is that if you were selling an anti-virus, identified a malicious virus, and then chose not to update millions of your machines with that virus’s signature, you’d also be in the wrong.
- naasking 2y ago
  > identified a malicious virus, and then chose not to update millions of your machines with that virus’s signature, you’d also be in the wrong. No, for exactly the reason we just saw, and the same reason why vaccines are tested before widespread rollout.
  - aforwardslash 2y ago
    On the other hand, diseases vaccines prevent dont have almost instantaneous propagation, thats why they are effective at containing propagation. As an example, reaction time is paramount to counter many kinds of attacks - thats why blocklists are so popular, and AS blackholing is a viable option.
- rco8786 2y ago
  I’m not saying don’t update? I’m talking about rolling the update over the course of a short amount of time, like under an hour. With the ability to stop the rollout.
VirusNewbie 2y ago
> But this is just an egregious lack of the most basic rollout standards. Agreed. It's crazy that the top tech companies enforce this in a biblical fashion, despite all sorts of pressure to ship and all that. Crowdstrike went YOLO at a global scale.
alsetmusic 2y ago
I worked at one of the big ones and we always shipped live to all consumer devices at the same time. But this was for a popular suite of products that generate a lot of consumer demand, so we had a rigorous QA process to make sure this wouldn't be a problem. As I was typing this, it occurred to me that zero people would have cared if this update was staggered making it pretty silly not to.
- trhway 2y ago
  As the QA manager said in our recent product meeting - "as the canary doesn't work we roll out and test on the production cloud".
robomc 2y ago
I wonder if there's a concern that staggering the malware signatures would open them up to lawsuits if somebody was hacked in between other customers getting the data and them getting the data.
- thundershart 2y ago
  > I wonder if there's a concern that staggering the malware signatures would open them up to lawsuits if somebody was hacked in between other customers getting the data and them getting the data. I'd assume that sort of thing would be covered in the EULA and contract -- but even if it weren't, it seems like allowing customers to define their own definition update strategy would give them a pretty compelling avenue to claim non-liability. If CrowdStrike can credibly claim "hey, we made the definitions available, you chose to wait for 2 weeks to apply them, that's on you", then it becomes much less of a concern.
otabdeveloper4 2y ago
Rolling it all out at once is "security best practices".
- rco8786 2y ago
  What's the reasoning here? Extreme time sensitivity?

gitfan86 2y ago

They probably don't get to claim agile story points until the ticket is in finished state. And they probably have a culture where vanity Metrics like "velocity" are prioritized

nmg 2y ago
This would answer the question that i've not heard anyone asking: what incentivized the bad decisions that led to this catastrophic failure?
- phs318u 2y ago
  My understanding is that the culture (as reported by some customers) is quite aggressive and pushy. They are quite vocal when customers don’t turn in automatic updates. It makes sense in a way - given their fast growth strategy (from nowhere to top 3) and desire to “do things differently” - the iconoclast upstarts that redefine the industry. Or to summarise - hubris.
  - hello_moto 2y ago
    To catch 0day quickly, EDR needs to know "how". The "how" here is AV definition or a way to identify the attack. In CS-speak: content. Catching 0day quickly results in good reputation that your EDR works well. If people turn off their AV definition auto-update, they are at-risk. Why use EDR if folks don't want to stop attack quickly?
    - LtWorf 2y ago
      In theory you're correct. In practice it seems that crowdstrike has crashed systems with their updates much more often than 0day attacks.
      - hello_moto 2y ago
        How many times? This is one bsod on windows 10. I saw another kernel panic on specific Linux distro. What else? One thing that is funny is that quite a few of their competitors are taking this opportunity to shit on them via Twitter and by marketing themselves as better than CrowdStrike. Twitter, with all its issues, apparently has a feature to prevent fake news and that feature will show crowd source sentiment to debunk fake news, in this case, Twitter users showed how many times Crowdstrike competitor BSOD windows
        phs318u 2y ago
        Bottom line is this: there is absolutely no good reason for not doing rolling updates. Do a few and make sure they are ok. Keep rolling out in groups. This single approach alone would've meant that this event was of marginal impact to most of the public, as sysadmins would've had the opportunity to halt further updates and work on remediating their first group (typically non-critical servers). Rolling out to everything all at once is just bad practice, period.
        hello_moto 2y ago
        Put yourself in the customer shoes: who wants to sign up to be rolled first? There's best practice and then there's customer
        RoyalHenOil 2y ago
        Give those customers a discount. The solution to customer reluctance to being the guinea pig is not to force every customer to be a guinea pig.
        2y ago
  - 77pt77 2y ago
    > They are quite vocal when customers don’t turn in automatic updates. I'm sorry but this is the customer's fault. If I'm using your services you work for me and you don't get to bully me into doing whatever you think needs to be done. People that chose this solution need to be penalized, but they won't.
    - mbreese 2y ago
      Customers don’t always have a choice here. They could be restricted by compliance programs (PCI, et al) and be required under those terms to have auto updates on. Compliance also has to share some of the blame here, if best practices (local testing) aren’t allowed to be followed in the name of “security”.
      - nerdjon 2y ago
        This needs to keep being repeated anytime someone wants to blame the company. Many don’t have a choice, a lot of compliance is doing x to satisfy a checkbox and you don’t have a lot of flexibility in that or you may not be able to things like process credit cards which is kinda unacceptable depending on your company. (Note: I didn’t say all) CrowdStrike automatic update happens to satisfy some of those checkboxes.
cruffle_duffle 2y ago
Oh the games I have to play with story points that have personal performance metrics attached to them. Splitting tickets to span sprints so there aren’t holes in some dudes “effort” because they didn’t compete some task they committed to. I never thought such stories were real until I encountered them…
- dboreham 2y ago
  Save yourself while you still can.

thundershart 2y ago

Surely, CrowdStrike's safety posture for update rollouts is in serious need of improvement. No argument there. But is there any responsibility for the clients consuming the data to have verified these updates prior to taking them in production? I haven't worn the sysadmin hat in a while now, but back when I was responsible for the upkeep of many thousands of machines, we'd never have blindly consumed updates without at least a basic smoke test in a production-adjacent UAT type environment. Core OS updates, firmware updates, third party software, whatever -- all of it would get at least some cursory smoke testing before allowing it to hit production. On the other hand, given EDR's real-world purpose and the speed at which novel attacks propagate, there's probably a compelling argument for always taking the latest definition/signature updates as soon as they're available, even in your production environments. I'm certainly not saying that CrowdStrike did nothing wrong here, that's clearly not the case. But if conventional wisdom says that you should kick the tires on the latest batch of OS updates from Microsoft in a test environment, maybe that same rationale should apply to EDR agents?

stoolpigeon 2y ago
I think point 3 of the grand parent indicates admins were not given an opportunity to test this. My company had a lot of Azure vms impacted by this and I'm not sure who the admin was who should have tested it. Microsoft? I don't think we have anything to do with crowdstrike software on our vms. ( I think - I'm sure I'll find out this week.) Edit: I just learned the Azure central region failure wasn't related to the larger event - and we weren't impacted by the crowd strike issue - I didn't know it was two different things. So my second part of the comment is irrelevant.
- thundershart 2y ago
  Oh, I'd missed point #3 somehow. If individual consumers weren't even given the opportunity to test this, whether by policy or by bug, then ... yeesh. Even worse than I'd thought. Exactly which team owns the testing is probably left up to each individual company to determine. But ultimately, if you have a team of admins supporting the production deployment of the machines that enable your business, then someone's responsible for ensuring the availability of those machines. Given how impactful this CrowdStrike incident was, maybe these kinds of third-party auto-update postures need to be reviewed and potentially brought back into the fold of admin-reviewed updates.

mbreese 2y ago

For me, number 1 is the worst of the bunch. You should always expect that there will be bugs in processes, input files, etc… the fact that their code wasn’t robust enough to recognize a corrupted file and not crash is inexcusable. Especially in kernel code that is so widely deployed. If any one of the five points above hadn’t happened, this event would have been avoided. However, if number 1 had been addressed - any of the others could have happened (or all at the same time) and it would have been fine. I understand that we should assume that bugs will be present anywhere, which is why staggered deployments are also important. If there had been staggered deployments, the. The damage would have happened, but it would have been localized. I think security people would argue against a staged deployment though, as if it were discovered what the new definitions protected against, an exploit could be developed quickly to put those servers that aren’t in the “canary” group at risk. (At least in theory — I can’t see how staggering deployment over a 6-12 hour window would have been that risky).

timmytokyo 2y ago
They're all terrible, but I agree #1 is particularly egregious for a company ostensibly dedicated to security. A simple fuzz tester would have caught this type of bug, so they clearly don't perform even a minimal amount of testing on their code.
- nsguy 2y ago
  Totally agree. Not only would a coverage guided fuzzer catch this they should also be adding every single file they send out to the corpus of that automated fuzz testing so they can get somewhat increased coverage on their parser. There may not be out of the box fuzzers that test device drivers so you hoist all the parser code, build it into a stand-alone application, and fuzz that. Likely this is a form of technical debt since I can understand not doing all of this day #1 when you have 5 customers but at some point as you scale up you need to change the way you look at risk.
- fluidcruft 2y ago
  Seems like it would be easy enough to add a new checkbox for this to audits.

layer8 2y ago

Malware signature updates are supposed to be deployed ASAP, because every minute may count when a new attack is spreading. The mistake may have been to apply that policy indiscriminately.

mrbombastic 2y ago

And here I thought shipping a new version on the app store was scary. Is there anything we can take from other professions/tradecraft/unions/legislation to ensure shops can’t skip the basic best practices we are aware of in the industry like staged rollouts? How do we set incentives to prevent this? Seriously the App Store was raking in $$ from us for years with no support for staged rollouts and no other options.

avree 2y ago

A lot of snarky replies to this comment, but the reality is that if you were selling an anti-virus, identified a malicious virus, and then chose not to update millions of your machines with that virus’s signature, you’d also be in the wrong.

naasking 2y ago
> identified a malicious virus, and then chose not to update millions of your machines with that virus’s signature, you’d also be in the wrong. No, for exactly the reason we just saw, and the same reason why vaccines are tested before widespread rollout.
- aforwardslash 2y ago
  On the other hand, diseases vaccines prevent dont have almost instantaneous propagation, thats why they are effective at containing propagation. As an example, reaction time is paramount to counter many kinds of attacks - thats why blocklists are so popular, and AS blackholing is a viable option.
rco8786 2y ago
I’m not saying don’t update? I’m talking about rolling the update over the course of a short amount of time, like under an hour. With the ability to stop the rollout.

VirusNewbie 2y ago

> But this is just an egregious lack of the most basic rollout standards. Agreed. It's crazy that the top tech companies enforce this in a biblical fashion, despite all sorts of pressure to ship and all that. Crowdstrike went YOLO at a global scale.

alsetmusic 2y ago

I worked at one of the big ones and we always shipped live to all consumer devices at the same time. But this was for a popular suite of products that generate a lot of consumer demand, so we had a rigorous QA process to make sure this wouldn't be a problem. As I was typing this, it occurred to me that zero people would have cared if this update was staggered making it pretty silly not to.

trhway 2y ago
As the QA manager said in our recent product meeting - "as the canary doesn't work we roll out and test on the production cloud".

robomc 2y ago

I wonder if there's a concern that staggering the malware signatures would open them up to lawsuits if somebody was hacked in between other customers getting the data and them getting the data.

thundershart 2y ago
> I wonder if there's a concern that staggering the malware signatures would open them up to lawsuits if somebody was hacked in between other customers getting the data and them getting the data. I'd assume that sort of thing would be covered in the EULA and contract -- but even if it weren't, it seems like allowing customers to define their own definition update strategy would give them a pretty compelling avenue to claim non-liability. If CrowdStrike can credibly claim "hey, we made the definitions available, you chose to wait for 2 weeks to apply them, that's on you", then it becomes much less of a concern.

otabdeveloper4 2y ago

Rolling it all out at once is "security best practices".

rco8786 2y ago
What's the reasoning here? Extreme time sensitivity?