8 ms·

> If thorough investigation revealed poor quality control investment compared to what would be appropriate for a company like this, then we can say for sure. W

> If thorough investigation revealed poor quality control investment compared to what would be appropriate for a company like this, then we can say for sure. We don't really need that thorough of an investigation. They had no staged deploys when servicing millions of machines. That alone is enough to say they're not running the company correctly.

dartos 2y ago
Totally agree. I’d consider staggering a rollout to be the absolute basics of due diligence. Especially when you’re building a critical part of millions of customer machines.
- wlonkly 2y ago
  I also fall on the side of "stagger the rollout" (or "give customers tools to stagger the rollout"), but at the same time I recognize that a lot of customers would not accept delays on the latest malware data. Before the incident, if you asked a customer if they would like to get updates faster even if it means that there is a remote chance of a problem with them... I bet they'd still want to get updates faster.
  - dartos 2y ago
    There must be balance
- mewpmewp2 2y ago
  I would say that canary release is an absolute must 100%. Except I can think of cases where it might still not be enough. So, I just don't feel comfortable judging them out of the box. Does all the evidence seem to point against them? For sure. But I just don't feel comfortable giving that final verdict without knowing for sure. Specifically because this is about fighting against malicious actors, where time can be of essence to deploy some sort of protection against a novel threat. If there's deadlines that you can go over, and nothing bad happens, for sure. Always have canary releases, and perfect QA, monitoring everything thoroughly, but I'm just saying, there can be cases where damage that could be done if you don't act fast enough, is just so much worse. And I don't know that it wasn't the case for them. I just don't know.
  - acdha 2y ago
    > Specifically because this is about fighting against malicious actors, where time can be of essence to deploy some sort of protection against a novel threat. This is severely overstating the problem: an extra few minutes is not going to be the difference between their customers being compromised. Most of the devices they run on are never compromised, because anyone remotely serious has defense in depth. If it was true, or even close to true, that would make the criticism more rather than less strong. If time is of the essence, you invest in things like reviewing test coverage (their most glaring lapse), fuzz testing, and common reliability engineering techniques like having the system roll back to the last known good configuration after it’s failed to load. We think of progressive rollouts as common now but they got to get that mainstream in large part because the Google Chrome team realized rapid updates are important but then asked what they needed to do to make them safe. CrowdStrike’s report suggests that they wanted rapid but weren’t willing to invest in the implementation because that isn’t a customer-visible feature – until it very painfully became one.
  - dartos 2y ago
    In this case, they pretty much caused a worst case scenario…
canucker2016 2y ago
They literally half-assed their deployment process - one part enterprisey, one part "move fast and break things". Guess which part took down much of the corporate world? from Preliminary Post Incident Review at https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio... : "CrowdStrike delivers security content configuration updates to our sensors in two ways: Sensor Content that is shipped with our sensor directly, and Rapid Response Content that is designed to respond to the changing threat landscape at operational speed. ... The sensor release process begins with automated testing, both prior to and after merging into our code base. This includes unit testing, integration testing, performance testing and stress testing. This culminates in a staged sensor rollout process that starts with dogfooding internally at CrowdStrike, followed by early adopters. It is then made generally available to customers. Customers then have the option of selecting which parts of their fleet should install the latest sensor release (‘N’), or one version older (‘N-1’) or two versions older (‘N-2’) through Sensor Update Policies. The event of Friday, July 19, 2024 was not triggered by Sensor Content, which is only delivered with the release of an updated Falcon sensor. Customers have complete control over the deployment of the sensor — which includes Sensor Content and Template Types. ... Rapid Response Content is used to perform a variety of behavioral pattern-matching operations on the sensor using a highly optimized engine. Newly released Template Types are stress tested across many aspects, such as resource utilization, system performance impact and event volume. For each Template Type, a specific Template Instance is used to stress test the Template Type by matching against any possible value of the associated data fields to identify adverse system interactions. Template Instances are created and configured through the use of the Content Configuration System, which includes the Content Validator that performs validation checks on the content before it is published. On July 19, 2024, two additional IPC Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data. Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production."
- hello_moto 2y ago
  > one part enterprisey, one part "move fast and break things". When there's 0day, how enterprisey you would like to catch the 0day?
Aeolun 2y ago
Nonsense. You don’t need any staged deploys if you simply make no mistakes. /s

dartos 2y ago

Totally agree. I’d consider staggering a rollout to be the absolute basics of due diligence. Especially when you’re building a critical part of millions of customer machines.

wlonkly 2y ago
I also fall on the side of "stagger the rollout" (or "give customers tools to stagger the rollout"), but at the same time I recognize that a lot of customers would not accept delays on the latest malware data. Before the incident, if you asked a customer if they would like to get updates faster even if it means that there is a remote chance of a problem with them... I bet they'd still want to get updates faster.
- dartos 2y ago
  There must be balance
mewpmewp2 2y ago
I would say that canary release is an absolute must 100%. Except I can think of cases where it might still not be enough. So, I just don't feel comfortable judging them out of the box. Does all the evidence seem to point against them? For sure. But I just don't feel comfortable giving that final verdict without knowing for sure. Specifically because this is about fighting against malicious actors, where time can be of essence to deploy some sort of protection against a novel threat. If there's deadlines that you can go over, and nothing bad happens, for sure. Always have canary releases, and perfect QA, monitoring everything thoroughly, but I'm just saying, there can be cases where damage that could be done if you don't act fast enough, is just so much worse. And I don't know that it wasn't the case for them. I just don't know.
- acdha 2y ago
  > Specifically because this is about fighting against malicious actors, where time can be of essence to deploy some sort of protection against a novel threat. This is severely overstating the problem: an extra few minutes is not going to be the difference between their customers being compromised. Most of the devices they run on are never compromised, because anyone remotely serious has defense in depth. If it was true, or even close to true, that would make the criticism more rather than less strong. If time is of the essence, you invest in things like reviewing test coverage (their most glaring lapse), fuzz testing, and common reliability engineering techniques like having the system roll back to the last known good configuration after it’s failed to load. We think of progressive rollouts as common now but they got to get that mainstream in large part because the Google Chrome team realized rapid updates are important but then asked what they needed to do to make them safe. CrowdStrike’s report suggests that they wanted rapid but weren’t willing to invest in the implementation because that isn’t a customer-visible feature – until it very painfully became one.
- dartos 2y ago
  In this case, they pretty much caused a worst case scenario…

canucker2016 2y ago

They literally half-assed their deployment process - one part enterprisey, one part "move fast and break things". Guess which part took down much of the corporate world? from Preliminary Post Incident Review at https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio... : "CrowdStrike delivers security content configuration updates to our sensors in two ways: Sensor Content that is shipped with our sensor directly, and Rapid Response Content that is designed to respond to the changing threat landscape at operational speed. ... The sensor release process begins with automated testing, both prior to and after merging into our code base. This includes unit testing, integration testing, performance testing and stress testing. This culminates in a staged sensor rollout process that starts with dogfooding internally at CrowdStrike, followed by early adopters. It is then made generally available to customers. Customers then have the option of selecting which parts of their fleet should install the latest sensor release (‘N’), or one version older (‘N-1’) or two versions older (‘N-2’) through Sensor Update Policies. The event of Friday, July 19, 2024 was not triggered by Sensor Content, which is only delivered with the release of an updated Falcon sensor. Customers have complete control over the deployment of the sensor — which includes Sensor Content and Template Types. ... Rapid Response Content is used to perform a variety of behavioral pattern-matching operations on the sensor using a highly optimized engine. Newly released Template Types are stress tested across many aspects, such as resource utilization, system performance impact and event volume. For each Template Type, a specific Template Instance is used to stress test the Template Type by matching against any possible value of the associated data fields to identify adverse system interactions. Template Instances are created and configured through the use of the Content Configuration System, which includes the Content Validator that performs validation checks on the content before it is published. On July 19, 2024, two additional IPC Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data. Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production."

hello_moto 2y ago
> one part enterprisey, one part "move fast and break things". When there's 0day, how enterprisey you would like to catch the 0day?

Aeolun 2y ago

Nonsense. You don’t need any staged deploys if you simply make no mistakes. /s