4 ms·

For me, number 1 is the worst of the bunch. You should always expect that there will be bugs in processes, input files, etc… the fact that their code wasn’t rob

For me, number 1 is the worst of the bunch. You should always expect that there will be bugs in processes, input files, etc… the fact that their code wasn’t robust enough to recognize a corrupted file and not crash is inexcusable. Especially in kernel code that is so widely deployed. If any one of the five points above hadn’t happened, this event would have been avoided. However, if number 1 had been addressed - any of the others could have happened (or all at the same time) and it would have been fine. I understand that we should assume that bugs will be present anywhere, which is why staggered deployments are also important. If there had been staggered deployments, the. The damage would have happened, but it would have been localized. I think security people would argue against a staged deployment though, as if it were discovered what the new definitions protected against, an exploit could be developed quickly to put those servers that aren’t in the “canary” group at risk. (At least in theory — I can’t see how staggering deployment over a 6-12 hour window would have been that risky).

timmytokyo 2y ago
They're all terrible, but I agree #1 is particularly egregious for a company ostensibly dedicated to security. A simple fuzz tester would have caught this type of bug, so they clearly don't perform even a minimal amount of testing on their code.
- nsguy 2y ago
  Totally agree. Not only would a coverage guided fuzzer catch this they should also be adding every single file they send out to the corpus of that automated fuzz testing so they can get somewhat increased coverage on their parser. There may not be out of the box fuzzers that test device drivers so you hoist all the parser code, build it into a stand-alone application, and fuzz that. Likely this is a form of technical debt since I can understand not doing all of this day #1 when you have 5 customers but at some point as you scale up you need to change the way you look at risk.
- fluidcruft 2y ago
  Seems like it would be easy enough to add a new checkbox for this to audits.
jayd16 2y ago
You admit that bugs are inevitable and then claim a bug free parser as the most important bullet. That seems flawed to me. It would certainly be nice, but is that achievable? Policy changes seem more reliable and would catch other, as of yet unknown classes of bugs.
- marsten 2y ago
  This shouldn't be an either-or situation; you do all of the above. A simple validating parser in the client would be easy to write and would have easily caught a null payload. What looks especially bad for Crowdstrike is how many things (relatively simple things) had to fail in order for this to slip through. It's like walking into Fort Knox, grabbing a gold bar, and walking out unimpeded. A complete systemic failure.
throwaway5752 2y ago
I disagree. Has to be 4, something will always go wrong, so you have to deliver in cohorts. That goes equally if it was a Windows Update rolled out in one motion that broke the falcon agent/driver, or if it was Crowdstrike. There is almost no excuse for a global rollout without telemetry checks, whether it's security agent updates or os patches.
contravariant 2y ago
It might be the worst mistake, but number 1 was always going to happen sometime. And even testing can't be trusted 100%, because writing code that does the right thing and code that tests things correctly are about equally hard, they just aren't always hard simultaneously.

timmytokyo 2y ago

They're all terrible, but I agree #1 is particularly egregious for a company ostensibly dedicated to security. A simple fuzz tester would have caught this type of bug, so they clearly don't perform even a minimal amount of testing on their code.

nsguy 2y ago
Totally agree. Not only would a coverage guided fuzzer catch this they should also be adding every single file they send out to the corpus of that automated fuzz testing so they can get somewhat increased coverage on their parser. There may not be out of the box fuzzers that test device drivers so you hoist all the parser code, build it into a stand-alone application, and fuzz that. Likely this is a form of technical debt since I can understand not doing all of this day #1 when you have 5 customers but at some point as you scale up you need to change the way you look at risk.
fluidcruft 2y ago
Seems like it would be easy enough to add a new checkbox for this to audits.

jayd16 2y ago

You admit that bugs are inevitable and then claim a bug free parser as the most important bullet. That seems flawed to me. It would certainly be nice, but is that achievable? Policy changes seem more reliable and would catch other, as of yet unknown classes of bugs.

marsten 2y ago
This shouldn't be an either-or situation; you do all of the above. A simple validating parser in the client would be easy to write and would have easily caught a null payload. What looks especially bad for Crowdstrike is how many things (relatively simple things) had to fail in order for this to slip through. It's like walking into Fort Knox, grabbing a gold bar, and walking out unimpeded. A complete systemic failure.

throwaway5752 2y ago

I disagree. Has to be 4, something will always go wrong, so you have to deliver in cohorts. That goes equally if it was a Windows Update rolled out in one motion that broke the falcon agent/driver, or if it was Crowdstrike. There is almost no excuse for a global rollout without telemetry checks, whether it's security agent updates or os patches.

contravariant 2y ago

It might be the worst mistake, but number 1 was always going to happen sometime. And even testing can't be trusted 100%, because writing code that does the right thing and code that tests things correctly are about equally hard, they just aren't always hard simultaneously.