3 ms·

Zero effort to fuzz test the parser too. I mean, we know how to harden parsers against bugs and attacks, and any semi-competent fuzzer would have caught such a

by rwmj 2y ago

Zero effort to fuzz test the parser too. I mean, we know how to harden parsers against bugs and attacks, and any semi-competent fuzzer would have caught such a trivial bug.

chrisjj 2y ago
The triggering file was all zeros. It is not possible that only this pattern caused the crash, and fuzzing omitted to try this unfuzzy pattern?
- monsieurbanana 2y ago
  In my limited experience, I thought any serious fuzzing program does test for all "standard" patters like only null bytes, empty strings, etc...
- gliptic 2y ago
  No, it wasn't. Crowdstrike denied it had to do with zeros in the files.
  - jojobas 2y ago
    At this point I wouldn't be paying too much attention to what Crowdstrike is saying.
    - hello_moto 2y ago
      Have to speak the truth albeit at minimum, in case legal...
      - kchr 2y ago
        Which also explains why they, only if needed to cover their back legally, confirm or deny details being shared on social and mass media.
- watwut 2y ago
  Possible? Yes. Likely? No.
- formerly_proven 2y ago
  Instrumented fuzzing (like AFL and friends) tweaks the input to traverse unseen code paths in the target, so they're super quick to find stuff like "heyyyyy, nobody is actually checking if this offset is in bounds before loading from that address".
- omeid2 2y ago
  The files in question has a magic number is "0xAAAAAAAA" so it is not possible that the file was all zeros.
- Retr0id 2y ago
  Competent fuzzers don't just use random bytes, they systematically explore the state-space of the target program. If there's a crash state to be found by feeding in a file full of null bytes, it's probably going to be found quickly. A fun example is that if you point AFL at a JPEG parser, it will eventually "learn" to produce valid JPEG files as test cases, without ever having been told what JPEG file is supposed to look like. https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-th...
  - rwmj 2y ago
    AFL is really "magical". It finds bugs very quickly and with little effort on our part except to leave it running and look at the results occasionally. We use it to fuzz test a variety of file formats and network interfaces, including QEMU image parsing, nbdkit, libnbd, hivex. We also use clang's libfuzzer with QEMU which is another good fuzzing solution. There's really no excuse for CrowdStrike not to have been using fuzzing.
- layer8 2y ago
  No, it wasn’t all zeros: https://x.com/patrickwardle/status/1814782404583936170 https://x.com/patrickwardle/status/1814782404583936170
mavhc 2y ago
AV software is a great target for malware, badly written, probably runs too much stuff in the kernel, tries to parse everything
- Comfy-Tinwork 2y ago
  And at the very least straight to system level access if not more.
  - MyFedora 2y ago
    Anti-cheats also whitelist legit AV drivers, even though cheaters exploit them to no end.
  - londons_explore 2y ago
    AV software needs kernel privilidges to have access to everything it needs to inspect, but the actual inspection of that data should be done with no privilidges. I think most AV companies now have a helper process to do that. If you successfully exploit the helper process, the worst damage you ought to be able to do is falsely find files to be clean.
    - bornfreddy 2y ago
      > ...the worst damage you ought to be able to do is... Ought. But it depends on the way the communication with the main process is done. I wouldn't be surprised if the main process trusts the output from the parser just a tiny bit too much.
jatins 2y ago
You are seriously overestimating the engineering practises at these companies. I have worked in "enterprise security" previously though not at this scale. In a previous life I worked with of the engineering leaders currently at Crowdstrike. I'll bet you this company has some arbitrary unit test coverage requirements for PRs which developers game be mocking the heck out of dependencies. I am sure they have some vanity sonarqube integration to ensure great "code quality". This likely also went through manual QA. However I am sure the topic of fuzz testing would not have come up once. These companies sell checkbox compliance, and they themselves develop their software the same way. Checking all the "quality engineering" boxes with very little regards for long term engineering initiatives that would provide real value. And I am not trying to kick Crowdstrike when they are down. It's the state of any software company run by suits with myopic vision. Their engineering blogs and their codebases are poles apart.

by rwmj 2y ago

Zero effort to fuzz test the parser too. I mean, we know how to harden parsers against bugs and attacks, and any semi-competent fuzzer would have caught such a trivial bug.

chrisjj 2y ago

The triggering file was all zeros. It is not possible that only this pattern caused the crash, and fuzzing omitted to try this unfuzzy pattern?

monsieurbanana 2y ago
In my limited experience, I thought any serious fuzzing program does test for all "standard" patters like only null bytes, empty strings, etc...
gliptic 2y ago
No, it wasn't. Crowdstrike denied it had to do with zeros in the files.
- jojobas 2y ago
  At this point I wouldn't be paying too much attention to what Crowdstrike is saying.
  - hello_moto 2y ago
    Have to speak the truth albeit at minimum, in case legal...
    - kchr 2y ago
      Which also explains why they, only if needed to cover their back legally, confirm or deny details being shared on social and mass media.
watwut 2y ago
Possible? Yes. Likely? No.
formerly_proven 2y ago
Instrumented fuzzing (like AFL and friends) tweaks the input to traverse unseen code paths in the target, so they're super quick to find stuff like "heyyyyy, nobody is actually checking if this offset is in bounds before loading from that address".
omeid2 2y ago
The files in question has a magic number is "0xAAAAAAAA" so it is not possible that the file was all zeros.
Retr0id 2y ago
Competent fuzzers don't just use random bytes, they systematically explore the state-space of the target program. If there's a crash state to be found by feeding in a file full of null bytes, it's probably going to be found quickly. A fun example is that if you point AFL at a JPEG parser, it will eventually "learn" to produce valid JPEG files as test cases, without ever having been told what JPEG file is supposed to look like. https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-th...
- rwmj 2y ago
  AFL is really "magical". It finds bugs very quickly and with little effort on our part except to leave it running and look at the results occasionally. We use it to fuzz test a variety of file formats and network interfaces, including QEMU image parsing, nbdkit, libnbd, hivex. We also use clang's libfuzzer with QEMU which is another good fuzzing solution. There's really no excuse for CrowdStrike not to have been using fuzzing.
layer8 2y ago
No, it wasn’t all zeros: https://x.com/patrickwardle/status/1814782404583936170 https://x.com/patrickwardle/status/1814782404583936170

mavhc 2y ago

AV software is a great target for malware, badly written, probably runs too much stuff in the kernel, tries to parse everything

Comfy-Tinwork 2y ago
And at the very least straight to system level access if not more.
- MyFedora 2y ago
  Anti-cheats also whitelist legit AV drivers, even though cheaters exploit them to no end.
- londons_explore 2y ago
  AV software needs kernel privilidges to have access to everything it needs to inspect, but the actual inspection of that data should be done with no privilidges. I think most AV companies now have a helper process to do that. If you successfully exploit the helper process, the worst damage you ought to be able to do is falsely find files to be clean.
  - bornfreddy 2y ago
    > ...the worst damage you ought to be able to do is... Ought. But it depends on the way the communication with the main process is done. I wouldn't be surprised if the main process trusts the output from the parser just a tiny bit too much.

jatins 2y ago

You are seriously overestimating the engineering practises at these companies. I have worked in "enterprise security" previously though not at this scale. In a previous life I worked with of the engineering leaders currently at Crowdstrike. I'll bet you this company has some arbitrary unit test coverage requirements for PRs which developers game be mocking the heck out of dependencies. I am sure they have some vanity sonarqube integration to ensure great "code quality". This likely also went through manual QA. However I am sure the topic of fuzz testing would not have come up once. These companies sell checkbox compliance, and they themselves develop their software the same way. Checking all the "quality engineering" boxes with very little regards for long term engineering initiatives that would provide real value. And I am not trying to kick Crowdstrike when they are down. It's the state of any software company run by suits with myopic vision. Their engineering blogs and their codebases are poles apart.