7 ms·

Did either release from MS or Crowdstrike explain how this crash bypassed QC? I'm still baffled that a 100% repro crash even made it anywhere near the later st

Did either release from MS or Crowdstrike explain how this crash bypassed QC? I'm still baffled that a 100% repro crash even made it anywhere near the later stages of QC. This is something easily caught by the earliest CI phases , at the developer and at least first build automation phase, let alone human QC.

magicalhippo 2y ago
From what I read in the previous thread, their test environment didn't actually test what was deployed. That is, there was a post-test pre-distribution packaging stage, and that's where the distributed file(s) got f'ed up. If true that would explain how it got past their testing, but would also be an incredible lack of competence IMHO. But yeah, curious if there's been some more concrete details there.
- tonymet 2y ago
  I heard something similar. that they deploy content separately from code, but they don't test all of the combinations of code + content. This crash was from "stable" code in the driver mixed with a corrupt or incomplete content file (config, etc) , triggering the null-ptr exception . Sounds like one of those companies where you get hired and are shocked by the sausage factory you just stepped into
  - rvnx 2y ago
    In February they added new code that allows to spy/block named pipes. Named pipes are pipes of communication that processes can use to talk to each other, as an alternative to sockets. For example Chrome uses them between the user interface and the actual page renderer. In March they tested it in staging, said it was fine, pushed to prod with few rules in April, still looked fine. In July they added a new rule, which was deployed to 100% immediately, as from their perspective, a new entry in a database definition doesn't need testing nor canary deploy (which is still irresponsible, because bad rules could cause damage as well like any security/antivirus software, even if the parser didn't crash, but it could have blocked legitimate actions or files)
    - tonymet 2y ago
      great summary thanks for the details. I hope more companies see this and consider adding more test diagnostics
  - magicalhippo 2y ago
    Right, so it seems two egregious errors: no (or highly lacking) fuzzing of kernel modules acceping arbitrary input, and no testing of configuration changes given it's ingested by a kernel module.
pas 2y ago
lack of fuzzing for their "parser + updater"
- MBCook 2y ago
  Plus parsing unknown files (as in not validated to be properly formatted) in kernel space is just asking for a crash.
  - pas 2y ago
    parsing is real validating! (obligatory "Parse, don't validate" post https://news.ycombinator.com/item?id=35053118 https://news.ycombinator.com/item?id=35053118 and some more discussion on the topic https://news.ycombinator.com/item?id=39322551 https://news.ycombinator.com/item?id=39322551 )
duped 2y ago
My understanding is that they ship precompiled, templated scripts. The "content" updates fill in these templates with configured values. They test the templated scripts, and they validate the content, but they don't validate the content bound to the script. The garbage content was apparently valid but its behavior when used was not. Their language for describing this design is obtuse and confusing.

magicalhippo 2y ago

From what I read in the previous thread, their test environment didn't actually test what was deployed. That is, there was a post-test pre-distribution packaging stage, and that's where the distributed file(s) got f'ed up. If true that would explain how it got past their testing, but would also be an incredible lack of competence IMHO. But yeah, curious if there's been some more concrete details there.

tonymet 2y ago
I heard something similar. that they deploy content separately from code, but they don't test all of the combinations of code + content. This crash was from "stable" code in the driver mixed with a corrupt or incomplete content file (config, etc) , triggering the null-ptr exception . Sounds like one of those companies where you get hired and are shocked by the sausage factory you just stepped into
- rvnx 2y ago
  In February they added new code that allows to spy/block named pipes. Named pipes are pipes of communication that processes can use to talk to each other, as an alternative to sockets. For example Chrome uses them between the user interface and the actual page renderer. In March they tested it in staging, said it was fine, pushed to prod with few rules in April, still looked fine. In July they added a new rule, which was deployed to 100% immediately, as from their perspective, a new entry in a database definition doesn't need testing nor canary deploy (which is still irresponsible, because bad rules could cause damage as well like any security/antivirus software, even if the parser didn't crash, but it could have blocked legitimate actions or files)
  - tonymet 2y ago
    great summary thanks for the details. I hope more companies see this and consider adding more test diagnostics
- magicalhippo 2y ago
  Right, so it seems two egregious errors: no (or highly lacking) fuzzing of kernel modules acceping arbitrary input, and no testing of configuration changes given it's ingested by a kernel module.

pas 2y ago

lack of fuzzing for their "parser + updater"

MBCook 2y ago
Plus parsing unknown files (as in not validated to be properly formatted) in kernel space is just asking for a crash.
- pas 2y ago
  parsing is real validating! (obligatory "Parse, don't validate" post https://news.ycombinator.com/item?id=35053118 https://news.ycombinator.com/item?id=35053118 and some more discussion on the topic https://news.ycombinator.com/item?id=39322551 https://news.ycombinator.com/item?id=39322551 )

duped 2y ago

My understanding is that they ship precompiled, templated scripts. The "content" updates fill in these templates with configured values. They test the templated scripts, and they validate the content, but they don't validate the content bound to the script. The garbage content was apparently valid but its behavior when used was not. Their language for describing this design is obtuse and confusing.