5 ms·

> The "content update" that CrowdStrike sent out was full of zeroes. Nothing else. Obviously not the intended content. And this simple data caused the driver to

> The "content update" that CrowdStrike sent out was full of zeroes. Nothing else. Obviously not the intended content. And this simple data caused the driver to crash CrowdStrike has stated that no the crash was not related to the file of zeros.

svantex 2y ago
Interesting, do you have a link to this statement? Also, do they state what did cause the crash? At least removing the file of zeroes does solve the problem, as the instructions both from Microsoft and CrowdStrike states "Boot into safe mode. Delete C-00000291*.sys." That's the file(s) with the zeroes... See https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio... and https://www.youtube.com/watch?v=Bn5eRUaMZXk https://www.youtube.com/watch?v=Bn5eRUaMZXk (3 minutes 20 seconds in).
- chrisjj 2y ago
  Yes I do. https://www.crowdstrike.com/blog/falcon-update-for-windows-hosts-technical-details/ https://www.crowdstrike.com/blog/falcon-update-for-windows-h... > CrowdStrike states "Boot into safe mode. Delete C-00000291*.sys." That's the file(s) with the zeroes That's potentially multiple files, but do we know only one comprises just zeros?
  - svantex 2y ago
    Right, they write rather cryptically "This is not related to null bytes contained within Channel File 291 or any other Channel File." That's not quite the same as saying "This is not related to Channel File 291 containing all nul bytes."... I don't have first to hand knowledge here, but rely on Dave Plummer's statement. Regardless of zeroes or single files or not, the fact is that bad data in C-00000291.sys in combination with bad validition in the driver causes it to crash. Deleting C-00000291.sys causes the driver to stop crashing. Anyway, my main point isn't really about this. It's about the big bang global roll out simultaneously to at least 8.5 million systems in one go that's irresponsible. The driver architecture is the lesser evil here, although it's bad enough!
    - chrisjj 2y ago
      > the fact is that bad data in C-00000291.sys in combination with bad validition in the driver causes it to crash. I think we've seen no evidence that data is to blame. > Deleting C-00000291.sys causes the driver to stop crashing. So perhaps just its existence is to blame. > The driver architecture is the lesser evil here Except if the crash had been limited to the driver, it would have left the machine running unprotected which is far greater an evil.
      - svantex 2y ago
        CrowdStrike does confirm that the data is to blame. "problematic content in Channel File 291 resulted in an out-of-bounds memory read triggering an exception" https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio... .
        chrisjj 2y ago
        Yes - subsequent to my comment. Thanks. But how can this latest statement can be true, if the previous statement that the crash was not related to the zero bytes content is true?
        svantex 2y ago
        Good question. There's some evidence that not all affected systems has seen this 'all zeroes' file, the first account stories varies. But something was definitely broken in the deployed data. But, once again, CrowdStrike does not paint a clear picture and it raises new questions and only partially answers old ones. Why is it so hard for manufacturers to just go ahead and explain what really went wrong, without a lot of corporate b..t? Probably, if they do really say what happened in so many words they might open themselves for negligence lawsuits. Hopefully somebody files one anyway. The industry needs to learn to be better, and the only thing that talks loudly enough is probably money. Lost revenue, liability damages, and share holder value loss.
    - FreakLegion 2y ago
- gruez 2y ago
  AFAIK in one of the older crowdstrike threads, there was a tweet that said the driver checked for a sentinel value of AAAAA... before loading it, so an entirely blank value wouldn't have caused the issue. I can't find the source now, but some comments do seem to corroborate it: https://news.ycombinator.com/item?id=41005546 https://news.ycombinator.com/item?id=41005546
- GuB-42 2y ago
  Speculation: this "all zero" file is part of a signed batch, they have to have signatures, they are not that dumb (I hope...). By removing a file, the batch becomes incomplete, fails the check, and some corruption recovery mechanism takes over, most likely disabling the update and triggering an update. In the meantime, they fixed the content update, fixing the crash.

svantex 2y ago

Interesting, do you have a link to this statement? Also, do they state what did cause the crash? At least removing the file of zeroes does solve the problem, as the instructions both from Microsoft and CrowdStrike states "Boot into safe mode. Delete C-00000291*.sys." That's the file(s) with the zeroes... See https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio... and https://www.youtube.com/watch?v=Bn5eRUaMZXk https://www.youtube.com/watch?v=Bn5eRUaMZXk (3 minutes 20 seconds in).

chrisjj 2y ago
Yes I do. https://www.crowdstrike.com/blog/falcon-update-for-windows-hosts-technical-details/ https://www.crowdstrike.com/blog/falcon-update-for-windows-h... > CrowdStrike states "Boot into safe mode. Delete C-00000291*.sys." That's the file(s) with the zeroes That's potentially multiple files, but do we know only one comprises just zeros?
- svantex 2y ago
  Right, they write rather cryptically "This is not related to null bytes contained within Channel File 291 or any other Channel File." That's not quite the same as saying "This is not related to Channel File 291 containing all nul bytes."... I don't have first to hand knowledge here, but rely on Dave Plummer's statement. Regardless of zeroes or single files or not, the fact is that bad data in C-00000291.sys in combination with bad validition in the driver causes it to crash. Deleting C-00000291.sys causes the driver to stop crashing. Anyway, my main point isn't really about this. It's about the big bang global roll out simultaneously to at least 8.5 million systems in one go that's irresponsible. The driver architecture is the lesser evil here, although it's bad enough!
  - chrisjj 2y ago
    > the fact is that bad data in C-00000291.sys in combination with bad validition in the driver causes it to crash. I think we've seen no evidence that data is to blame. > Deleting C-00000291.sys causes the driver to stop crashing. So perhaps just its existence is to blame. > The driver architecture is the lesser evil here Except if the crash had been limited to the driver, it would have left the machine running unprotected which is far greater an evil.
    - svantex 2y ago
      CrowdStrike does confirm that the data is to blame. "problematic content in Channel File 291 resulted in an out-of-bounds memory read triggering an exception" https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio... .
      - chrisjj 2y ago
        Yes - subsequent to my comment. Thanks. But how can this latest statement can be true, if the previous statement that the crash was not related to the zero bytes content is true?
        svantex 2y ago
        Good question. There's some evidence that not all affected systems has seen this 'all zeroes' file, the first account stories varies. But something was definitely broken in the deployed data. But, once again, CrowdStrike does not paint a clear picture and it raises new questions and only partially answers old ones. Why is it so hard for manufacturers to just go ahead and explain what really went wrong, without a lot of corporate b..t? Probably, if they do really say what happened in so many words they might open themselves for negligence lawsuits. Hopefully somebody files one anyway. The industry needs to learn to be better, and the only thing that talks loudly enough is probably money. Lost revenue, liability damages, and share holder value loss.
  - FreakLegion 2y ago
gruez 2y ago
AFAIK in one of the older crowdstrike threads, there was a tweet that said the driver checked for a sentinel value of AAAAA... before loading it, so an entirely blank value wouldn't have caused the issue. I can't find the source now, but some comments do seem to corroborate it: https://news.ycombinator.com/item?id=41005546 https://news.ycombinator.com/item?id=41005546
GuB-42 2y ago
Speculation: this "all zero" file is part of a signed batch, they have to have signatures, they are not that dumb (I hope...). By removing a file, the batch becomes incomplete, fails the check, and some corruption recovery mechanism takes over, most likely disabling the update and triggering an update. In the meantime, they fixed the content update, fixing the crash.