4 ms·

This was apparently caused by a faulty "channel file"[0], which is presumably some kind of configuration database that the software uses to identify malware. S

This was apparently caused by a faulty "channel file"[0], which is presumably some kind of configuration database that the software uses to identify malware. So there wasn't any new kernel driver deployed, the existing kernel driver just doesn't fail gracefully. [0]: https://x.com/brody_n77/status/1814185935476863321 https://x.com/brody_n77/status/1814185935476863321

vesinisa 2y ago
Why on earth don't they have staged rollouts for updates?
- tehlike 2y ago
  Everytime i look into such catastrophic issues, it always boils down to lack of robust canarying mechanisms. They have enough client base that they can even run an A/B test on the whole binary level, but no.
- malkia 2y ago
  Also, why not have some sort of graceful degradation (well kind of), like: OS Boots, loads CS driver, the driver loads some new feature/config, and before/after new recent thing ("runtime flag") marked whether it successfully worked, and if not on the next reboot that thing gets either disabled, or the previous known good config (obviously some combination of things might cause another issue), but instead of blindly rebooting to the same state....
  - ta345346633 2y ago
    I think pfsense does this (from memory, been a while using it). Basically dual-partitions, and if it failed to come up on the active partition after an update it'd revert. Granted you need to have the space to have two partitions, but for a small partition/image not so bad. What surprises me is if its a content update, and the code fell over when dealing with it - just basically bad release engineering isn't it not to cater for that in the first place? i.e. some tests in the pipeline before releasing the content update would've picked it up given it sounds like 100% failure rates.
  - butlike 2y ago
    The problem space kind of dictates that this couldn't be a solution, cause malware could load an arbitrary feature/config and mark it as 0, then the AV would be disabled on next boot, right?
    - malkia 2y ago
      fair point indeed!
- averageRoyalty 2y ago
  More importantly, why are CS customers not validating? Upstream patches should be treated as faulty/malicious if not tested to show otherwise, especially if they're kernel level.
- callalex 2y ago
  Why put effort in engineering when you can just fear monger in marketing and buy politicians in sales?

vesinisa 2y ago

Why on earth don't they have staged rollouts for updates?

tehlike 2y ago
Everytime i look into such catastrophic issues, it always boils down to lack of robust canarying mechanisms. They have enough client base that they can even run an A/B test on the whole binary level, but no.
malkia 2y ago
Also, why not have some sort of graceful degradation (well kind of), like: OS Boots, loads CS driver, the driver loads some new feature/config, and before/after new recent thing ("runtime flag") marked whether it successfully worked, and if not on the next reboot that thing gets either disabled, or the previous known good config (obviously some combination of things might cause another issue), but instead of blindly rebooting to the same state....
- ta345346633 2y ago
  I think pfsense does this (from memory, been a while using it). Basically dual-partitions, and if it failed to come up on the active partition after an update it'd revert. Granted you need to have the space to have two partitions, but for a small partition/image not so bad. What surprises me is if its a content update, and the code fell over when dealing with it - just basically bad release engineering isn't it not to cater for that in the first place? i.e. some tests in the pipeline before releasing the content update would've picked it up given it sounds like 100% failure rates.
- butlike 2y ago
  The problem space kind of dictates that this couldn't be a solution, cause malware could load an arbitrary feature/config and mark it as 0, then the AV would be disabled on next boot, right?
  - malkia 2y ago
    fair point indeed!
averageRoyalty 2y ago
More importantly, why are CS customers not validating? Upstream patches should be treated as faulty/malicious if not tested to show otherwise, especially if they're kernel level.
callalex 2y ago
Why put effort in engineering when you can just fear monger in marketing and buy politicians in sales?