3 ms·

A lot of mitigation actions but nothing to really stop it happening again: a fail safe system in their boot start driver. Bad programming and QA caused the issu

A lot of mitigation actions but nothing to really stop it happening again: a fail safe system in their boot start driver. Bad programming and QA caused the issue, but bad design allowed it to happen

l00tr 2y ago
famous windows "guru" Alex Ionescu was their main kernel architector for long time, funny he didn't comment anything about that fail
simiones 2y ago
I think the QA issues are by far the most important part. A security component of this type, by its nature, has to be able to prevent your computer from doing anything at all, since any part of userspace (at least) could be compromised. The "fail safe" for a security component is in fact to prevent any user space code from running at all - better that than having it actively harm other systems, exfiltrate data, destroy connected hardware etc. So, no amount of clever design can prevent the CrowdStrike sensor from nuking your system if bad security rules get deployed. For example, if a bad definition file makes it think that the legit libc or win32 libraries are compromised, it should prevent any userspace program from running, which is just as destructive as failing during boot. That is why appropriate QA is critical for this type of program. I would expect any definition update of any kind to be tested on dozens of systems with a wide variety of Windows configurations and known-good software far before ever being deployed to any customer system. It seems that CrowdStrike thought the exact opposite of this, and in fact their customers were the first to ever run their new code end-to-end, not the last...
- acdha 2y ago
  > So, no amount of clever design can prevent the CrowdStrike sensor from nuking your system if bad security rules get deployed. This is too binary a way to think about a complex system. Availability is also a security goal so we shouldn’t cavalierly trade it for minor risks which are mostly edge cases. For example, say that the fail-safe was an old, old idea where they kept the second most recent version, and if the system failed to start or crashed repeatedly, it automatically rolled back to the last known good version. That turns this kind of problem into at most a reboot – a huge win every customer would have taken - and the only case it would introduce a vulnerability is if there’s an active attack which only the latest rules will block which is so virulent that the number of systems approximates the number who’ll be affected by a bad update. That’s an unlikely set of events, especially because there’s a really tight window where such a fast-spreading attack wouldn’t have compromised the host before CrowdStrike could ship the update. Another variation of that idea: any time the system fails to start repeatedly, the service blocks processes other than its updater so normal apps aren’t exposed as potential vectors but the system can self-heal in most cases.

A lot of mitigation actions but nothing to really stop it happening again: a fail safe system in their boot start driver. Bad programming and QA caused the issue, but bad design allowed it to happen

l00tr 2y ago

famous windows "guru" Alex Ionescu was their main kernel architector for long time, funny he didn't comment anything about that fail

simiones 2y ago

I think the QA issues are by far the most important part. A security component of this type, by its nature, has to be able to prevent your computer from doing anything at all, since any part of userspace (at least) could be compromised. The "fail safe" for a security component is in fact to prevent any user space code from running at all - better that than having it actively harm other systems, exfiltrate data, destroy connected hardware etc. So, no amount of clever design can prevent the CrowdStrike sensor from nuking your system if bad security rules get deployed. For example, if a bad definition file makes it think that the legit libc or win32 libraries are compromised, it should prevent any userspace program from running, which is just as destructive as failing during boot. That is why appropriate QA is critical for this type of program. I would expect any definition update of any kind to be tested on dozens of systems with a wide variety of Windows configurations and known-good software far before ever being deployed to any customer system. It seems that CrowdStrike thought the exact opposite of this, and in fact their customers were the first to ever run their new code end-to-end, not the last...

acdha 2y ago
> So, no amount of clever design can prevent the CrowdStrike sensor from nuking your system if bad security rules get deployed. This is too binary a way to think about a complex system. Availability is also a security goal so we shouldn’t cavalierly trade it for minor risks which are mostly edge cases. For example, say that the fail-safe was an old, old idea where they kept the second most recent version, and if the system failed to start or crashed repeatedly, it automatically rolled back to the last known good version. That turns this kind of problem into at most a reboot – a huge win every customer would have taken - and the only case it would introduce a vulnerability is if there’s an active attack which only the latest rules will block which is so virulent that the number of systems approximates the number who’ll be affected by a bad update. That’s an unlikely set of events, especially because there’s a really tight window where such a fast-spreading attack wouldn’t have compromised the host before CrowdStrike could ship the update. Another variation of that idea: any time the system fails to start repeatedly, the service blocks processes other than its updater so normal apps aren’t exposed as potential vectors but the system can self-heal in most cases.