10 ms·
again, this is why I was snarky in my earlier post, this was not a software update. they should have used canary deployments still but in many cases prior to th
by notepad0x90 2y ago
again, this is why I was snarky in my earlier post, this was not a software update. they should have used canary deployments still but in many cases prior to this incident, it was not acceptable to wait even a few hours because it can make the difference between companies getting ransomwared/hacked, so they focused on making the actual code/driver that interprets the channel file updates robust enough to handle real-time updates. Even if other players were doing canary deployments with behavioral detection updates, they're not the market leader, crowdstrike is for a reason.
Everyone that worked in an operational incident response role has blocked some indicator like an ip address or a domain. you don't do gradual roll outs for those either, and i've seen people cause outages by skipping a check or making a mistake. this is similar in many ways to that except it was for a named pipe. This could probably have waited for a canary deployment, but in general the class of content that is being deployed would be deployed right away, I'd be surprised if their practice is considered "bad" by any measure. I've seen Microsoft also deploy email quarantine signatures and defender updates that caused large scale impacts.
Here is a link of what Microsoft did earlier this year:
https://www.techradar.com/news/google-chrome-not-working-microsoft-defender-may-be-to-blame https://www.techradar.com/news/google-chrome-not-working-mic...
If they had canary deployments, that wouldn't have happened. I had rules that were causing chaos because of that. Now imagine if defender had a bug that caused it to crash because of a signature update. The impact would be magnitudes greater than what you saw with Crowdstrike. It's really frustrating to see the lack of technical critical thinking and arm-chair experts acting like they know what they're talking about.
- Dylan16807 2y ago> it was not acceptable to wait even a few hours Hours... Wouldn't a 15 minute canary have found this problem about 14 minutes before it hit wider deployment?
- mort96 2y agoLet's say the driver was "robust enough" to handle a broken channel file. How would that look exactly? Say you're responsible for writing the code which loads a new channel file. These channel files are critical; without them, your security critical product doesn't know how to do its job. The channel file parser returns a parse error. How should the driver respond? Surely you're not going to just silently disable your security critical product if someone puts a bad channel file in there?
- PleasureBot 2y agoDelete the file or mark it as corrupt so that the parser doesn't keep trying to read it, and send some telemetry back to CS to indicate there is a problem with the one of the channel files. It doesn't seem very complicated at all. There are plenty of options in between "catastrophically crash the OS" and "silently disable the entire product".
- mort96 2y agoThat seems pretty dangerous if that channel file included security critical configuration, which it presumably did