5 ms·

Every company I've ever been at rolls out updates slowly. Rolling out a change to 8.5 million computers at the same time seems ridiculous. Even the most cash st

by mort96 2y ago

Every company I've ever been at rolls out updates slowly. Rolling out a change to 8.5 million computers at the same time seems ridiculous. Even the most cash strapped start-ups with every incentive to cut corners tends to get staged roll-outs more or less right. It's crazy.

binkHN 2y ago
Beyond crazy. I even have a small app that never makes it to production before being rolled out to internal and open testing first. And, even then, it's slowly rolled out to a percentage at each stage before being fully deployed. One would think a major company with kernel level access would do this at minimum.
geon 2y ago
I had a fleet of only maybe 200 computers I updated remotely. I did canary staged roll outs.
- doubled112 2y ago
  When I managed ~ 15 developer’s Arch Linux workstations, I found it very beneficial to be the canary, and then rollout to a couple of the more capable of troubleshooting devs, and then the rest. I can always fix my own box. 8.5M all at once feels insane.
- notepad0x90 2y ago
  not a software update!
  - mort96 2y ago
    Not relevant!
    - notepad0x90 2y ago
      details are always relevant in a technical discussion. look at my other comments where i pointed out microsoft performing similar immediate av signature updates and causing chaos.
      - mort96 2y ago
        Some details are relevant, some are not. I'm more than comfortable labelling parts of Microsoft as incompetent as well.
        notepad0x90 2y ago
        We can agree on that, but it is relevant because this isn't an unusual practice. Crowdstrike didn't ignore some pre-existing best practice. Lots of things need improving but facts and details matter when you talk about RCA. it isn't about blame but fixing the root cause.
        mort96 2y ago
        Deploying an update to 8.5 million devices without doing any form of staged roll-outs, not even just updating a few machines in the office first, is absolutely bonkers. If that's not an unusual practice it bloody should be. I don't care if it's a software update or a config update.
notepad0x90 2y ago
again, this is why I was snarky in my earlier post, this was not a software update. they should have used canary deployments still but in many cases prior to this incident, it was not acceptable to wait even a few hours because it can make the difference between companies getting ransomwared/hacked, so they focused on making the actual code/driver that interprets the channel file updates robust enough to handle real-time updates. Even if other players were doing canary deployments with behavioral detection updates, they're not the market leader, crowdstrike is for a reason. Everyone that worked in an operational incident response role has blocked some indicator like an ip address or a domain. you don't do gradual roll outs for those either, and i've seen people cause outages by skipping a check or making a mistake. this is similar in many ways to that except it was for a named pipe. This could probably have waited for a canary deployment, but in general the class of content that is being deployed would be deployed right away, I'd be surprised if their practice is considered "bad" by any measure. I've seen Microsoft also deploy email quarantine signatures and defender updates that caused large scale impacts. Here is a link of what Microsoft did earlier this year: https://www.techradar.com/news/google-chrome-not-working-microsoft-defender-may-be-to-blame https://www.techradar.com/news/google-chrome-not-working-mic... If they had canary deployments, that wouldn't have happened. I had rules that were causing chaos because of that. Now imagine if defender had a bug that caused it to crash because of a signature update. The impact would be magnitudes greater than what you saw with Crowdstrike. It's really frustrating to see the lack of technical critical thinking and arm-chair experts acting like they know what they're talking about.
- Dylan16807 2y ago
  > it was not acceptable to wait even a few hours Hours... Wouldn't a 15 minute canary have found this problem about 14 minutes before it hit wider deployment?
- mort96 2y ago
  Let's say the driver was "robust enough" to handle a broken channel file. How would that look exactly? Say you're responsible for writing the code which loads a new channel file. These channel files are critical; without them, your security critical product doesn't know how to do its job. The channel file parser returns a parse error. How should the driver respond? Surely you're not going to just silently disable your security critical product if someone puts a bad channel file in there?

by mort96 2y ago

binkHN 2y ago

Beyond crazy. I even have a small app that never makes it to production before being rolled out to internal and open testing first. And, even then, it's slowly rolled out to a percentage at each stage before being fully deployed. One would think a major company with kernel level access would do this at minimum.

geon 2y ago

I had a fleet of only maybe 200 computers I updated remotely. I did canary staged roll outs.

doubled112 2y ago
When I managed ~ 15 developer’s Arch Linux workstations, I found it very beneficial to be the canary, and then rollout to a couple of the more capable of troubleshooting devs, and then the rest. I can always fix my own box. 8.5M all at once feels insane.
notepad0x90 2y ago
not a software update!
- mort96 2y ago
  Not relevant!
  - notepad0x90 2y ago
    details are always relevant in a technical discussion. look at my other comments where i pointed out microsoft performing similar immediate av signature updates and causing chaos.
    - mort96 2y ago
      Some details are relevant, some are not. I'm more than comfortable labelling parts of Microsoft as incompetent as well.
      - notepad0x90 2y ago
        We can agree on that, but it is relevant because this isn't an unusual practice. Crowdstrike didn't ignore some pre-existing best practice. Lots of things need improving but facts and details matter when you talk about RCA. it isn't about blame but fixing the root cause.
        mort96 2y ago
        Deploying an update to 8.5 million devices without doing any form of staged roll-outs, not even just updating a few machines in the office first, is absolutely bonkers. If that's not an unusual practice it bloody should be. I don't care if it's a software update or a config update.

notepad0x90 2y ago

again, this is why I was snarky in my earlier post, this was not a software update. they should have used canary deployments still but in many cases prior to this incident, it was not acceptable to wait even a few hours because it can make the difference between companies getting ransomwared/hacked, so they focused on making the actual code/driver that interprets the channel file updates robust enough to handle real-time updates. Even if other players were doing canary deployments with behavioral detection updates, they're not the market leader, crowdstrike is for a reason. Everyone that worked in an operational incident response role has blocked some indicator like an ip address or a domain. you don't do gradual roll outs for those either, and i've seen people cause outages by skipping a check or making a mistake. this is similar in many ways to that except it was for a named pipe. This could probably have waited for a canary deployment, but in general the class of content that is being deployed would be deployed right away, I'd be surprised if their practice is considered "bad" by any measure. I've seen Microsoft also deploy email quarantine signatures and defender updates that caused large scale impacts. Here is a link of what Microsoft did earlier this year: https://www.techradar.com/news/google-chrome-not-working-microsoft-defender-may-be-to-blame https://www.techradar.com/news/google-chrome-not-working-mic... If they had canary deployments, that wouldn't have happened. I had rules that were causing chaos because of that. Now imagine if defender had a bug that caused it to crash because of a signature update. The impact would be magnitudes greater than what you saw with Crowdstrike. It's really frustrating to see the lack of technical critical thinking and arm-chair experts acting like they know what they're talking about.

Dylan16807 2y ago
> it was not acceptable to wait even a few hours Hours... Wouldn't a 15 minute canary have found this problem about 14 minutes before it hit wider deployment?
mort96 2y ago
Let's say the driver was "robust enough" to handle a broken channel file. How would that look exactly? Say you're responsible for writing the code which loads a new channel file. These channel files are critical; without them, your security critical product doesn't know how to do its job. The channel file parser returns a parse error. How should the driver respond? Surely you're not going to just silently disable your security critical product if someone puts a bad channel file in there?