4 ms·

There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular select

There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effect; and 3. This was cosmically stupid and they’re going to stop doing that. Software that does 1. and 2. has absolutely no place in critical infrastructure like hospitals and emergency services. I predict we’ll see other vendors removing similar bonehead “features” very very quietly over the next few months.

hello_moto 2y ago
> I predict we’ll see other vendors removing similar bonehead “features” very very quietly over the next few months. Absolutely this is what will happen. I don't know much about the practice of AV definition-like feature across Cybersecurity but I would imagine there might be a possibility that no vendors do rolling update today because it involves Opt-in/Opt-out which might influence the vendor's speed to identify attack which in turns affect their "Reputation" as well. "I bought Vendor-A solution but I got hacked and have to pay Ransomware" (with a side note: because I did not consume the latest critical update of AV definition) is what Vendors worried. Now that this Global Outage happened, it will change the landscape a bit.
- XlA5vEKsMISoIln 2y ago
  >Now that this Global Outage happened, it will change the landscape a bit. I seriously doubt that. Questions like "why should we use CrowdStrike" will be met with "suppose they've learned their lesson".
  - hello_moto 2y ago
    I'm referring to the landscape how current Cybersecurity vendors deliver "detection definition" (for lack of better phrase) to their customers. If you don't send them fast to your customer and your customer gets compromised, your reputation gets hit. If you send them fast, this BSOD happened. It's more like damn if you do, damn if you don't.
    - skydhash 2y ago
      > If you don't send them fast to your customer and your customer gets compromised, your reputation gets hit. > If you send them fast, this BSOD happened. > It's more like damn if you do, damn if you don't. What about notifications? If someone has an update policy that disable auto-updates to a critical piece of infrastructure, you can still let him know that there's a critical update is available. Now, he can do follow his own checklist in order to ensure everything goes well.
      - hello_moto 2y ago
        What if they're sleeping and won't read the notification until they wake up? Wouldn't they get compromised?
      - samcat116 2y ago
        most people will defer updates indefinitely if they are able to.
    - XlA5vEKsMISoIln 2y ago
      Okay, but who has more domain knowledge when to deploy? A "security expert" that created the "security product" that operates with root privileges and full telemetry, or IT staff member that looked at said "security expert" value proposition and didn't have issue with it. Honestly, this reads as a suggestion that even more blame ought to be shifted to the customer.
      - hello_moto 2y ago
        The AV definition delivery is part of UX of the product.
mr_mitm 2y ago
Does anyone test their antivirus updates individually as a customer? I thought they happen multiple times a day, who has time for that?
- packetlost 2y ago
  Yes? Not consumers typically, but many IT departments with certain risk profiles absolutely do.
- toast0 2y ago
  Some sort of comprehensive test is unlikely. But canary / smoke tests, you can do, if the vendor provides the right tools. It's a cycle: pick the latest release, do some small cluster testing, including rollback testing, then roll out to 1%, if those machines are (mostly) still available in 5 minutes, roll out to 2%, if the 3% is (mostly) still available in 5 minutes, roll out to 4%, etc. If updates are fast and everything works, it goes quick. If there's a big problem, you'll have still have a lot of working nodes. If there's a small problem, you have a small problem. It's gotta be automated though, but with an easy way for a person to pause if something is going wrong that the automation doesn't catch. If the pace is several updates a day, that's too much for people, IMHO.
  - mr_mitm 2y ago
    Which EDR vendor provides a mechanism for testing virus signatures? This is the first time I'm hearing it and I'd like to learn more to close that knowledge gap. I always thought they are all updated ASAP, no exceptions.
    - toast0 2y ago
      Microsoft Defender isn't the most sophisticated EDR out there, but you can manage its updates with WSUS. It's been a long time since I've been subject to a corporate imposed EDR or similar, but I seem to recall them pulling updates from a company owned server for bandwidth savings, if nothing else. You can trickle update those with network controls even if the vendor doesn't provide proper tools. If corporate can't figure out how to manage software updates on their managed systems, the EDR software is the command and control malware the EDR software is supposed to prevent.
packetlost 2y ago
I really wish we would get some regulation as a result of this. I know people that almost died due to hospitals being down. It should be absolutely mandatory for users, IT departments, etc. to be able to control when and where updates happen on their infrastructure but *especially* so for critical infrastructure.
SketchySeaBeast 2y ago
Unfortunately, putting the onus on risk adverse organizations like hospitals and governments to validate the AV changes means they just won't get pushed and will be chronically exposed. That said, maybe Crowdstrike should considering validating every step of the delivery pipeline before pushing to customers.
- throw0101d 2y ago
  > Unfortunately, putting the onus on risk adverse organizations like hospitals and governments to validate the AV changes means they just won't get pushed and will be chronically exposed. I have a similar feeling. At the very least perhaps have an "A" and a "B" update channel, where "B" is x hours behind A. This way if, in an HA configuration, one side goes down there's time to deal with it while your B-side is still up.
- dmazzoni 2y ago
  Why can't they just do it more like Microsoft security patches, making them mandatory but giving admins control over when they're deployed?
  - XlA5vEKsMISoIln 2y ago
    That would be equivalent to asking "would you prefer your fleet to bluescreen now, or later" in this case.
    - jaggederest 2y ago
      Presumably you could roll out to 1% and report issues back to the vendor before the update was applied to the last 99%. So a headache but not "stop the world and reboot" levels of hassle.
    - echoangle 2y ago
      With the slight difference that you can stop applying the update once you notice the bluescreens
    - 2y ago
bawolff 2y ago
> They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effect; and 3. This was cosmically stupid and they’re going to stop doing that. Is it really all that surprising? This is basically their business model - its a fancy virus scanner that is supposed to instantly respond to threats.
98codes 2y ago
Combined with this, presented as a change they could potentially make, it's a killer: > Implement a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base, starting with a canary deployment. They weren't doing any test deployments at all before blasting the world with an update? Reckless.
- dijksterhuis 2y ago
  > our staging environment, which consists of a variety of operating systems and workloads they have a staging environment at least, but no idea what they were running in it or what testing was done there.
nathanlied 2y ago
>I predict we’ll see other vendors removing similar bonehead “features” very very quietly over the next few months. If indeed this happens, I'd hail this event as a victory overall; but industry experience tells me that most of those companies will say "it'd never happen with us, we're a lot more careful", and keep doing what they're doing.
tptacek 2y ago
They deployed changes to their software directly to customer production machines This is part of the premise of EDR software.
koolba 2y ago
> They didn’t allow their clients any opportunity to test those changes before they took effect I’d argue that anyone that agrees to this is the idiot. Sure they have blame for being the source of the problem, but any CXO that signed off on software that a third party can update whenever they’d like is also at fault. It’s not an “if” situation, it’s a “when”.
- throwaway2037 2y ago
  I felt exactly the same when I read about the outage. What kind of CTO would allow 3rd party "security" software to automatically update? That's just crazy. Of course, your own security team would do some careful (canary-like) upgrades locally... run for a bit... run some tests, then sign-off. Then upgrade in a staged manner.
  - lesuorac 2y ago
    Pretty sure many people see the point of having Falcon as a reason to not have an internal security team. Outsource everything.
    - throwaway2037 2y ago
      This is a great point that I never considered. Many companies subscribing to CrowdStrike services probably thought they took a shortcut to completely outsource they cyber-security needs. Oops, that was a mistake.
Fire-Dragon-DoL 2y ago
Now let's see if Microsoft listen and fixes Windows updates

hello_moto 2y ago

> I predict we’ll see other vendors removing similar bonehead “features” very very quietly over the next few months. Absolutely this is what will happen. I don't know much about the practice of AV definition-like feature across Cybersecurity but I would imagine there might be a possibility that no vendors do rolling update today because it involves Opt-in/Opt-out which might influence the vendor's speed to identify attack which in turns affect their "Reputation" as well. "I bought Vendor-A solution but I got hacked and have to pay Ransomware" (with a side note: because I did not consume the latest critical update of AV definition) is what Vendors worried. Now that this Global Outage happened, it will change the landscape a bit.

XlA5vEKsMISoIln 2y ago
>Now that this Global Outage happened, it will change the landscape a bit. I seriously doubt that. Questions like "why should we use CrowdStrike" will be met with "suppose they've learned their lesson".
- hello_moto 2y ago
  I'm referring to the landscape how current Cybersecurity vendors deliver "detection definition" (for lack of better phrase) to their customers. If you don't send them fast to your customer and your customer gets compromised, your reputation gets hit. If you send them fast, this BSOD happened. It's more like damn if you do, damn if you don't.
  - skydhash 2y ago
    > If you don't send them fast to your customer and your customer gets compromised, your reputation gets hit. > If you send them fast, this BSOD happened. > It's more like damn if you do, damn if you don't. What about notifications? If someone has an update policy that disable auto-updates to a critical piece of infrastructure, you can still let him know that there's a critical update is available. Now, he can do follow his own checklist in order to ensure everything goes well.
    - hello_moto 2y ago
      What if they're sleeping and won't read the notification until they wake up? Wouldn't they get compromised?
    - samcat116 2y ago
      most people will defer updates indefinitely if they are able to.
  - XlA5vEKsMISoIln 2y ago
    Okay, but who has more domain knowledge when to deploy? A "security expert" that created the "security product" that operates with root privileges and full telemetry, or IT staff member that looked at said "security expert" value proposition and didn't have issue with it. Honestly, this reads as a suggestion that even more blame ought to be shifted to the customer.
    - hello_moto 2y ago
      The AV definition delivery is part of UX of the product.

mr_mitm 2y ago

Does anyone test their antivirus updates individually as a customer? I thought they happen multiple times a day, who has time for that?

packetlost 2y ago
Yes? Not consumers typically, but many IT departments with certain risk profiles absolutely do.
toast0 2y ago
Some sort of comprehensive test is unlikely. But canary / smoke tests, you can do, if the vendor provides the right tools. It's a cycle: pick the latest release, do some small cluster testing, including rollback testing, then roll out to 1%, if those machines are (mostly) still available in 5 minutes, roll out to 2%, if the 3% is (mostly) still available in 5 minutes, roll out to 4%, etc. If updates are fast and everything works, it goes quick. If there's a big problem, you'll have still have a lot of working nodes. If there's a small problem, you have a small problem. It's gotta be automated though, but with an easy way for a person to pause if something is going wrong that the automation doesn't catch. If the pace is several updates a day, that's too much for people, IMHO.
- mr_mitm 2y ago
  Which EDR vendor provides a mechanism for testing virus signatures? This is the first time I'm hearing it and I'd like to learn more to close that knowledge gap. I always thought they are all updated ASAP, no exceptions.
  - toast0 2y ago
    Microsoft Defender isn't the most sophisticated EDR out there, but you can manage its updates with WSUS. It's been a long time since I've been subject to a corporate imposed EDR or similar, but I seem to recall them pulling updates from a company owned server for bandwidth savings, if nothing else. You can trickle update those with network controls even if the vendor doesn't provide proper tools. If corporate can't figure out how to manage software updates on their managed systems, the EDR software is the command and control malware the EDR software is supposed to prevent.

packetlost 2y ago

I really wish we would get some regulation as a result of this. I know people that almost died due to hospitals being down. It should be absolutely mandatory for users, IT departments, etc. to be able to control when and where updates happen on their infrastructure but *especially* so for critical infrastructure.

SketchySeaBeast 2y ago

Unfortunately, putting the onus on risk adverse organizations like hospitals and governments to validate the AV changes means they just won't get pushed and will be chronically exposed. That said, maybe Crowdstrike should considering validating every step of the delivery pipeline before pushing to customers.

throw0101d 2y ago
> Unfortunately, putting the onus on risk adverse organizations like hospitals and governments to validate the AV changes means they just won't get pushed and will be chronically exposed. I have a similar feeling. At the very least perhaps have an "A" and a "B" update channel, where "B" is x hours behind A. This way if, in an HA configuration, one side goes down there's time to deal with it while your B-side is still up.
dmazzoni 2y ago
Why can't they just do it more like Microsoft security patches, making them mandatory but giving admins control over when they're deployed?
- XlA5vEKsMISoIln 2y ago
  That would be equivalent to asking "would you prefer your fleet to bluescreen now, or later" in this case.
  - jaggederest 2y ago
    Presumably you could roll out to 1% and report issues back to the vendor before the update was applied to the last 99%. So a headache but not "stop the world and reboot" levels of hassle.
  - echoangle 2y ago
    With the slight difference that you can stop applying the update once you notice the bluescreens
  - 2y ago

bawolff 2y ago

> They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effect; and 3. This was cosmically stupid and they’re going to stop doing that. Is it really all that surprising? This is basically their business model - its a fancy virus scanner that is supposed to instantly respond to threats.

98codes 2y ago

Combined with this, presented as a change they could potentially make, it's a killer: > Implement a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base, starting with a canary deployment. They weren't doing any test deployments at all before blasting the world with an update? Reckless.

dijksterhuis 2y ago
> our staging environment, which consists of a variety of operating systems and workloads they have a staging environment at least, but no idea what they were running in it or what testing was done there.

nathanlied 2y ago

>I predict we’ll see other vendors removing similar bonehead “features” very very quietly over the next few months. If indeed this happens, I'd hail this event as a victory overall; but industry experience tells me that most of those companies will say "it'd never happen with us, we're a lot more careful", and keep doing what they're doing.

tptacek 2y ago

They deployed changes to their software directly to customer production machines This is part of the premise of EDR software.

koolba 2y ago

> They didn’t allow their clients any opportunity to test those changes before they took effect I’d argue that anyone that agrees to this is the idiot. Sure they have blame for being the source of the problem, but any CXO that signed off on software that a third party can update whenever they’d like is also at fault. It’s not an “if” situation, it’s a “when”.

throwaway2037 2y ago
I felt exactly the same when I read about the outage. What kind of CTO would allow 3rd party "security" software to automatically update? That's just crazy. Of course, your own security team would do some careful (canary-like) upgrades locally... run for a bit... run some tests, then sign-off. Then upgrade in a staged manner.
- lesuorac 2y ago
  Pretty sure many people see the point of having Falcon as a reason to not have an internal security team. Outsource everything.
  - throwaway2037 2y ago
    This is a great point that I never considered. Many companies subscribing to CrowdStrike services probably thought they took a shortcut to completely outsource they cyber-security needs. Oops, that was a mistake.

Fire-Dragon-DoL 2y ago

Now let's see if Microsoft listen and fixes Windows updates