4 ms·

Maybe the IT departments at the affected orgs take solace in the fact that so many other orgs had issues that the heat is off - but in my opinion this was still

by jmuguy 2y ago

Maybe the IT departments at the affected orgs take solace in the fact that so many other orgs had issues that the heat is off - but in my opinion this was still a failure of IT itself. There's no reason that update should have been pushed automatically to the entire fleet. If Crowdstrike's software doesn't give you a way to rollout updates on a portion of your network before the entire fleet, it shouldn't be used.

candiddevmike 2y ago
The update bypassed the controls orgs had in place to defer/schedule updates, AFAIK.
- jmuguy 2y ago
  I've had trouble nailing down if thats the case from searching around online. And if thats true - thats absolutely on Crowdstrike. And that behavior should disqualify it from being used on critical systems. I imagine this incident will cause a lot of teams to consider just what can happen automatically on their systems.
  - tantalor 2y ago
    [flagged]
    - lucianbr 2y ago
      How did 911 services go down then? Whatever system caused that, should be by definition critical, imho.
    - Sohcahtoa82 2y ago
      You're living in a different reality. I can't fathom how anybody could legitimately make that claim. Even if you're defining "critical system" as "critical to humans" and not "critical to the business", then sure, you can say "Airlines aren't critical" and for most passengers, yeah, you're probably right. Most industries aren't critical, so businesses being ground to a halt doesn't matter for the consumers. But 911 systems were affected, and those are certainly critical to humans. If 911 doesn't work, ambulances and fire trucks can't be dispatched, and people die. EDIT: Computers attached to hospital beds, including trauma surgery rooms, were affected. I'm really curious what you think defines a critical system.
      - SkyBelow 2y ago
        One interesting thing I saw is, per a snippet that claimed to be part of Crowd Strike's ToS, it shouldn't have been installed on any of those machines where human life depended upon it (along with no nuclear facilities and a few other exceptions). Is there going to be any fallout from people installing it on systems the software wasn't designed for? Did Crowd Strike perhaps know it was being installed on these systems but ignored it since they were getting paid and it wasn't them violating the agreement?
        manvillej 2y ago
        if a user does something the manufacturer told them specifically not to do, I have a hard time blaming the manufacturer for it. Within an approved use? absolutely, blame the manufacturer. but if you shoot yourself in the foot, don't blame the bowyer just because they sold the bow to you.
        terribleperson 2y ago
        Supposedly, CrowdStrike sales would pressure companies to have the software installed on every system in their network.
      - tantalor 2y ago
        The 911 system itself is critical sure. I never said it wasn't. When the computer systems supporting 911 went down due to crowdstrike, those functions were replaced with available backups, that were planned for situations like this, e.g. using analog phones and taking notes by hand (just like they used to do it). If the system survives (albeit with diminished capacity) loss of a component, then that component is not critical for the system. That's basically the definition of "critical". Source: https://www.usatoday.com/story/news/nation/2024/07/19/crowdstrike-outages-what-happened/74474725007/ https://www.usatoday.com/story/news/nation/2024/07/19/crowds...
    - peddling-brink 2y ago
      According to the crowdstrike tos, sure..
  - wisemang 2y ago
    It’s definitely the case. See Crowdstrike’s preliminary post incident review here: https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio... The nature of “content updates” vs a full product update. Though you may be right, perhaps they provide controls for those updates, I’ve never used their software. But doesn’t sound like it.
  - CydeWeys 2y ago
    It's on CrowdStrike, but it's also on IT for even allowing installation of critical software like this that has a bypass at all. Updates shouldn't even be allowed to bypass IT's safe rollout procedures, at least not without IT signing off on it anyway.
- kelnos 2y ago
  If that's the case, that doesn't change GP's point: if Crowdstrike can bypass your org's controls on rolling out updates to its software, it shouldn't be used.
- qaq 2y ago
  Generally none gates content updates as they happen multiple times a day
- b4ckup 2y ago
  Didn't day say in their incident report that they have a batched rollout strategy for software updates but this was a config update and the update path for configs does not have such a mechanism in place.
  - bonestamp2 2y ago
    Ya, so hopefully it's obvious to them that every rollout needs some kind of batching. I get that all devices within one org might need to have the same config, but in that case batch it out to different orgs over 2-3 days. Maybe the more critical infrastructure and health care orgs are at the end of that rollout plan so they are at lower risk. It's not ideal if one sandwich shop in Idaho can't run their reports that day, but that's far better than shutting down the hospital next door. CrowdStrike could even compensate those one system shops that are on the front line when something goes down. Again, better to pay a sandwich shop a few thousand dollars for their lost day of sales than get sued by the people in the hospital who couldn't get their meds, x-rays, etc in time.
lucianbr 2y ago
Management decides to use Crowdstrike, not IT, and IT has no way to rollout updates in controlled fashion. So not really a failure of IT, at least not for this reason.
- jmuguy 2y ago
  My comment assumes that the IT department (including its executive) gets to make these sort decisions - why wouldn't they?
  - Retric 2y ago
    Major purchases tend to be pushed up the ladder. It’s not uncommon for a CEO or non technical director etc to decide what IT systems to use.
    - fullspectrumdev 2y ago
      In my experience the decisions on any non trivial IT system rollout are made by entirely unqualified, non technical execs who are usually swayed by marketing such as clownstrikes Super Bowl advertisement. Technical people will make a recommendation, knowing it’s going to be ignored and that the decisions already been made.
  - lucianbr 2y ago
    Sure, if there's a security exec who made the decision, it may be their fault. I was thinking more of the rank and file, but that's just my bias.
    - hello_moto 2y ago
      When a ransomeware attack is happening, organizations will engage to cybersecurity vendors and will start PoC with a bunch of them and list the pros and cons of each vendors before they negotiate and ended up selecting the winner.
  - mingus88 2y ago

by jmuguy 2y ago

candiddevmike 2y ago

The update bypassed the controls orgs had in place to defer/schedule updates, AFAIK.

jmuguy 2y ago
I've had trouble nailing down if thats the case from searching around online. And if thats true - thats absolutely on Crowdstrike. And that behavior should disqualify it from being used on critical systems. I imagine this incident will cause a lot of teams to consider just what can happen automatically on their systems.
- tantalor 2y ago
  [flagged]
  - lucianbr 2y ago
    How did 911 services go down then? Whatever system caused that, should be by definition critical, imho.
  - Sohcahtoa82 2y ago
    You're living in a different reality. I can't fathom how anybody could legitimately make that claim. Even if you're defining "critical system" as "critical to humans" and not "critical to the business", then sure, you can say "Airlines aren't critical" and for most passengers, yeah, you're probably right. Most industries aren't critical, so businesses being ground to a halt doesn't matter for the consumers. But 911 systems were affected, and those are certainly critical to humans. If 911 doesn't work, ambulances and fire trucks can't be dispatched, and people die. EDIT: Computers attached to hospital beds, including trauma surgery rooms, were affected. I'm really curious what you think defines a critical system.
    - SkyBelow 2y ago
      One interesting thing I saw is, per a snippet that claimed to be part of Crowd Strike's ToS, it shouldn't have been installed on any of those machines where human life depended upon it (along with no nuclear facilities and a few other exceptions). Is there going to be any fallout from people installing it on systems the software wasn't designed for? Did Crowd Strike perhaps know it was being installed on these systems but ignored it since they were getting paid and it wasn't them violating the agreement?
      - manvillej 2y ago
        if a user does something the manufacturer told them specifically not to do, I have a hard time blaming the manufacturer for it. Within an approved use? absolutely, blame the manufacturer. but if you shoot yourself in the foot, don't blame the bowyer just because they sold the bow to you.
        terribleperson 2y ago
        Supposedly, CrowdStrike sales would pressure companies to have the software installed on every system in their network.
    - tantalor 2y ago
      The 911 system itself is critical sure. I never said it wasn't. When the computer systems supporting 911 went down due to crowdstrike, those functions were replaced with available backups, that were planned for situations like this, e.g. using analog phones and taking notes by hand (just like they used to do it). If the system survives (albeit with diminished capacity) loss of a component, then that component is not critical for the system. That's basically the definition of "critical". Source: https://www.usatoday.com/story/news/nation/2024/07/19/crowdstrike-outages-what-happened/74474725007/ https://www.usatoday.com/story/news/nation/2024/07/19/crowds...
  - peddling-brink 2y ago
    According to the crowdstrike tos, sure..
- wisemang 2y ago
  It’s definitely the case. See Crowdstrike’s preliminary post incident review here: https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/ https://www.crowdstrike.com/falcon-content-update-remediatio... The nature of “content updates” vs a full product update. Though you may be right, perhaps they provide controls for those updates, I’ve never used their software. But doesn’t sound like it.
- CydeWeys 2y ago
  It's on CrowdStrike, but it's also on IT for even allowing installation of critical software like this that has a bypass at all. Updates shouldn't even be allowed to bypass IT's safe rollout procedures, at least not without IT signing off on it anyway.
kelnos 2y ago
If that's the case, that doesn't change GP's point: if Crowdstrike can bypass your org's controls on rolling out updates to its software, it shouldn't be used.
qaq 2y ago
Generally none gates content updates as they happen multiple times a day
b4ckup 2y ago
Didn't day say in their incident report that they have a batched rollout strategy for software updates but this was a config update and the update path for configs does not have such a mechanism in place.
- bonestamp2 2y ago
  Ya, so hopefully it's obvious to them that every rollout needs some kind of batching. I get that all devices within one org might need to have the same config, but in that case batch it out to different orgs over 2-3 days. Maybe the more critical infrastructure and health care orgs are at the end of that rollout plan so they are at lower risk. It's not ideal if one sandwich shop in Idaho can't run their reports that day, but that's far better than shutting down the hospital next door. CrowdStrike could even compensate those one system shops that are on the front line when something goes down. Again, better to pay a sandwich shop a few thousand dollars for their lost day of sales than get sued by the people in the hospital who couldn't get their meds, x-rays, etc in time.

lucianbr 2y ago

Management decides to use Crowdstrike, not IT, and IT has no way to rollout updates in controlled fashion. So not really a failure of IT, at least not for this reason.

jmuguy 2y ago
My comment assumes that the IT department (including its executive) gets to make these sort decisions - why wouldn't they?
- Retric 2y ago
  Major purchases tend to be pushed up the ladder. It’s not uncommon for a CEO or non technical director etc to decide what IT systems to use.
  - fullspectrumdev 2y ago
    In my experience the decisions on any non trivial IT system rollout are made by entirely unqualified, non technical execs who are usually swayed by marketing such as clownstrikes Super Bowl advertisement. Technical people will make a recommendation, knowing it’s going to be ignored and that the decisions already been made.
- lucianbr 2y ago
  Sure, if there's a security exec who made the decision, it may be their fault. I was thinking more of the rank and file, but that's just my bias.
  - hello_moto 2y ago
    When a ransomeware attack is happening, organizations will engage to cybersecurity vendors and will start PoC with a bunch of them and list the pros and cons of each vendors before they negotiate and ended up selecting the winner.
- mingus88 2y ago