4 ms·

"Diversity" (but not in the sense of marginalized people) If more of the critical machines were running different OS's, the damage would be contained. When we

"Diversity" (but not in the sense of marginalized people) If more of the critical machines were running different OS's, the damage would be contained. When we talk about the dangers of "monoculture" it's usually about plants. The same danger applies to computing infrastructure.

mr90210 2y ago
On top of that, I am still struggling to understand how the people in charge of running orgs that run highly critical systems were OK with the idea that a 3rd party software provider could push at anytime patches to the software they provide. Sorry for being harsh with my following statement, but I believe that the companies affected by Crowdstrike share some responsibility on what happened yesterday.
- chrisjj 2y ago
  The are OK with "push at anytime patches to the software" because that's a big part of what they are paying for. Rapid response to threats.
  - pixl97 2y ago
    >Ping reply from 127.0.0.1 The threat is inside the building!
- lambdaone 2y ago
  You're making the mistake of assuming that the people running those companies care about anything other than their job security, and buying in solutions is the best way to have a ready-made scapegoat when things go wrong. The mantra "no-one ever got sacked for buying IBM" still holds, you can just substitute "Oracle", or "Microsoft", or now - apparently - "Crowdstrike".
- meiraleal 2y ago
  They share the whole of the responsibility of it. "my antivirus was updating" is not an acceptable excuse for a service to be down.
- notabee 2y ago
  It's not harsh. The tide went out and it turns out a lot of people were swimming naked.
- ck45 2y ago
  I think I agree with you. On the other hand, I can also imagine that if autoupdates weren't the case, then 90% of installations would be a terribly outdated and probably vulnerable version. It's hard to imagine a common sense middle ground.
- cynicalsecurity 2y ago
  Surprisingly, the mantra "if it works, don't touch it" doesn't really work so great.
- cdchn 2y ago
  One could make the argument that automatically patched software is, in aggregate, more secure/less problematic than chronically under-patched software that requires manual, human attention.
  - AlbertCory 2y ago
    One could, but in the old days when vulnerabilities happened, they didn't hit everyone at once. And if it hit your system, the vendor's first response would be "are you on the latest update? that's been fixed." (Unless the latest update IS the problem. In that case, being lazy was a good defense.)
- RedShift1 2y ago
  As I understand it, customers do have control, but in this instance CrowdStrike overrode the settings of the customers.
- slt2021 2y ago
  - pushing patches is objectively a good idea, rapid response to threats and all. - Whats bad is instant global 0->1 rollout, instead of more gradual, blue/green/canary however you call it. With gradual rollout policy this whole thing could have been caught at their first couple guinea pig customers, and not the whole world
  - patrick451 2y ago
    You don't understand the word objective. It is beyond arrogant to think that controlling when a customer's day gets ruined is your prerogative. Let them make that decision.
- dehrmann 2y ago
  They chose a major vendor and it checks off a compliance requirement.
jnwatson 2y ago
We're already there. The fact that we didn't see civilization collapse is evidence that there is a ton of infrastructure not running Windows and Crowdstrike.
- AlbertCory 2y ago
  For a long time after Burroughs was almost ancient history, banks still ran Burroughs machines. They've probably thrown in the sponge by now. I'm sure IBM mainframes are still running critical stuff, too.
- treyd 2y ago
  This wasn't nearly as bad as it could have been. What if the crash wasn't just a crash but resulted in data corruption? And what if it took longer to stop the rollout and deploy a fixed version? How long would it have taken to recover from this kind of incident? If affected machines didn't fix themselves after several reboots but needed to be actively reimaged?
chrisjj 2y ago
> If more of the critical machines were running different OS's, the damage would be contained. Not if they were running the same CrowdStrike.
- AlbertCory 2y ago
  given it's a kernel module (AFAIK), how could that be if it were different OS's?
  - chrisjj 2y ago
    Regardless, much code would be shared. Likely including the offending null pointer access of this case.
    - AlbertCory 2y ago
      > Regaress "regardless" ? So you don't actually know, is what you're saying.
      - chrisjj 2y ago
        Thanks. Typo corrected.
cynicalsecurity 2y ago
Computers are not people. No need to be afraid to discriminate. Windows is shit. Mac is more or less. Linux is best of all.
- forrestthewoods 2y ago
  You do realize that CrowdStrike also runs on Linux and that there have been a variety of instances of bad CrowdStrike updates breaking Linux machines, right? https://access.redhat.com/solutions/7068083 https://access.redhat.com/solutions/7068083
dehrmann 2y ago
Not necessarily. CrowdStrike isn't even the #1 player in this space, but this still happened because of network effects. The number of platforms you'd need for this much safety is impractically high.
- AlbertCory 2y ago
  I'm not saying you're wrong, but: "Network effects"? You mean like, "I'd be fine, but I depend on a service from a Windows machine, so I'm still screwed" ? > The number of platforms you'd need for this much safety is impractically high I don't see why this becomes an impossible problem. If all the essential services are not provided by a single software infrastructure, then we have the required diversity, right?
  - dehrmann 2y ago
    > "Network effects"? You mean like... In the case of airports, losing ATC at just a few major US airports would effectively paralyze the network. Or yes, the case you mentioned where you depend on four SaaS offerings, and odds are one of them will go down.
    - AlbertCory 2y ago
      Like I said: I'm not arguing with you. It's ironic that the original DARPA justification for packet-switching was, if a nuclear war takes down some nodes, the packets will still get through somehow.

mr90210 2y ago

On top of that, I am still struggling to understand how the people in charge of running orgs that run highly critical systems were OK with the idea that a 3rd party software provider could push at anytime patches to the software they provide. Sorry for being harsh with my following statement, but I believe that the companies affected by Crowdstrike share some responsibility on what happened yesterday.

chrisjj 2y ago
The are OK with "push at anytime patches to the software" because that's a big part of what they are paying for. Rapid response to threats.
- pixl97 2y ago
  >Ping reply from 127.0.0.1 The threat is inside the building!
lambdaone 2y ago
You're making the mistake of assuming that the people running those companies care about anything other than their job security, and buying in solutions is the best way to have a ready-made scapegoat when things go wrong. The mantra "no-one ever got sacked for buying IBM" still holds, you can just substitute "Oracle", or "Microsoft", or now - apparently - "Crowdstrike".
meiraleal 2y ago
They share the whole of the responsibility of it. "my antivirus was updating" is not an acceptable excuse for a service to be down.
notabee 2y ago
It's not harsh. The tide went out and it turns out a lot of people were swimming naked.
ck45 2y ago
I think I agree with you. On the other hand, I can also imagine that if autoupdates weren't the case, then 90% of installations would be a terribly outdated and probably vulnerable version. It's hard to imagine a common sense middle ground.
cynicalsecurity 2y ago
Surprisingly, the mantra "if it works, don't touch it" doesn't really work so great.
cdchn 2y ago
One could make the argument that automatically patched software is, in aggregate, more secure/less problematic than chronically under-patched software that requires manual, human attention.
- AlbertCory 2y ago
  One could, but in the old days when vulnerabilities happened, they didn't hit everyone at once. And if it hit your system, the vendor's first response would be "are you on the latest update? that's been fixed." (Unless the latest update IS the problem. In that case, being lazy was a good defense.)
RedShift1 2y ago
As I understand it, customers do have control, but in this instance CrowdStrike overrode the settings of the customers.
slt2021 2y ago
- pushing patches is objectively a good idea, rapid response to threats and all. - Whats bad is instant global 0->1 rollout, instead of more gradual, blue/green/canary however you call it. With gradual rollout policy this whole thing could have been caught at their first couple guinea pig customers, and not the whole world
- patrick451 2y ago
  You don't understand the word objective. It is beyond arrogant to think that controlling when a customer's day gets ruined is your prerogative. Let them make that decision.
dehrmann 2y ago
They chose a major vendor and it checks off a compliance requirement.

jnwatson 2y ago

We're already there. The fact that we didn't see civilization collapse is evidence that there is a ton of infrastructure not running Windows and Crowdstrike.

AlbertCory 2y ago
For a long time after Burroughs was almost ancient history, banks still ran Burroughs machines. They've probably thrown in the sponge by now. I'm sure IBM mainframes are still running critical stuff, too.
treyd 2y ago
This wasn't nearly as bad as it could have been. What if the crash wasn't just a crash but resulted in data corruption? And what if it took longer to stop the rollout and deploy a fixed version? How long would it have taken to recover from this kind of incident? If affected machines didn't fix themselves after several reboots but needed to be actively reimaged?

chrisjj 2y ago

> If more of the critical machines were running different OS's, the damage would be contained. Not if they were running the same CrowdStrike.

AlbertCory 2y ago
given it's a kernel module (AFAIK), how could that be if it were different OS's?
- chrisjj 2y ago
  Regardless, much code would be shared. Likely including the offending null pointer access of this case.
  - AlbertCory 2y ago
    > Regaress "regardless" ? So you don't actually know, is what you're saying.
    - chrisjj 2y ago
      Thanks. Typo corrected.

cynicalsecurity 2y ago

Computers are not people. No need to be afraid to discriminate. Windows is shit. Mac is more or less. Linux is best of all.

forrestthewoods 2y ago
You do realize that CrowdStrike also runs on Linux and that there have been a variety of instances of bad CrowdStrike updates breaking Linux machines, right? https://access.redhat.com/solutions/7068083 https://access.redhat.com/solutions/7068083

dehrmann 2y ago

Not necessarily. CrowdStrike isn't even the #1 player in this space, but this still happened because of network effects. The number of platforms you'd need for this much safety is impractically high.

AlbertCory 2y ago
I'm not saying you're wrong, but: "Network effects"? You mean like, "I'd be fine, but I depend on a service from a Windows machine, so I'm still screwed" ? > The number of platforms you'd need for this much safety is impractically high I don't see why this becomes an impossible problem. If all the essential services are not provided by a single software infrastructure, then we have the required diversity, right?
- dehrmann 2y ago
  > "Network effects"? You mean like... In the case of airports, losing ATC at just a few major US airports would effectively paralyze the network. Or yes, the case you mentioned where you depend on four SaaS offerings, and odds are one of them will go down.
  - AlbertCory 2y ago
    Like I said: I'm not arguing with you. It's ironic that the original DARPA justification for packet-switching was, if a nuclear war takes down some nodes, the packets will still get through somehow.