7 ms·

In terms of analysing risk factors to minimise something like this happening again, what are the factors at play here? A Crowdstrike update being able to blue-

In terms of analysing risk factors to minimise something like this happening again, what are the factors at play here? A Crowdstrike update being able to blue-screen Windows Desktops and Servers. Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn't be able to do this kind of damage to an operating system. This makes me think that, maybe, Crowdstrike were unlucky enough to have accidentally discovered a bug that affects multiple versions of Windows (ie. it's a Windows bug, maybe more-so than it is a Crowdstrike bug). There also seems to have been a ball-dropped in regards to auto-updating all the things. Yes, gotta keep your infrastructure up to date to prevent security incidents, but is this done in test environments before it's put into production? Un-audited dependence on an increasingly long chain of third-parties. All the answers are difficult, time consuming, and therefore expensive, and are only useful in times like now. And if everyone else is down, then there's safety in the crowd. Just point at "them too", and stay the path. This isn't a profitable differentiation. But it should be! (raised fists towards the sky).

aloha2436 2y ago
> an application shouldn't be able to do this kind of damage to an operating system Antivirus software by its nature probably needs the kind of access that would let it bluescreen your computer.
- BLKNSLVR 2y ago
  Wading out my depth here, so forgive any stupidity following. And there's a certain amount of sense to that, it has to get "under" the layer that viruses can typically get to, but I still think there should be another layer at which the OS is protected from misbehaving anti-virus software (which has been known to happen).
  - eddythompson80 2y ago
    It's a kernel mode driver. There aren't layers in kernel drivers. Any kernel module/driver can crash your system if it wants to.
    - quietbritishjim 2y ago
      You're taking about how things are, the comment you're replying to is talking about how things could be. There's not a contradiction there. Originally, x86 processors had 4 levels of hardware protection, from ring 0 up to ring 3 (if I remember right). The idea was indeed that non-OS drivers could operate at the intermediate levels. But no one used them and they're effectively abandoned now. (There's "level -1" now for hypervisors and maybe other stuff but that's besides the point.) Whether those x86 were really suitable or not is not exactly important. The point is, it's possible to imagine a world where device drivers could have less than 100% permissions.
  - lnx01 2y ago
    It runs at Ring 0, there's no lower ring (besides maybe IME and the like).
    - BLKNSLVR 2y ago
      The problem I have with this is that anti-virus software has never felt like the most reliable, well-written, trustworthy software that's deserving of it's place in Ring 0. I understand I'm yelling into the storm here, because anti-virus also requires that level of system access due to the nature of what it's trying to detect. But then again, does it only need Ring 0 access for the worst of the worst? Can it run 99% of the time in Ring 1, or user space, and only instantiate it's Ring 0 privileges for regular but infrequent scans or if it detects something else may be 'off'? Default Ring 0? Earn it. This turns into a "what's your threat level" discussion.
    - yjftsjthsd-h 2y ago
      Technically, there are rings -1 through -3; hypervisor/-1 actually seems widely used and maybe could be used here. https://en.wikipedia.org/wiki/Protection_ring#Miscellaneous https://en.wikipedia.org/wiki/Protection_ring#Miscellaneous
  - DaoVeles 2y ago
    Need something like a hypervisor OS/hardware that isnt IME.
    - dist-epoch 2y ago
      Modern Windows installs already run under a hypervisor. It's called Core Isolation or Virtualization Based Security.
  - prmoustache 2y ago
    That usually makes it a port of entry for attacks. Antivirus are really malwares waiting to be exploited.
  - chrisjj 2y ago
    [delayed]
- fulafel 2y ago
  This is not the case. There are many possible AV architectures, with or without kernel drivers and/or administrator level permissions.
gamma032 2y ago
At the end of the day, if you give an application a deep set of permissions, that's on you as an administrator, not the OS. This unchecked global rollout appears to just be a violation of every good software engineering practice we know.
- BLKNSLVR 2y ago
  > This unchecked global rollout appears to just be a violation of every good software engineering practice we know. Yeah, this is what surprises me. Corporate infrastructure policy seems to have been matched to smart phone default settings.
  - freetanga 2y ago
    I have worked in Finance for 25 years, and the amount of pressure I had to stand from Auditing on "Why do we have a 20-day-window on applying most updates as we get them from suppliers? We are not best practice!" is gruelling. These people report to the Board Chairman, don't understand any real implication of their work, and believe the world is a simplistic Red - Amber - Green grid. I understand most CIOs / CTOs / CISOs in Corporate would buckle.
    - chii 2y ago
      So the silver lining from this incident would be that you can simply point to it, and tell those auditors to fuck off.
  - vbezhenar 2y ago
    I'm pretty sure Apple does gradual rollouts of upgrades, so default smartphone settings are better than that.
  - strunz 2y ago
    It's actually worse than phone updates. Ever looked at your phone and noticed it hasn't updated to the new OS despite it having been out for a few days already? This is why.
- 2y ago
zsims 2y ago
> Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn't be able to do this kind of damage to an operating system. It doesn't operate in user space, they install a kernel driver.
- Reason077 2y ago
  > "they install a kernel driver" And therein lies the problem!
  - Tech-Untangler 2y ago
    Not like they have an option. Kernel drivers are required.
    - viraptor 2y ago
      There's a grey area between "kernel drivers are required for crowdstrike" and "windows is not modular enough to expose necessary functionality to userspace". It could be solved differently given enough motivation.
      - viraptor 2y ago
        An expanded explanation with the third option of: even with existing options, it was really badly implemented - https://social.treehouse.systems/@marcan/112812791936639598 https://social.treehouse.systems/@marcan/112812791936639598
    - Reason077 2y ago
      Are they? Apple has pretty much banned kernel drivers (kexts) in macOS on Apple Silicon. When they were still used, they were a common cause of crashes and instability, not to mention potential gaping security holes. Most things that third-party kernel drivers used to do (device drivers, file systems, etc) are now done just as well, and much more safely, in userspace. I'm surprised if Microsoft isn't heading in this direction too? Presumably, Crowdstrike runs on macOS without a kernel extension?
nikau 2y ago
I feel the need to make a mustang car crowdsrike meme
- CoastalCoder 2y ago
  Could you elaborate?
  - nikau 2y ago
    Mustangs are famous for their high power and poor handling - there are lots of videos showing drivers doing burnouts, losing control, and striking the crowd they are showing off to.
  - Arrath 2y ago
    Mustangs have a reputation as being 'crowd (or streetlight) seeking' missiles. This is due to their price making them relatively more available to the enthusiasts than say Hellcats, enthusiasts who may not be experienced enough to deal with having that much power available to them in a RWD car. This confluence of power, confidence and lack of skill often comes to a head when the enthusiast goes to a car meet to show off and meet with like minded folks. At the conclusion of the meet, or during a group drive, they'll often pull a sick burnout as they pull out of the parking lot on to a street. A sick burnout they haven't practiced, and will often cause them to lose the back end sending the car into the curb, a tree, or a crowd of like minded attendees at the car meet. Therefore, the reputation. For example: https://www.youtube.com/watch?v=DPx5aBI8UTQ https://www.youtube.com/watch?v=DPx5aBI8UTQ
- ncr100 2y ago
  I think it was a Dodge charger. That pro-trump KKK guy in the gray car who drove through the crowd while the crowd was I think it was a George Floyd protest? Also heard today crowd stroke.
- ric2b 2y ago
  There's a better joke, Crowdstrike sponsors the Mercedes Formula 1 team and in 1955 Mercedes was involved in the worst motorsport accident ever, killing over 80 people watching from the stands when parts of the cars flew off and... striked the crowd...
bratwurst3000 2y ago
maybe they installed crowdstrike because they wanted updated without testing. and crowdstrike failed at testing them in their environment. sounds like they didnt test all cases and stumbled on a windows bug
AmericanChopper 2y ago
My experience working with Crowdstrike was that they were super arrogant about these risks. I was working on a ~50k enterprise rollout, and our CS guy was very belligerent about how long we were taking to do it, how much testing we wanted to do, the way that we were staggering roll outs and managing rollback plans. He didn’t think any of this was necessary, that we should roll it out in one fell swoop, have everything to auto-update all the time, and constantly yapped about how many bigger enterprises than ours completed their rollouts in just a couple of weeks. He actually threatened to fire us as a client because he claimed he didn’t want the CS brand associated with an org that wasn’t “fully protected” by CS. By far the worst vendor contact I’ve ever had. I’ve had nicer meetings with Oracle lawyers than I was having with this guy. I hope this sort of thing humbles them a little.
- BLKNSLVR 2y ago
  > constantly yapped about how many bigger enterprises than ours completed their rollouts in just a couple of weeks. Evidence is pointing towards him actually being right about this, despite likely being wrong about everything else. It'd be worth giving him a call, just to check in how he's going, and take him up on the offer to fire you as a client.
  - AmericanChopper 2y ago
    I was just a contractor there, and don’t work with them at the moment. But I’m a customer of theirs and they’re definitely having an outage right now, so I’m guessing it’s all still in place.
    - deleted 2y ago
      [deleted]
    - CoastalCoder 2y ago
      Mind rephrasing? I don't understand what you're saying.
      - 2y ago
midtake 2y ago
Maybe it's time that critical systems switch to Linux. The major public clouds are already primarily running Linux. Emergency services, booking, and traditional point-of-sale have no strong reason to run Windows. In the past 10 years, the technological capability differences between Windows and Linux have widened considerably, with Linux being the most advanced operating system in the world without question. Concerns about usability between Windows and Linux in the modern day are disingenuous at best and malicious at worst. There is no UX concern when everything runs off a webapp these days. Just use Linux. You will save money and time, and your system will be supported for many years, you won't be charged per E-Core, you won't suffer BSoDs in 2024. Red Hat is a trustworthy American company based out of Raleigh, NC, in case you have concerns of provenance. Really there's no downside. If you were building your own company you would base your tech stack on Linux and not Windows. Critical systems cannot go down; therefore they cannot run Windows. If they do, they are being mismanaged and run negligently. Management should have no issue finding Linux engineers, they are everywhere. I could make a killing right now as a consultant going from company to company and just swapping out Windows backends for Linux. And quite frankly I might just do that, literally starting right now.
- DaoVeles 2y ago
  All good points but Windows didnt win because it had the best tech or user interface. Merely the most developer support thus user numbers. Legacy momentum is an incredibly difficult thing to sway. It has taken Apple decades an potentially hundreds of billions of dollars of marketing and good will to carve out its share of the market. Linux doesn't have that despites its clear technical advantages. It is an incredibly frustrated battle akin to Sisyphus.
- attentive 2y ago
  oh, you think security won't mandate to run CS on linux. Granted it didn't down linux this time but nothing is stopping it.
  - snotrockets 2y ago
    It’s not security, it’s compliance. The two are sometimes aligned, sometimes less so.
mhh__ 2y ago
Other than a lack of redundant systems, it should be illegal to roll out updates like this to more than x% of any gov stuff at a time. Brute force way of avoiding correlational Armageddon.
JonChesterfield 2y ago
Don't root kit yourself then cry about it when it falls over. Problem solved.
btbuildem 2y ago
Crowdstrike is basically corporate malware - the failure is in large part with security dept deciders who signed off on policies that compel people to install these viruses on their work machines.
- stefantalpalaru 2y ago
  [dead]

aloha2436 2y ago

> an application shouldn't be able to do this kind of damage to an operating system Antivirus software by its nature probably needs the kind of access that would let it bluescreen your computer.

BLKNSLVR 2y ago
Wading out my depth here, so forgive any stupidity following. And there's a certain amount of sense to that, it has to get "under" the layer that viruses can typically get to, but I still think there should be another layer at which the OS is protected from misbehaving anti-virus software (which has been known to happen).
- eddythompson80 2y ago
  It's a kernel mode driver. There aren't layers in kernel drivers. Any kernel module/driver can crash your system if it wants to.
  - quietbritishjim 2y ago
    You're taking about how things are, the comment you're replying to is talking about how things could be. There's not a contradiction there. Originally, x86 processors had 4 levels of hardware protection, from ring 0 up to ring 3 (if I remember right). The idea was indeed that non-OS drivers could operate at the intermediate levels. But no one used them and they're effectively abandoned now. (There's "level -1" now for hypervisors and maybe other stuff but that's besides the point.) Whether those x86 were really suitable or not is not exactly important. The point is, it's possible to imagine a world where device drivers could have less than 100% permissions.
- lnx01 2y ago
  It runs at Ring 0, there's no lower ring (besides maybe IME and the like).
  - BLKNSLVR 2y ago
    The problem I have with this is that anti-virus software has never felt like the most reliable, well-written, trustworthy software that's deserving of it's place in Ring 0. I understand I'm yelling into the storm here, because anti-virus also requires that level of system access due to the nature of what it's trying to detect. But then again, does it only need Ring 0 access for the worst of the worst? Can it run 99% of the time in Ring 1, or user space, and only instantiate it's Ring 0 privileges for regular but infrequent scans or if it detects something else may be 'off'? Default Ring 0? Earn it. This turns into a "what's your threat level" discussion.
  - yjftsjthsd-h 2y ago
    Technically, there are rings -1 through -3; hypervisor/-1 actually seems widely used and maybe could be used here. https://en.wikipedia.org/wiki/Protection_ring#Miscellaneous https://en.wikipedia.org/wiki/Protection_ring#Miscellaneous
- DaoVeles 2y ago
  Need something like a hypervisor OS/hardware that isnt IME.
  - dist-epoch 2y ago
    Modern Windows installs already run under a hypervisor. It's called Core Isolation or Virtualization Based Security.
- prmoustache 2y ago
  That usually makes it a port of entry for attacks. Antivirus are really malwares waiting to be exploited.
- chrisjj 2y ago
  [delayed]
fulafel 2y ago
This is not the case. There are many possible AV architectures, with or without kernel drivers and/or administrator level permissions.

gamma032 2y ago

At the end of the day, if you give an application a deep set of permissions, that's on you as an administrator, not the OS. This unchecked global rollout appears to just be a violation of every good software engineering practice we know.

BLKNSLVR 2y ago
> This unchecked global rollout appears to just be a violation of every good software engineering practice we know. Yeah, this is what surprises me. Corporate infrastructure policy seems to have been matched to smart phone default settings.
- freetanga 2y ago
  I have worked in Finance for 25 years, and the amount of pressure I had to stand from Auditing on "Why do we have a 20-day-window on applying most updates as we get them from suppliers? We are not best practice!" is gruelling. These people report to the Board Chairman, don't understand any real implication of their work, and believe the world is a simplistic Red - Amber - Green grid. I understand most CIOs / CTOs / CISOs in Corporate would buckle.
  - chii 2y ago
    So the silver lining from this incident would be that you can simply point to it, and tell those auditors to fuck off.
- vbezhenar 2y ago
  I'm pretty sure Apple does gradual rollouts of upgrades, so default smartphone settings are better than that.
- strunz 2y ago
  It's actually worse than phone updates. Ever looked at your phone and noticed it hasn't updated to the new OS despite it having been out for a few days already? This is why.
2y ago

zsims 2y ago

> Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn't be able to do this kind of damage to an operating system. It doesn't operate in user space, they install a kernel driver.

Reason077 2y ago
> "they install a kernel driver" And therein lies the problem!
- Tech-Untangler 2y ago
  Not like they have an option. Kernel drivers are required.
  - viraptor 2y ago
    There's a grey area between "kernel drivers are required for crowdstrike" and "windows is not modular enough to expose necessary functionality to userspace". It could be solved differently given enough motivation.
    - viraptor 2y ago
      An expanded explanation with the third option of: even with existing options, it was really badly implemented - https://social.treehouse.systems/@marcan/112812791936639598 https://social.treehouse.systems/@marcan/112812791936639598
  - Reason077 2y ago
    Are they? Apple has pretty much banned kernel drivers (kexts) in macOS on Apple Silicon. When they were still used, they were a common cause of crashes and instability, not to mention potential gaping security holes. Most things that third-party kernel drivers used to do (device drivers, file systems, etc) are now done just as well, and much more safely, in userspace. I'm surprised if Microsoft isn't heading in this direction too? Presumably, Crowdstrike runs on macOS without a kernel extension?

nikau 2y ago

I feel the need to make a mustang car crowdsrike meme

CoastalCoder 2y ago
Could you elaborate?
- nikau 2y ago
  Mustangs are famous for their high power and poor handling - there are lots of videos showing drivers doing burnouts, losing control, and striking the crowd they are showing off to.
- Arrath 2y ago
  Mustangs have a reputation as being 'crowd (or streetlight) seeking' missiles. This is due to their price making them relatively more available to the enthusiasts than say Hellcats, enthusiasts who may not be experienced enough to deal with having that much power available to them in a RWD car. This confluence of power, confidence and lack of skill often comes to a head when the enthusiast goes to a car meet to show off and meet with like minded folks. At the conclusion of the meet, or during a group drive, they'll often pull a sick burnout as they pull out of the parking lot on to a street. A sick burnout they haven't practiced, and will often cause them to lose the back end sending the car into the curb, a tree, or a crowd of like minded attendees at the car meet. Therefore, the reputation. For example: https://www.youtube.com/watch?v=DPx5aBI8UTQ https://www.youtube.com/watch?v=DPx5aBI8UTQ
ncr100 2y ago
I think it was a Dodge charger. That pro-trump KKK guy in the gray car who drove through the crowd while the crowd was I think it was a George Floyd protest? Also heard today crowd stroke.
ric2b 2y ago
There's a better joke, Crowdstrike sponsors the Mercedes Formula 1 team and in 1955 Mercedes was involved in the worst motorsport accident ever, killing over 80 people watching from the stands when parts of the cars flew off and... striked the crowd...

bratwurst3000 2y ago

maybe they installed crowdstrike because they wanted updated without testing. and crowdstrike failed at testing them in their environment. sounds like they didnt test all cases and stumbled on a windows bug

AmericanChopper 2y ago

My experience working with Crowdstrike was that they were super arrogant about these risks. I was working on a ~50k enterprise rollout, and our CS guy was very belligerent about how long we were taking to do it, how much testing we wanted to do, the way that we were staggering roll outs and managing rollback plans. He didn’t think any of this was necessary, that we should roll it out in one fell swoop, have everything to auto-update all the time, and constantly yapped about how many bigger enterprises than ours completed their rollouts in just a couple of weeks. He actually threatened to fire us as a client because he claimed he didn’t want the CS brand associated with an org that wasn’t “fully protected” by CS. By far the worst vendor contact I’ve ever had. I’ve had nicer meetings with Oracle lawyers than I was having with this guy. I hope this sort of thing humbles them a little.

BLKNSLVR 2y ago
> constantly yapped about how many bigger enterprises than ours completed their rollouts in just a couple of weeks. Evidence is pointing towards him actually being right about this, despite likely being wrong about everything else. It'd be worth giving him a call, just to check in how he's going, and take him up on the offer to fire you as a client.
- AmericanChopper 2y ago
  I was just a contractor there, and don’t work with them at the moment. But I’m a customer of theirs and they’re definitely having an outage right now, so I’m guessing it’s all still in place.
  - deleted 2y ago
    [deleted]
  - CoastalCoder 2y ago
    Mind rephrasing? I don't understand what you're saying.
    - 2y ago

midtake 2y ago

Maybe it's time that critical systems switch to Linux. The major public clouds are already primarily running Linux. Emergency services, booking, and traditional point-of-sale have no strong reason to run Windows. In the past 10 years, the technological capability differences between Windows and Linux have widened considerably, with Linux being the most advanced operating system in the world without question. Concerns about usability between Windows and Linux in the modern day are disingenuous at best and malicious at worst. There is no UX concern when everything runs off a webapp these days. Just use Linux. You will save money and time, and your system will be supported for many years, you won't be charged per E-Core, you won't suffer BSoDs in 2024. Red Hat is a trustworthy American company based out of Raleigh, NC, in case you have concerns of provenance. Really there's no downside. If you were building your own company you would base your tech stack on Linux and not Windows. Critical systems cannot go down; therefore they cannot run Windows. If they do, they are being mismanaged and run negligently. Management should have no issue finding Linux engineers, they are everywhere. I could make a killing right now as a consultant going from company to company and just swapping out Windows backends for Linux. And quite frankly I might just do that, literally starting right now.

DaoVeles 2y ago
All good points but Windows didnt win because it had the best tech or user interface. Merely the most developer support thus user numbers. Legacy momentum is an incredibly difficult thing to sway. It has taken Apple decades an potentially hundreds of billions of dollars of marketing and good will to carve out its share of the market. Linux doesn't have that despites its clear technical advantages. It is an incredibly frustrated battle akin to Sisyphus.
attentive 2y ago
oh, you think security won't mandate to run CS on linux. Granted it didn't down linux this time but nothing is stopping it.
- snotrockets 2y ago
  It’s not security, it’s compliance. The two are sometimes aligned, sometimes less so.

mhh__ 2y ago

Other than a lack of redundant systems, it should be illegal to roll out updates like this to more than x% of any gov stuff at a time. Brute force way of avoiding correlational Armageddon.

JonChesterfield 2y ago

Don't root kit yourself then cry about it when it falls over. Problem solved.

btbuildem 2y ago

Crowdstrike is basically corporate malware - the failure is in large part with security dept deciders who signed off on policies that compel people to install these viruses on their work machines.

stefantalpalaru 2y ago
[dead]