5 ms·

Windows is not the issue here. If all of the businesses used Linux, a similar software product, deployed as widely as Crowdstrike, with auto-update, could resul

Windows is not the issue here. If all of the businesses used Linux, a similar software product, deployed as widely as Crowdstrike, with auto-update, could result in the same issue. Same goes for the OS; if let's say majority of businesses used RHEL with auto updates, RedHat could in theory push an update, that would result bring down all machines.

antihero 2y ago
I would posit that RedHat have a slightly longer and more proven track record than Crowdstrike, and more transparent process with how they release updates. No entity is infallible but letting one closed source opaque corporation have the keys to break everything isn’t resilient.
- oblio 2y ago
  His example actually had 2 parts. One RH bricking the OS, the other one with a commercial vendor creating software with separate auto update. You've only addressed the RH OS angle.
  - antihero 2y ago
    Yes but the problem here was bricking the OS
phkahler 2y ago
>> Windows is not the issue here. Yes it is. Windows was created for the "Personal Computer" with zero thought initially put in to security. It has been fighting that heritage for 30 years. The reason Crowdstrike exists at all is due to shortcomings (real or perceived) in Windows security. Unix (and hence Linux and MacOS) was designed as a multi-user system from the start, so access controls and permissions were there from the start. It may have been a flawed security model and has been updated over time, but at least it started some notion of security. These ideas had already expanded to networks before Microsoft ever heard the word Netscape.
- crowdstriker 2y ago
  [flagged]
  - stalfosknight 2y ago
    The Mac predates Windows.
  - tremon 2y ago
    The Linux kernel predates Windows 95 (as do the first distributions). GNU predates even the first version of Windows.
- qwytw 2y ago
  > was designed as a multi-user system from the start, so access controls and permissions were there from the start. Right and Windows NT wasn't? Obviously it supported all of those things from the very beginning (possibly even in a superior way to Unix in some cases considering it's a significantly more modern OS)... The fact that MS developed another OS called Windows (3.1 -> 95 -> 98) prior to that which was to some extent binary compatible with NT seems somewhat tangential. Otherwise the same arguments would surely apply to MacOS as well? > These ideas had already expanded to networks before Microsoft ever heard the word Netscape. Does not seem like a good thing on its own to me. Just solidifies the fact the it's an inherently less modern OS than Windows(NT) (which still might have various design flaws obviously, that might be worth discussing, it just has nothing to do whatsoever with what you're claiming here...)
- tapoxi 2y ago
  We have Crowdstrike on our Linux fleet. It is not merely a malware scanner but is capable of identifying and stopping zero-day attacks that attempt local privilege escalation. It can, for example, detect and block attempts to exploit CVE-2024-3094 - the xz backdoor. Perhaps we need to move to an even more restrictive design like Fuschia, or standardize on an open source eBPF based utility that's built, tested, and shipped with a distribution's specific kernel, but Windows is not the issue here.
- sabas123 2y ago
crowdstriker 2y ago
[flagged]
LrnByTeach 2y ago
yes, staggered software update is the way to go. there was reply in this thread why Crowdstrike did not do it -- don't want extra cost of Engineering for that having 1/3 of Airlines computers Windows, RHEL, Ubuntu .. all unlikely to hit same problems at same time.
- bentcorner 2y ago
  But you're more likely to encounter problems. That's likely a good thing as it improves your DR documentation and processes but could be a harder sell to the suits.
- hello_moto 2y ago
  The update here is relevant to catch 0day exploit. Without the update, your system is "naked" for the duration.
doubled112 2y ago
https://access.redhat.com/solutions/7068083 https://access.redhat.com/solutions/7068083 Just last month there were issues between RHEL's kernel update and Crowdstrike.
- dima55 2y ago
  I'd like to read more about it, but that link is... paywalled I think? It's not even clear.
  - doubled112 2y ago
    I keep a RedHat developer account active for their documentation. I didn't notice. I did find these related forum/reddit threads: https://forums.rockylinux.org/t/crowdstrike-freezing-rockylinux-after-9-4-upgrade/14041 https://forums.rockylinux.org/t/crowdstrike-freezing-rockyli... https://www.reddit.com/r/crowdstrike/comments/1cluxzz/crowdstrike_kernel_panic_rhel_94/ https://www.reddit.com/r/crowdstrike/comments/1cluxzz/crowds... Good to see Rocky keeping their promise of bug for bug compatibility.
fulafel 2y ago
That's assuming in this alternate universe we'd also be using kernel antivirus software to counter malware. It's far from obvious.
aflag 2y ago
But then it'd be putting all eggs in the Linux pc basket, wouldn't it? I think they point was that more heterogeneity would make this not be a problem. If all your potatoes are the same potato it only takes one bad blight epidemic to kill off all farmed potatoes in a country. If there's more heterogeneity things like that doesn't happen.
kjellsbells 2y ago
Agree. The monoculture simply accelerates the infection because there are no sizable natural barriers to stop it. Windows and even Intel must take some blame, because in this day and age of vPro on the board and rollbacks built into the OS it's incredible that there is no "last known good" procedure to boot into the most recent successfully booted environment (didnt NT have this 30 years ago?), or remotely recover the system. I pity the IT staff that are going to have to talk Bob in Accounting through bitlocker and some sys file, times 1000s. IT get some blame, because this notion that an update from a third party can reach past the logical gatekeeping function that IT provides, directly into their estate, and change things, is unconscionable. Why dont the PCs update from a local mirror that IT has that has been through canary testing? Do we trust vendors that much now? Poor Crowdstrike. This might be the end for them.
Natsu 2y ago
More secure OSes would consider an application being able to take down the entire OS as a security issue and would make that impossible.
- 13of40 2y ago
  When notepad hits an unhandled exception and the OS decides it's in an unpredictable state, the OS shuts down notepad's process. When there's an unhandled exception in kernel mode, the OS shuts down the entire computer. That's a BSOD in Windows or a kernel panic in Linux. The problem isn't that CrowdStrike is a normal user mode application that is taking down Windows because Windows just lets that happen, it's that CrowdStrike has faulty code that runs in kernel mode. This isn't unique to Windows or Linux. The main reason they need to run in kernel mode is you can't do behavior monitoring hooks in user mode without making your security tool open to detection and evasion. For example, if your security tool wants to detect whenever a process calls ShellExecute, you can inject a DLL into the process that hooks the ShellExecute API, but malware can just check for that in its own process and either work around it or refuse to run. That means the hook needs to be in kernel mode, or the OS needs to provide instrumentation that allows third party code to monitor calls like that without running in kernel mode. IMO, Windows (and probably any OS you're likely to encounter in the wild) could do better providing that kind of instrumentation. Windows and Office have made progress in the last several years with things like enabling monitoring of PowerShell and VBA script block execution, but it's not enough that solutions like CrowdStrike can do their thing without going low level. Beyond that, there's also going to be a huge latency between when a security researcher finds a new technique for creating processes, doing persistence, or whatever and when the engineering team for an OS can update their instrumentation to support detecting it, so there's always going to be some need for a presence in kernel mode if you want up to date protection.
  - Natsu 2y ago
    I mean, to me that's just a convincing argument against using kernel-mode spywa-, err, endpoint protection, with OTA updates that give you no way to stage or test them yourself cannot be secure.
lanstin 2y ago
The difference being that RHEL has a QA process which crowd strike apparently does not. The quality practices for open source involved companies is apparently much higher than for large closed source "security" firms. I guess getting whined at because obscure things break in beta or rc releases has a good effect for the people using LTS.
Ajedi32 2y ago
Maybe this is pie-in-the-sky thinking, but if all the businesses used some sort of desktop variant of Android, the Crowdstrike app (to the extent that such a thing would even be necessary in the first place) would be sandboxed and wouldn't have the necessary permissions to bring down the whole operating system.

antihero 2y ago

I would posit that RedHat have a slightly longer and more proven track record than Crowdstrike, and more transparent process with how they release updates. No entity is infallible but letting one closed source opaque corporation have the keys to break everything isn’t resilient.

oblio 2y ago
His example actually had 2 parts. One RH bricking the OS, the other one with a commercial vendor creating software with separate auto update. You've only addressed the RH OS angle.
- antihero 2y ago
  Yes but the problem here was bricking the OS

phkahler 2y ago

>> Windows is not the issue here. Yes it is. Windows was created for the "Personal Computer" with zero thought initially put in to security. It has been fighting that heritage for 30 years. The reason Crowdstrike exists at all is due to shortcomings (real or perceived) in Windows security. Unix (and hence Linux and MacOS) was designed as a multi-user system from the start, so access controls and permissions were there from the start. It may have been a flawed security model and has been updated over time, but at least it started some notion of security. These ideas had already expanded to networks before Microsoft ever heard the word Netscape.

crowdstriker 2y ago
[flagged]
- stalfosknight 2y ago
  The Mac predates Windows.
- tremon 2y ago
  The Linux kernel predates Windows 95 (as do the first distributions). GNU predates even the first version of Windows.
qwytw 2y ago
> was designed as a multi-user system from the start, so access controls and permissions were there from the start. Right and Windows NT wasn't? Obviously it supported all of those things from the very beginning (possibly even in a superior way to Unix in some cases considering it's a significantly more modern OS)... The fact that MS developed another OS called Windows (3.1 -> 95 -> 98) prior to that which was to some extent binary compatible with NT seems somewhat tangential. Otherwise the same arguments would surely apply to MacOS as well? > These ideas had already expanded to networks before Microsoft ever heard the word Netscape. Does not seem like a good thing on its own to me. Just solidifies the fact the it's an inherently less modern OS than Windows(NT) (which still might have various design flaws obviously, that might be worth discussing, it just has nothing to do whatsoever with what you're claiming here...)
tapoxi 2y ago
We have Crowdstrike on our Linux fleet. It is not merely a malware scanner but is capable of identifying and stopping zero-day attacks that attempt local privilege escalation. It can, for example, detect and block attempts to exploit CVE-2024-3094 - the xz backdoor. Perhaps we need to move to an even more restrictive design like Fuschia, or standardize on an open source eBPF based utility that's built, tested, and shipped with a distribution's specific kernel, but Windows is not the issue here.
sabas123 2y ago

crowdstriker 2y ago

[flagged]

LrnByTeach 2y ago

yes, staggered software update is the way to go. there was reply in this thread why Crowdstrike did not do it -- don't want extra cost of Engineering for that having 1/3 of Airlines computers Windows, RHEL, Ubuntu .. all unlikely to hit same problems at same time.

bentcorner 2y ago
But you're more likely to encounter problems. That's likely a good thing as it improves your DR documentation and processes but could be a harder sell to the suits.
hello_moto 2y ago
The update here is relevant to catch 0day exploit. Without the update, your system is "naked" for the duration.

doubled112 2y ago

https://access.redhat.com/solutions/7068083 https://access.redhat.com/solutions/7068083 Just last month there were issues between RHEL's kernel update and Crowdstrike.

dima55 2y ago
I'd like to read more about it, but that link is... paywalled I think? It's not even clear.
- doubled112 2y ago
  I keep a RedHat developer account active for their documentation. I didn't notice. I did find these related forum/reddit threads: https://forums.rockylinux.org/t/crowdstrike-freezing-rockylinux-after-9-4-upgrade/14041 https://forums.rockylinux.org/t/crowdstrike-freezing-rockyli... https://www.reddit.com/r/crowdstrike/comments/1cluxzz/crowdstrike_kernel_panic_rhel_94/ https://www.reddit.com/r/crowdstrike/comments/1cluxzz/crowds... Good to see Rocky keeping their promise of bug for bug compatibility.

fulafel 2y ago

That's assuming in this alternate universe we'd also be using kernel antivirus software to counter malware. It's far from obvious.

aflag 2y ago

But then it'd be putting all eggs in the Linux pc basket, wouldn't it? I think they point was that more heterogeneity would make this not be a problem. If all your potatoes are the same potato it only takes one bad blight epidemic to kill off all farmed potatoes in a country. If there's more heterogeneity things like that doesn't happen.

kjellsbells 2y ago

Agree. The monoculture simply accelerates the infection because there are no sizable natural barriers to stop it. Windows and even Intel must take some blame, because in this day and age of vPro on the board and rollbacks built into the OS it's incredible that there is no "last known good" procedure to boot into the most recent successfully booted environment (didnt NT have this 30 years ago?), or remotely recover the system. I pity the IT staff that are going to have to talk Bob in Accounting through bitlocker and some sys file, times 1000s. IT get some blame, because this notion that an update from a third party can reach past the logical gatekeeping function that IT provides, directly into their estate, and change things, is unconscionable. Why dont the PCs update from a local mirror that IT has that has been through canary testing? Do we trust vendors that much now? Poor Crowdstrike. This might be the end for them.

Natsu 2y ago

More secure OSes would consider an application being able to take down the entire OS as a security issue and would make that impossible.

13of40 2y ago
When notepad hits an unhandled exception and the OS decides it's in an unpredictable state, the OS shuts down notepad's process. When there's an unhandled exception in kernel mode, the OS shuts down the entire computer. That's a BSOD in Windows or a kernel panic in Linux. The problem isn't that CrowdStrike is a normal user mode application that is taking down Windows because Windows just lets that happen, it's that CrowdStrike has faulty code that runs in kernel mode. This isn't unique to Windows or Linux. The main reason they need to run in kernel mode is you can't do behavior monitoring hooks in user mode without making your security tool open to detection and evasion. For example, if your security tool wants to detect whenever a process calls ShellExecute, you can inject a DLL into the process that hooks the ShellExecute API, but malware can just check for that in its own process and either work around it or refuse to run. That means the hook needs to be in kernel mode, or the OS needs to provide instrumentation that allows third party code to monitor calls like that without running in kernel mode. IMO, Windows (and probably any OS you're likely to encounter in the wild) could do better providing that kind of instrumentation. Windows and Office have made progress in the last several years with things like enabling monitoring of PowerShell and VBA script block execution, but it's not enough that solutions like CrowdStrike can do their thing without going low level. Beyond that, there's also going to be a huge latency between when a security researcher finds a new technique for creating processes, doing persistence, or whatever and when the engineering team for an OS can update their instrumentation to support detecting it, so there's always going to be some need for a presence in kernel mode if you want up to date protection.
- Natsu 2y ago
  I mean, to me that's just a convincing argument against using kernel-mode spywa-, err, endpoint protection, with OTA updates that give you no way to stage or test them yourself cannot be secure.

lanstin 2y ago

The difference being that RHEL has a QA process which crowd strike apparently does not. The quality practices for open source involved companies is apparently much higher than for large closed source "security" firms. I guess getting whined at because obscure things break in beta or rc releases has a good effect for the people using LTS.

Ajedi32 2y ago

Maybe this is pie-in-the-sky thinking, but if all the businesses used some sort of desktop variant of Android, the Crowdstrike app (to the extent that such a thing would even be necessary in the first place) would be sandboxed and wouldn't have the necessary permissions to bring down the whole operating system.