4 ms·

Except further up this thread another poster mentions that CrowdStrike took down their Debian servers back in April as well. As soon as you're injecting third

Except further up this thread another poster mentions that CrowdStrike took down their Debian servers back in April as well. As soon as you're injecting third party software into your critical path with self-triggered updates you're vulnerable to the quality (or lack of) that software despite platform. Honestly your comment highlights one of the few defenses... don't sit all on one platform.

noveltyaccount 2y ago
Hopefully they won't botch the update for two operating systems at the same time. But yeah. Hope.
potatolicious 2y ago
Yeah, I see a lot of noise on social media blaming this on Microsoft/Windows... but AFAIK if you install a bad kernel driver into any major OS the result would be the same. The specific of this CrowdStrike kernel driver (which AFAIK is intended to intercept and log/deny syscalls depending on threat assessment?) means that this is badnewsbears no matter which platform you're on. Like sure, if an OS is vulnerable to kernel panics from code in userland, that's on the OS vendor, but this level of danger is intrinsic to kernel drivers!
- gjsman-1000 2y ago
  True; except here's what's baffling: CloudStrike only uses a kernel level driver on Windows. It's not necessary for Mac, it's not necessary for Linux. Why did they feel that they needed kernel level interventions on Windows devices specifically? Windows may have some blame there.
  - potatolicious 2y ago
    Yeah, I think your point is totally valid. Why does CrowdStrike need syscall access on Windows when it doesn't need it elsewhere? I do think there's an argument to be made that CrowdStrike is more invasive on Windows because Windows is intrinsically less secure. If this is true then yeah, MSFT has blame to share here.
    - AlotOfReading 2y ago
      I don't know about MacOS, but at least as recently as a couple years ago crowdstrike did ship a Linux kernel module. People were always complaining about the fact that it advertised the licensing as GPL and refused to distribute source. I imagine they've simply moved to eBPF if they're not shipping the kernel module anymore.
    - nrr 2y ago
      I haven't looked too deeply into how EDRs are implemented on Linux and macOS, but I'd wager that CrowdStrike goes the way of its own bit of code in kernel space to overcome shortcomings in how ETW telemetry works. It was never meant for security applications; ETW's purpose was to aid in software diagnostics. In particular, while it looks like macOS's Endpoint Security API[0] and Linux 4.x's inclusion of eBPF are both reasonably robust (if the literature I'm skimming is to be believed), ETW is still pretty susceptible to blinding attacks. (But what about PatchGuard? Well, as it turns out, that doesn't seem to keep someone from loading their own driver and monkey patching whatever WMI_LOGGER_CONTEXT structures they can find in order to call ControlTraceW() with ControlCode = EVENT_TRACE_CONTROL_STOP against them.) 0: https://developer.apple.com/documentation/endpointsecurity https://developer.apple.com/documentation/endpointsecurity
  - TeMPOraL 2y ago
    > Why did they feel that they needed kernel level interventions on Windows devices specifically? Maybe because everyone else in "security" and DRM does it, so they figured this is how it's done and they should do it too? My prior on competence of "cybersecurity" companies is very, very low.
    - musjleman 2y ago
      > Maybe because everyone else in "security" and DRM does it, so they figured this is how it's done and they should do it too? What DRM uses kernel drivers? And how do you plan to prevent malware from usermode?
    - bostik 2y ago
      > My prior on competence of "cybersecurity" companies is very, very low. Dmitri Alperovitch agrees with you.[0] He went on record a few months back in a podcast, and said that some of the most atrocious code he has ever seen was in security products. I am certain he was implicitly referring, at least in part, to some of the code seen inside his past company's own code base. 0: https://nationalsecurity.gmu.edu/dmitri-alperovitch/ https://nationalsecurity.gmu.edu/dmitri-alperovitch/ ["Co-founder and former CTO of Crowdstrike"]
  - PantaloonFlames 2y ago
    > CloudStrike ONLY uses a kernel level driver on Windows Crowdstrike uses a kernel level driver ONLY on Windows.
    - Cyphase 2y ago
      CrowdStrike uses a kernel level driver on Windows ONLY. Even better.. ONLY on Windows does CrowdStrike use a kernel level driver.
  - cjcampbell 2y ago
    Apple deprecated kernel extensions with 10.15 in order to improve reliability and eventually added a requirement that end users must disable SIP in order to install kexts. Security vendors moved to leverage the endpoint security framework and related APIs. On Linux, ebpf provides an alternative, and I assume, plenty of advantages over trying to maintain kernel level extensions. I haven’t researched, but my guess is that Microsoft hasn’t produced a suitable alternative for Windows security vendors.
- nextos 2y ago
  > AFAIK if you install a bad kernel driver into any major OS the result would be the same Updates should not be destructive. Linux doesn't typically overwrite previous kernels, and bootloaders let users choose a kernel during startup. Furthermore, an immutable OS makes rollback trivial for the entire system, not just the kernel (reboot, select previous configuration). I hope organizations learn from this, and we move to that model for all major OSes. Immutability is great, as we know from functional programming. Nix and Guix are pushing these ideas forward, and other OSes should borrow them.
- grumpyprole 2y ago
  It's interesting to me that lay people are asking the right questions, but many in the industry, such as the parent here, seem to just accept the status quo. If you want to be part of the solution, you have to admit there is a problem.
- akira2501 2y ago
  Non hardware "drivers" which cause a BSOD should be disabled automatically on next boot. Windows offers it's users nothing here.
II2II 2y ago
Sure, but note the sales pitch was to encourage resiliency through diversity. While that may not be helpful in cases where one vendor may push the same breaking change through to multiple platforms, it also may be helpful. I remember doing some work with a mathematics package under Solaris while in university, while my peers were using the same package under Windows. Both had the same issue, but the behaviour was different. Under Solaris, it was possible to diagnose since the application crashed with useful diagnostic information. Under Windows, it was impossible to diagnose since it took out the operating system and (because of that) it was unable to provide diagnostic information. (It's worth noting that I've seen the opposite happen as well, so this isn't meant to belittle Windows.)
- bonestamp2 2y ago
  Yes, I already heard one manager at my company today say they're getting a mac for their next computer. That's great, the whole management team shouldn't be on Windows. The engineering team is already pretty diversified between mac, windows, and linux. The next one might take down all 3 but at least we tried to diversify the risk.
__MatrixMan__ 2y ago
You can also make rollback easy. Just load the config before the one where you took the bad update. Of course that means putting the user in control of when they apply updates, but maybe that would be a good thing anyway.
lolinder 2y ago
Yep, these episodes are the banana monoculture [0] applied to IT. The solution isn't to use this vendor or avoid that vendor, it's to diversify your systems such that you can have partial operability even if one major component is down. [0] https://en.m.wikipedia.org/wiki/Gros_Michel_banana https://en.m.wikipedia.org/wiki/Gros_Michel_banana
emporas 2y ago
> don't sit all on one platform. Debian has automatic updates but they can be manual as well. That's not the case in Windows. The best practice for security critical infrastructure in which peoples lives are at stake, is to install some version of BSD stripped down to it's bare minimum. But then the company has to pay for much more expensive admins. Windows admins are much cheaper and plentiful. Also as a user of Ubuntu and Debian for more than a decade, i have a hunch that this will not happen in India [1]. [1] https://news.itsfoss.com/indian-govt-linux-windows/ https://news.itsfoss.com/indian-govt-linux-windows/
- vitorgrs 2y ago
  Windows updates can definitely be manual. And anyway, this was not a Windows update. It was a CrowdStrike update.
  - emporas 2y ago
    Oh, i thought it was tied to OS updates. So Windows is not to blame, if that's the case.
    - berkes 2y ago
      well, in another sense, Windows is certainly to blame partially. Several technical solutions have been put forward here and in other places, that would've at least limited the blast radius of a faulty update/driver/critical path. Windows didn't implement any of those. Presumably by choice and for good reasons: A tradeoff would be that software like crowdstrike is more limited in protecting you. So the Windows devs deliberately opted for this risk. Or they never considered it, which is far worse.

noveltyaccount 2y ago

Hopefully they won't botch the update for two operating systems at the same time. But yeah. Hope.

potatolicious 2y ago

Yeah, I see a lot of noise on social media blaming this on Microsoft/Windows... but AFAIK if you install a bad kernel driver into any major OS the result would be the same. The specific of this CrowdStrike kernel driver (which AFAIK is intended to intercept and log/deny syscalls depending on threat assessment?) means that this is badnewsbears no matter which platform you're on. Like sure, if an OS is vulnerable to kernel panics from code in userland, that's on the OS vendor, but this level of danger is intrinsic to kernel drivers!

gjsman-1000 2y ago
True; except here's what's baffling: CloudStrike only uses a kernel level driver on Windows. It's not necessary for Mac, it's not necessary for Linux. Why did they feel that they needed kernel level interventions on Windows devices specifically? Windows may have some blame there.
- potatolicious 2y ago
  Yeah, I think your point is totally valid. Why does CrowdStrike need syscall access on Windows when it doesn't need it elsewhere? I do think there's an argument to be made that CrowdStrike is more invasive on Windows because Windows is intrinsically less secure. If this is true then yeah, MSFT has blame to share here.
  - AlotOfReading 2y ago
    I don't know about MacOS, but at least as recently as a couple years ago crowdstrike did ship a Linux kernel module. People were always complaining about the fact that it advertised the licensing as GPL and refused to distribute source. I imagine they've simply moved to eBPF if they're not shipping the kernel module anymore.
  - nrr 2y ago
    I haven't looked too deeply into how EDRs are implemented on Linux and macOS, but I'd wager that CrowdStrike goes the way of its own bit of code in kernel space to overcome shortcomings in how ETW telemetry works. It was never meant for security applications; ETW's purpose was to aid in software diagnostics. In particular, while it looks like macOS's Endpoint Security API[0] and Linux 4.x's inclusion of eBPF are both reasonably robust (if the literature I'm skimming is to be believed), ETW is still pretty susceptible to blinding attacks. (But what about PatchGuard? Well, as it turns out, that doesn't seem to keep someone from loading their own driver and monkey patching whatever WMI_LOGGER_CONTEXT structures they can find in order to call ControlTraceW() with ControlCode = EVENT_TRACE_CONTROL_STOP against them.) 0: https://developer.apple.com/documentation/endpointsecurity https://developer.apple.com/documentation/endpointsecurity
- TeMPOraL 2y ago
  > Why did they feel that they needed kernel level interventions on Windows devices specifically? Maybe because everyone else in "security" and DRM does it, so they figured this is how it's done and they should do it too? My prior on competence of "cybersecurity" companies is very, very low.
  - musjleman 2y ago
    > Maybe because everyone else in "security" and DRM does it, so they figured this is how it's done and they should do it too? What DRM uses kernel drivers? And how do you plan to prevent malware from usermode?
  - bostik 2y ago
    > My prior on competence of "cybersecurity" companies is very, very low. Dmitri Alperovitch agrees with you.[0] He went on record a few months back in a podcast, and said that some of the most atrocious code he has ever seen was in security products. I am certain he was implicitly referring, at least in part, to some of the code seen inside his past company's own code base. 0: https://nationalsecurity.gmu.edu/dmitri-alperovitch/ https://nationalsecurity.gmu.edu/dmitri-alperovitch/ ["Co-founder and former CTO of Crowdstrike"]
- PantaloonFlames 2y ago
  > CloudStrike ONLY uses a kernel level driver on Windows Crowdstrike uses a kernel level driver ONLY on Windows.
  - Cyphase 2y ago
    CrowdStrike uses a kernel level driver on Windows ONLY. Even better.. ONLY on Windows does CrowdStrike use a kernel level driver.
- cjcampbell 2y ago
  Apple deprecated kernel extensions with 10.15 in order to improve reliability and eventually added a requirement that end users must disable SIP in order to install kexts. Security vendors moved to leverage the endpoint security framework and related APIs. On Linux, ebpf provides an alternative, and I assume, plenty of advantages over trying to maintain kernel level extensions. I haven’t researched, but my guess is that Microsoft hasn’t produced a suitable alternative for Windows security vendors.
nextos 2y ago
> AFAIK if you install a bad kernel driver into any major OS the result would be the same Updates should not be destructive. Linux doesn't typically overwrite previous kernels, and bootloaders let users choose a kernel during startup. Furthermore, an immutable OS makes rollback trivial for the entire system, not just the kernel (reboot, select previous configuration). I hope organizations learn from this, and we move to that model for all major OSes. Immutability is great, as we know from functional programming. Nix and Guix are pushing these ideas forward, and other OSes should borrow them.
grumpyprole 2y ago
It's interesting to me that lay people are asking the right questions, but many in the industry, such as the parent here, seem to just accept the status quo. If you want to be part of the solution, you have to admit there is a problem.
akira2501 2y ago
Non hardware "drivers" which cause a BSOD should be disabled automatically on next boot. Windows offers it's users nothing here.

II2II 2y ago

Sure, but note the sales pitch was to encourage resiliency through diversity. While that may not be helpful in cases where one vendor may push the same breaking change through to multiple platforms, it also may be helpful. I remember doing some work with a mathematics package under Solaris while in university, while my peers were using the same package under Windows. Both had the same issue, but the behaviour was different. Under Solaris, it was possible to diagnose since the application crashed with useful diagnostic information. Under Windows, it was impossible to diagnose since it took out the operating system and (because of that) it was unable to provide diagnostic information. (It's worth noting that I've seen the opposite happen as well, so this isn't meant to belittle Windows.)

bonestamp2 2y ago
Yes, I already heard one manager at my company today say they're getting a mac for their next computer. That's great, the whole management team shouldn't be on Windows. The engineering team is already pretty diversified between mac, windows, and linux. The next one might take down all 3 but at least we tried to diversify the risk.

__MatrixMan__ 2y ago

You can also make rollback easy. Just load the config before the one where you took the bad update. Of course that means putting the user in control of when they apply updates, but maybe that would be a good thing anyway.

lolinder 2y ago

Yep, these episodes are the banana monoculture [0] applied to IT. The solution isn't to use this vendor or avoid that vendor, it's to diversify your systems such that you can have partial operability even if one major component is down. [0] https://en.m.wikipedia.org/wiki/Gros_Michel_banana https://en.m.wikipedia.org/wiki/Gros_Michel_banana

emporas 2y ago

> don't sit all on one platform. Debian has automatic updates but they can be manual as well. That's not the case in Windows. The best practice for security critical infrastructure in which peoples lives are at stake, is to install some version of BSD stripped down to it's bare minimum. But then the company has to pay for much more expensive admins. Windows admins are much cheaper and plentiful. Also as a user of Ubuntu and Debian for more than a decade, i have a hunch that this will not happen in India [1]. [1] https://news.itsfoss.com/indian-govt-linux-windows/ https://news.itsfoss.com/indian-govt-linux-windows/

vitorgrs 2y ago
Windows updates can definitely be manual. And anyway, this was not a Windows update. It was a CrowdStrike update.
- emporas 2y ago
  Oh, i thought it was tied to OS updates. So Windows is not to blame, if that's the case.
  - berkes 2y ago
    well, in another sense, Windows is certainly to blame partially. Several technical solutions have been put forward here and in other places, that would've at least limited the blast radius of a faulty update/driver/critical path. Windows didn't implement any of those. Presumably by choice and for good reasons: A tradeoff would be that software like crowdstrike is more limited in protecting you. So the Windows devs deliberately opted for this risk. Or they never considered it, which is far worse.