7 ms·

Some Canonical guy I think many years ago mentioned this as their sales strategy a few year ago after a particularly nasty Windows outage: We don't ask custome

Some Canonical guy I think many years ago mentioned this as their sales strategy a few year ago after a particularly nasty Windows outage: We don't ask customers to switch all systems from Windows to Ubuntu, but to consider moving maybe a third to Ubuntu so they won't sit completely helpless next time Windows fail spectacularly. While I see more and more Ubuntu systems, and recently have even spotted Landscape in the wild I don't think they were as successful as they hoped with that strategy. That said, maybe there is a silver lining on todays clouds both WRT Ubuntu and Linux in general, and also WRT IT departments stopping to reconsider some security best practices.

51Cards 2y ago
Except further up this thread another poster mentions that CrowdStrike took down their Debian servers back in April as well. As soon as you're injecting third party software into your critical path with self-triggered updates you're vulnerable to the quality (or lack of) that software despite platform. Honestly your comment highlights one of the few defenses... don't sit all on one platform.
- noveltyaccount 2y ago
  Hopefully they won't botch the update for two operating systems at the same time. But yeah. Hope.
- potatolicious 2y ago
  Yeah, I see a lot of noise on social media blaming this on Microsoft/Windows... but AFAIK if you install a bad kernel driver into any major OS the result would be the same. The specific of this CrowdStrike kernel driver (which AFAIK is intended to intercept and log/deny syscalls depending on threat assessment?) means that this is badnewsbears no matter which platform you're on. Like sure, if an OS is vulnerable to kernel panics from code in userland, that's on the OS vendor, but this level of danger is intrinsic to kernel drivers!
  - gjsman-1000 2y ago
    True; except here's what's baffling: CloudStrike only uses a kernel level driver on Windows. It's not necessary for Mac, it's not necessary for Linux. Why did they feel that they needed kernel level interventions on Windows devices specifically? Windows may have some blame there.
    - potatolicious 2y ago
      Yeah, I think your point is totally valid. Why does CrowdStrike need syscall access on Windows when it doesn't need it elsewhere? I do think there's an argument to be made that CrowdStrike is more invasive on Windows because Windows is intrinsically less secure. If this is true then yeah, MSFT has blame to share here.
      - AlotOfReading 2y ago
        I don't know about MacOS, but at least as recently as a couple years ago crowdstrike did ship a Linux kernel module. People were always complaining about the fact that it advertised the licensing as GPL and refused to distribute source. I imagine they've simply moved to eBPF if they're not shipping the kernel module anymore.
      - nrr 2y ago
        I haven't looked too deeply into how EDRs are implemented on Linux and macOS, but I'd wager that CrowdStrike goes the way of its own bit of code in kernel space to overcome shortcomings in how ETW telemetry works. It was never meant for security applications; ETW's purpose was to aid in software diagnostics. In particular, while it looks like macOS's Endpoint Security API[0] and Linux 4.x's inclusion of eBPF are both reasonably robust (if the literature I'm skimming is to be believed), ETW is still pretty susceptible to blinding attacks. (But what about PatchGuard? Well, as it turns out, that doesn't seem to keep someone from loading their own driver and monkey patching whatever WMI_LOGGER_CONTEXT structures they can find in order to call ControlTraceW() with ControlCode = EVENT_TRACE_CONTROL_STOP against them.) 0: https://developer.apple.com/documentation/endpointsecurity https://developer.apple.com/documentation/endpointsecurity
- II2II 2y ago
  Sure, but note the sales pitch was to encourage resiliency through diversity. While that may not be helpful in cases where one vendor may push the same breaking change through to multiple platforms, it also may be helpful. I remember doing some work with a mathematics package under Solaris while in university, while my peers were using the same package under Windows. Both had the same issue, but the behaviour was different. Under Solaris, it was possible to diagnose since the application crashed with useful diagnostic information. Under Windows, it was impossible to diagnose since it took out the operating system and (because of that) it was unable to provide diagnostic information. (It's worth noting that I've seen the opposite happen as well, so this isn't meant to belittle Windows.)
  - bonestamp2 2y ago
    Yes, I already heard one manager at my company today say they're getting a mac for their next computer. That's great, the whole management team shouldn't be on Windows. The engineering team is already pretty diversified between mac, windows, and linux. The next one might take down all 3 but at least we tried to diversify the risk.
- __MatrixMan__ 2y ago
  You can also make rollback easy. Just load the config before the one where you took the bad update. Of course that means putting the user in control of when they apply updates, but maybe that would be a good thing anyway.
- lolinder 2y ago
  Yep, these episodes are the banana monoculture [0] applied to IT. The solution isn't to use this vendor or avoid that vendor, it's to diversify your systems such that you can have partial operability even if one major component is down. [0] https://en.m.wikipedia.org/wiki/Gros_Michel_banana https://en.m.wikipedia.org/wiki/Gros_Michel_banana
- emporas 2y ago
  > don't sit all on one platform. Debian has automatic updates but they can be manual as well. That's not the case in Windows. The best practice for security critical infrastructure in which peoples lives are at stake, is to install some version of BSD stripped down to it's bare minimum. But then the company has to pay for much more expensive admins. Windows admins are much cheaper and plentiful. Also as a user of Ubuntu and Debian for more than a decade, i have a hunch that this will not happen in India [1]. [1] https://news.itsfoss.com/indian-govt-linux-windows/ https://news.itsfoss.com/indian-govt-linux-windows/
  - vitorgrs 2y ago
    Windows updates can definitely be manual. And anyway, this was not a Windows update. It was a CrowdStrike update.
    - emporas 2y ago
      Oh, i thought it was tied to OS updates. So Windows is not to blame, if that's the case.
      - berkes 2y ago
        well, in another sense, Windows is certainly to blame partially. Several technical solutions have been put forward here and in other places, that would've at least limited the blast radius of a faulty update/driver/critical path. Windows didn't implement any of those. Presumably by choice and for good reasons: A tradeoff would be that software like crowdstrike is more limited in protecting you. So the Windows devs deliberately opted for this risk. Or they never considered it, which is far worse.
baggy_trough 2y ago
Still haven't forgiven Ubuntu for pushing a bad kernel of their own that caused a boot loop if you used containers...
- j33zusjuice 2y ago
  I’ll never forgive them for the spyware they defaulted to on in their desktop stuff. It wasn’t the worst thing in the world, but they’re also the only major distro to ever do it, so Ubuntu (and Canonical as a whole) can get fucked, imo.
  - genewitch 2y ago
    [flagged]
    - ricardobayes 2y ago
      That's a long grudge to hold over a feature that was reconsidered and removed.
      - pbhjpbhj 2y ago
        Maybe, but Canonical didn't learn and are back to pushing advertising and forcing unwanted changes.
      - slow_typist 2y ago
        To say it rather politely, the mindset exposed by introducing this feature is unlikely to go away.
        eitland 2y ago
        As shown by Mozilla.
      - ikrenji 2y ago
        canonical has some of the most ridiculous IT job postings i’ve come across. just sounds like a bananas software shop. didn’t give me much confidence in whatever they cooking up in there
aiauthoritydev 2y ago
Seems like a reasonable strategy. Not just Ubuntu but some redundancy in some systems.
samcat116 2y ago
Sure but if that Canonical sales person was successful in that, I'd almost guarantee that after they switched the first third they'd be in there arguing to switch out the rest.
- eitland 2y ago
  Absolutely. I'm just saying what they said their strategy was, not judging their sales people.
make3 2y ago
things are so interdependent that in this scenario you might now just end up crashing the system if either Windows or Ubuntu are down instead of just the one of them you chose
secondcoming 2y ago
Ubuntu has unattended-upgrades enabled by default
- sgarland 2y ago
  Yes, but by default the only repo enabled for it is $(cat /etc/os-release)-security.
  - Hardwired8976 2y ago
    But CrowdStrike is security as well?
    - kchr 2y ago
      Yes, but it's not included in the upstream Ubuntu security repository. In fact, it's not available via any repository AFAIK. It updates itself via fetching new versions from CrowdStrike backend according to your update policy for the host in question. However, as we've learned the past days, that policy does not apply to the "update channel" files...
mrintegrity 2y ago
Many years ago an Ubuntu tech sales guy demoed their (openstack?) Self hosted cloud offering, his laptop was running windows..
grumpyprole 2y ago
Linux and open source also have the potential to be far more modular than Windows is. At the moment we have airport display boards running a full windows stack including anti-virus/spyware/audit etc, just to display a table ... madness
pbhjpbhj 2y ago
I'm a Kubuntu user that, seemingly due to Canonical's decision to ship untested software regularly, has been repeatedly hit by problems with snaps. What were initially basic, obvious, and widespread issues with major software. Yes, distribute your eggs, but check the handles on the baskets being sold to you by the guy pointing out bad handles.
- eitland 2y ago
  FWIW, while some people like Kubuntu, I have had much better results with KDE Neon. Stable Ubuntu core under the surface, and everything desktop related delivered by the KDE team.
  - pbhjpbhj 2y ago
    Thanks for the tip, I'm looking to jump ship to MX-Linux, just procrastinating the move right now.
totallywrong 2y ago
Canonical in particular are no better, they do the exact same thing with that aberration called snap. They have brought entire clusters down before with automatic updates.

51Cards 2y ago

Except further up this thread another poster mentions that CrowdStrike took down their Debian servers back in April as well. As soon as you're injecting third party software into your critical path with self-triggered updates you're vulnerable to the quality (or lack of) that software despite platform. Honestly your comment highlights one of the few defenses... don't sit all on one platform.

noveltyaccount 2y ago
Hopefully they won't botch the update for two operating systems at the same time. But yeah. Hope.
potatolicious 2y ago
Yeah, I see a lot of noise on social media blaming this on Microsoft/Windows... but AFAIK if you install a bad kernel driver into any major OS the result would be the same. The specific of this CrowdStrike kernel driver (which AFAIK is intended to intercept and log/deny syscalls depending on threat assessment?) means that this is badnewsbears no matter which platform you're on. Like sure, if an OS is vulnerable to kernel panics from code in userland, that's on the OS vendor, but this level of danger is intrinsic to kernel drivers!
- gjsman-1000 2y ago
  True; except here's what's baffling: CloudStrike only uses a kernel level driver on Windows. It's not necessary for Mac, it's not necessary for Linux. Why did they feel that they needed kernel level interventions on Windows devices specifically? Windows may have some blame there.
  - potatolicious 2y ago
    Yeah, I think your point is totally valid. Why does CrowdStrike need syscall access on Windows when it doesn't need it elsewhere? I do think there's an argument to be made that CrowdStrike is more invasive on Windows because Windows is intrinsically less secure. If this is true then yeah, MSFT has blame to share here.
    - AlotOfReading 2y ago
      I don't know about MacOS, but at least as recently as a couple years ago crowdstrike did ship a Linux kernel module. People were always complaining about the fact that it advertised the licensing as GPL and refused to distribute source. I imagine they've simply moved to eBPF if they're not shipping the kernel module anymore.
    - nrr 2y ago
      I haven't looked too deeply into how EDRs are implemented on Linux and macOS, but I'd wager that CrowdStrike goes the way of its own bit of code in kernel space to overcome shortcomings in how ETW telemetry works. It was never meant for security applications; ETW's purpose was to aid in software diagnostics. In particular, while it looks like macOS's Endpoint Security API[0] and Linux 4.x's inclusion of eBPF are both reasonably robust (if the literature I'm skimming is to be believed), ETW is still pretty susceptible to blinding attacks. (But what about PatchGuard? Well, as it turns out, that doesn't seem to keep someone from loading their own driver and monkey patching whatever WMI_LOGGER_CONTEXT structures they can find in order to call ControlTraceW() with ControlCode = EVENT_TRACE_CONTROL_STOP against them.) 0: https://developer.apple.com/documentation/endpointsecurity https://developer.apple.com/documentation/endpointsecurity
II2II 2y ago
Sure, but note the sales pitch was to encourage resiliency through diversity. While that may not be helpful in cases where one vendor may push the same breaking change through to multiple platforms, it also may be helpful. I remember doing some work with a mathematics package under Solaris while in university, while my peers were using the same package under Windows. Both had the same issue, but the behaviour was different. Under Solaris, it was possible to diagnose since the application crashed with useful diagnostic information. Under Windows, it was impossible to diagnose since it took out the operating system and (because of that) it was unable to provide diagnostic information. (It's worth noting that I've seen the opposite happen as well, so this isn't meant to belittle Windows.)
- bonestamp2 2y ago
  Yes, I already heard one manager at my company today say they're getting a mac for their next computer. That's great, the whole management team shouldn't be on Windows. The engineering team is already pretty diversified between mac, windows, and linux. The next one might take down all 3 but at least we tried to diversify the risk.
__MatrixMan__ 2y ago
You can also make rollback easy. Just load the config before the one where you took the bad update. Of course that means putting the user in control of when they apply updates, but maybe that would be a good thing anyway.
lolinder 2y ago
Yep, these episodes are the banana monoculture [0] applied to IT. The solution isn't to use this vendor or avoid that vendor, it's to diversify your systems such that you can have partial operability even if one major component is down. [0] https://en.m.wikipedia.org/wiki/Gros_Michel_banana https://en.m.wikipedia.org/wiki/Gros_Michel_banana
emporas 2y ago
> don't sit all on one platform. Debian has automatic updates but they can be manual as well. That's not the case in Windows. The best practice for security critical infrastructure in which peoples lives are at stake, is to install some version of BSD stripped down to it's bare minimum. But then the company has to pay for much more expensive admins. Windows admins are much cheaper and plentiful. Also as a user of Ubuntu and Debian for more than a decade, i have a hunch that this will not happen in India [1]. [1] https://news.itsfoss.com/indian-govt-linux-windows/ https://news.itsfoss.com/indian-govt-linux-windows/
- vitorgrs 2y ago
  Windows updates can definitely be manual. And anyway, this was not a Windows update. It was a CrowdStrike update.
  - emporas 2y ago
    Oh, i thought it was tied to OS updates. So Windows is not to blame, if that's the case.
    - berkes 2y ago
      well, in another sense, Windows is certainly to blame partially. Several technical solutions have been put forward here and in other places, that would've at least limited the blast radius of a faulty update/driver/critical path. Windows didn't implement any of those. Presumably by choice and for good reasons: A tradeoff would be that software like crowdstrike is more limited in protecting you. So the Windows devs deliberately opted for this risk. Or they never considered it, which is far worse.

baggy_trough 2y ago

Still haven't forgiven Ubuntu for pushing a bad kernel of their own that caused a boot loop if you used containers...

j33zusjuice 2y ago
I’ll never forgive them for the spyware they defaulted to on in their desktop stuff. It wasn’t the worst thing in the world, but they’re also the only major distro to ever do it, so Ubuntu (and Canonical as a whole) can get fucked, imo.
- genewitch 2y ago
  [flagged]
  - ricardobayes 2y ago
    That's a long grudge to hold over a feature that was reconsidered and removed.
    - pbhjpbhj 2y ago
      Maybe, but Canonical didn't learn and are back to pushing advertising and forcing unwanted changes.
    - slow_typist 2y ago
      To say it rather politely, the mindset exposed by introducing this feature is unlikely to go away.
      - eitland 2y ago
        As shown by Mozilla.
    - ikrenji 2y ago
      canonical has some of the most ridiculous IT job postings i’ve come across. just sounds like a bananas software shop. didn’t give me much confidence in whatever they cooking up in there

aiauthoritydev 2y ago

Seems like a reasonable strategy. Not just Ubuntu but some redundancy in some systems.

samcat116 2y ago

Sure but if that Canonical sales person was successful in that, I'd almost guarantee that after they switched the first third they'd be in there arguing to switch out the rest.

eitland 2y ago
Absolutely. I'm just saying what they said their strategy was, not judging their sales people.

make3 2y ago

things are so interdependent that in this scenario you might now just end up crashing the system if either Windows or Ubuntu are down instead of just the one of them you chose

secondcoming 2y ago

Ubuntu has unattended-upgrades enabled by default

sgarland 2y ago
Yes, but by default the only repo enabled for it is $(cat /etc/os-release)-security.
- Hardwired8976 2y ago
  But CrowdStrike is security as well?
  - kchr 2y ago
    Yes, but it's not included in the upstream Ubuntu security repository. In fact, it's not available via any repository AFAIK. It updates itself via fetching new versions from CrowdStrike backend according to your update policy for the host in question. However, as we've learned the past days, that policy does not apply to the "update channel" files...

mrintegrity 2y ago

Many years ago an Ubuntu tech sales guy demoed their (openstack?) Self hosted cloud offering, his laptop was running windows..

grumpyprole 2y ago

Linux and open source also have the potential to be far more modular than Windows is. At the moment we have airport display boards running a full windows stack including anti-virus/spyware/audit etc, just to display a table ... madness

pbhjpbhj 2y ago

I'm a Kubuntu user that, seemingly due to Canonical's decision to ship untested software regularly, has been repeatedly hit by problems with snaps. What were initially basic, obvious, and widespread issues with major software. Yes, distribute your eggs, but check the handles on the baskets being sold to you by the guy pointing out bad handles.

eitland 2y ago
FWIW, while some people like Kubuntu, I have had much better results with KDE Neon. Stable Ubuntu core under the surface, and everything desktop related delivered by the KDE team.
- pbhjpbhj 2y ago
  Thanks for the tip, I'm looking to jump ship to MX-Linux, just procrastinating the move right now.

totallywrong 2y ago

Canonical in particular are no better, they do the exact same thing with that aberration called snap. They have brought entire clusters down before with automatic updates.