5 ms·
why the fuck is our critical infrastructure running on WINDOWS. Fuck the sad state of IT. CIOs and CTOs across the board need to be fired and held accountable f
by xyst 2y ago
why the fuck is our critical infrastructure running on WINDOWS. Fuck the sad state of IT. CIOs and CTOs across the board need to be fired and held accountable for their shitty decisions in these industries.
yes CRWD is a shitty company but seems they are a "necessity" by some stupid audit/regulatory board that oversees these industries. But at the end of the day, these CIOs/CTOs are completely fucking clueless as to the exact functions this software does on a regular basis. A few minions might raise an issue but they stupidly ignore them because "rEgUlAtOrY aUdIt rEqUiReS iT!1!"
- lnrd 2y agoThe OS doesn't matter, the question should be why is critical infrastructure online and allowed to receive OTA updates from third parties.
- teej 2y agoThe OS absolutely matters
- deleted 2y ago[deleted]
- sixothree 2y agoThe alternatives aren't in a position fill the roles needed for the tasks at hand.
- rbanffy 2y agoWhat are the tasks in question?
- jeremycarter 2y agoThis is true. Linux large fleet management is still missing some features large enterprises demand. Do they need all those features, idk, but they demand them if they're switching from Windows.
- xvector 2y agoNo, what is stopping a similarly designed EDR from causing the same problem on Linux?
- lallysingh 2y agoFrom a comment above, Linux has features (ebpf) that key crowdstrike stay out of the kernel. The old "everyone else is just as bad" adage is bullshit. Some OSs are better suited than others.
- fragmede 2y agoFrom a comment elsewhere, a CS update took out Linux machines earlier this year.
- marcosdumay 2y agoThe culture around the OS matters. But this is a 3rd party software with ring-0 access to all of your computers deciding to break them. The technical features of the OS absolutely do not matter.
- rbanffy 2y agoThe question is whether other OSs would require it to have kernel mode privileges. People run complicated stuff in kernel mode for performance, because the switch to/from userspace is expensive. Guess what’s also expensive? A global outage is expensive. Much more than taking the performance hit a better, more isolated, design would avoid.
- marcosdumay 2y agoEDS run in kernel mode for access, not performance. They monkey-patch your syscalls.
- aylons 2y agoIt matters as in it makes it easy for this kind of issue to cause this much damage with little to no recourse for a fast correction. Not that Linux or whatever are all immune, but it definitely matters.
- meibo 2y agoYou should look into what a kernel driver is. You can panic a Linux kernel with 2 lines of code just as you can panic a Windows kernel, they just got lucky that this fault didn't occur in their Linux version. And to be honest, I don't think recovering from this would be that much easier for non-technical folk on a fully encrypted Linux machine, not that it's particularly hard on Windows, it's just a lot of machines to do it on.
- rbanffy 2y agoIn Linux it could be implemented as an eBPF thing while most of the app runs in userspace. And, for specialised uses, such as airline or ER systems, a cut-down specialised kernel with a minimal userland would not require the kind of protection Crowdstrike provides. I’m sure the NSA wasn’t affected by this.
- ashayh 2y agoebpf works in Windows as well.
- bg24 2y ago+1 With so much of tooling and products, it should come down to - What am I running and their current security state - Supply chain of any change that's happening - Test/Stage/Rollout any change - do not trust the vendor, as they do not know your infrastructure By allowing OTA update, they assumed that the vendor has tested all permutations.
- cpill 2y agoOf course the OS matters! Windows is a nasty ball of patches in order to maintain backward compatibility with the 80s. Linux and OSX don't have to maintain all the nasty hacks to keep this backward compatibility. Also, Crowdstrike is a security (patch) company because Windows security sucks to the point they have, by default, real-time virus protection running constantly (runs my CPU white hot for half the day, can you imagine the global impact on the environment?!). It's so bad on security that its given birth to a whole industry to fix it i.e. Crowdstrike. Every time I pass a bluescreen in a train station or advertisement I'd like "hA! you deserve that for choosing Windows".
- cyberax 2y ago> Of course the OS matters! Windows is a nasty ball of patches in order to maintain backward compatibility with the 80s. Linux and OSX don't have to maintain all the nasty hacks to keep this backward compatibility. Just don't tell that to Linus Torvalds :) Because Linux absolutely does maintain compatibility with old ABI from 90-s.
- rbanffy 2y ago> Just don't tell that to Linus Torvalds :) Because Linux absolutely does maintain compatibility with old ABI from 90-s. That’s nothing. IBM’s z/OS maintains compatibility with systems dating all the way back to the 60’s. If they want to think they are reading a stack of punch cards, the OS is happy to fool them.
- rbanffy 2y agoIBM’s z/OS maintains compatibility with the 60’s, and machines running it continued to process billions of transactions every second without taking a break. The OS matters, as well as the ecosystem and, and this is most important, the developer and operations culture around it.
- tivert 2y ago> The OS doesn't matter, the question should be why is critical infrastructure online and allowed to receive OTA updates from third parties. Not exactly. I think the question is why is critical infrastructure getting OTA updates from third parties automatically deployed directly to PROD without any testing. These updates need to go to a staging environment first, get vetted, and only then go to PROD. Another upside of that it won't go to PROD everywhere all at once, resulting in such a worldwide shitshow.
- gtvwill 2y agoLol if they can't do staging to mitigate balls ups on the high availability infrastructure side (optus in aus earlier this year pushed a router config that took down 000 emergency for a good chunk of the nation) we got bugger all hope of big companies getting it further up the stack in software.
- worik 2y ago> why is critical infrastructure getting OTA updates from third parties automatically deployed directly to PROD without any testing. I am missing some details, perhaps From what I see this was an update from Crowdstrike. They are a first party, no? Was another party involved?
- kortilla 2y agoThey are a third party software provider
- pertymcpert 2y agoI think you have the priority backwards. We shouldn’t be relying on trusting the QA process of a private company for national security systems. Our systems should have been resilient in the face of Crowdstrike incompetence.
- tivert 2y ago> I think you have the priority backwards. We shouldn’t be relying on trusting the QA process of a private company for national security systems. Our systems should have been resilient in the face of Crowdstrike incompetence. I think you misunderstood me. I wasn't talking about Crowdstrike having a staging environment, I was talking about their customers. So 911 doesn't go down immediately once Crowdstrike pushes a bad update, because the 911 center administrator stages the update, sees that it's bad, and refuses to push it to PROD. I think that would even provide some resiliency in the fact of incompetent system administrators, because even if they just hit "install" on every update, they'll tend to do it at different times of day, which will slow the rollout of bad updates and limit their impact. And the incompetent admin might not hit "install" because he read the news that day.
- lambda 2y agoWhile Linux isn't a panacea, the OS does matter as Linux provides tools for security scanners like Crowdstrike to operate entirely in userspace, with just a sandboxed eBPF program performing the filtering and blocking within the kernel. And yes, CrowdStrike supports this mode of operation, which I'll be advocating we switch over to on Monday. So yeah, for this specific issue, Linux provides a specific feature that would have prevented this issue.
- taftster 2y agoRight! Windows should NEVER blue screen. Ever. From a third-party software. Maybe Windows doesn't provide the right ABI or whatever for CS, but come on, you should never be able to kernel panic Windows. That this blue screened is 100% Microsoft's fault. It's a mess all the way around.
- CaliforniaKarl 2y agoBPF-based CrowdStrike is relatively recent, partially because, from the Enterprise Linux perspective, kernel support is relatively recent. For example, BPF-based CrowdStrike works on Enterprise Linux 9 and Debian 12. I don't know if the necessary support was in EL 8 or Debian 11.
- killerstorm 2y agoOS absolutely does matter. Windows has an enormous attack surface because Microsoft doesn't care. There's a number of minimal operating systems without all bells and whistles. The reason they aren't as popular choice is the "OS doesn't matter". If OS is minimal it doesn't need OTA updates, let alone from a third party...
- nequo 2y agoIn this case it wasn’t an update to the OS but an update to something running on the OS supplied by an unrelated vendor. But if we entertain the idea that another OS would not need CrowdStrike or anything else that required updates to begin with, I have doubts. Even your CPU needs microcode updates nowadays.
- mvdtnz 2y agoWindows is no more or less vulnerable to this class of issues than any other OS.
- evilmwnci 2y agoDebatable. macOS did away with third-party kernel extensions. On Windows, CS runs in the kernel, and the kernel can't load properly because of CS.
- fortran77 2y agoWindows isn’t Crowdstrike.
- cpill 2y agoNo, its just soooooo bad at security/stability that it gave birth to Crowdstrike. They very fact that Crowdstrike is so big and prevalent means is proof of the gapping hole in Windows security. Its given birth to a multibillion dollar industry!
- supergirl 2y agospoiler: crowdstrike is used by companies running on mac and linux as well
- foobarchu 2y agoCrowdstrike/falcon use is not by any means limited to Windows. Plenty of Linux heavy companies mandate it on all infrastructure (although I hope that changes after this incident).
- rbanffy 2y agoIt’s mandated because someone believes Linux is as bad as Windows in that regard. And, quite frankly, a well configured and properly locked down Windows would be as secure as a locked down Linux install. It’d also be a pain to use, but that’s a different question. Critical systems should run a limited set of applications precisely to reduce attack surface.
- sbuk 2y agoThe reality is the wetwear that interfaces with any OS is always going to be the weakest link. Doesn't matter what OS they run, I guarantee they will click links and download files from anywhere.
- sudosysgen 2y agoI can pretty easily make it so a user on Linux can't download executables and can't even then can't do any damage without a severe vulnerability. That is actually pretty difficult to do in a typical Windows AD deployment. There is a big difference between the two OSes. In fact, there's a couple billion Linux devices running around locked down hard enough that the most clueless users you can imagine don't get their bank details stolen.
- afavour 2y ago> why the fuck is our critical infrastructure running on WINDOWS Because it’s cheaper. I feel like many in this thread are obsessing over the choice of OS when the actual core question is why, given the insane money we spend on healthcare, are all healthcare systems shitty and underinvested? A sensible, well constructed system would have fallbacks, no matter if the OS of choice is Windows or Linux.
- rbanffy 2y agoThe difference is that lots of different companies can share the burden of implementing all that in Linux (or BSD, or anything else) while only Microsoft can implement that functionality in Windows and even their resources are limited.
- afavour 2y agoVery little healthcare functionality would ever need to be created at the OS level. The burden could be shared no matter if machines were running Windows or Linux, they’re mostly just regular applications.
- rbanffy 2y agoNot talking about the applications - those could be ported and, ideally, financed by something like the UNDP so that the same tools are available everywhere to any interested part. I'm talking about Crowdstrike's Falcon-like monitoring. It exists to intercept "suspicious" activity by userland applications and/or other kernel modules.
- freeopinion 2y agoCheaper? Well, perhaps when you require your OS to have some sort of support contract. And your support vendor charges you unhealthy sums. And then you get to see the value of the millions of dollars you've paid for support contracts that don't protect your systems at all. But those contracts do protect specific employees. When the sky falls down, the big money execs don't have a solution. But it's not their fault because the support experts they pay huge sums don't have solutions either. Somehow paying millions of dollars to support contractors that can't save you is not seen as a fireable offense. Instead it is a career-saving scapegoat. Within companies that have been bitten this time, the team that wasn't affected because they made better process decisions will not be promoted as smarter. Their voice will continue to be marginalized by the people whose decisions led to this disaster. Because, hey, look, everyone got bit right? Nobody looks around to notice the people who were not bitten and recognize their better choices. And "I told you so" is a pretty bad look right now.
- ar_lan 2y agoOdd that you choose Windows to swipe at when this was largely CRWD's problem + a mix of awful due diligence by IT departments.
- marcosdumay 2y ago> yes CRWD is a shitty company but seems they are a "necessity" by some stupid audit/regulatory board that oversees these industries. Yep, this is the problem. The part about Windows is a distraction here. That bullshit regulation is a much larger security issue than Windows. Incomparably so. If you run it over Linux, you'll get basically the same lack of security.
- Saris 2y agoDidn't CRWD cause a similar issue with Debian/RHEL a little while ago? It sounds to me that the problem lies with CRWD and not with whatever OS it's installed on.
- rbanffy 2y agoA kernel driver can, definitely, take down a Linux machine. The question is whether someone should implement something like this as a kernel module when there are better ways.
- okanat 2y agoWindows also has better ways such as filter drivers and hooks. If everybody used Linux, Crowd Strike would still opt for the kernel driver since the software they create is effectively spyware that wants access to stuff as deep as possible. If they opted for an eBPF service but put that into early boot chain, the bootloop or getting stuck could still happen. The only long time solution is to stop buying software from a company that has a track record of being pushy and having terrible software practices like rolling out updates to the entire field.
- rbanffy 2y ago> If they opted for an eBPF service but put that into early boot chain, the bootloop or getting stuck could still happen. If the in-kernel part is simple and passes data to a trusted userland application the likelyhood of a major outage like the one we saw is much reduced.
- aenis 2y agoI think the only real solution is for MSFT to stop allowing kernel level drivers, as Apple has already (sorf of, but nearly) done. Sure, lots and lots of crap runs on windows in kernelspace, but what happened today cost a sizable fraction of world's GDP. There won't be a better wake up call.
- rbanffy 2y ago
- worik 2y ago> why the fuck is our critical infrastructure running on WINDOWS. That hits the nail on the head. But it is a rhetorical question. We know why, generally, software sacks, and specifically why Windows is the worst and is the most popular Good software is developed by pointy headed needs (like us) and successful software is marketed to business executives are have serious pathologies There are exceptions (I am struggling to think of one) where a serious piece of good software has survived being mass marketed, but the constraints (basically business and science) conflict
- luxuryballs 2y agoMore specifically why is critical stuff not equipped properly to revert itself and keep working and/or fail over? This should be built-in stuff at this point, have the last working OS snapshot on its own storage chip and automatically flash it back, even if it takes a physical switch… things like this just shouldn’t happen.
- heurist 2y agoThe primary answer to your question is because it's expensive to switch.
- aenis 2y agoNah, nope. 1/ linux is as vulnerable to kernel panics induced by such software. In fact, CS had a similar snafu mid April, affecting linux kernels. Luckily, there are far fewer moronic companies running CS on linux boxes at scale. 2/ it does offer protection - if you are running total shit architecture and you need to trust your endpoints not to be compromised, something like this is sadly a must. Incidentally, google, which prides itself at running a zero-trust architecture, sent a lot of people home on Friday. Not so zero-trust after all, it seems. Lots of armchair CIOs/CTOs in the comments today.
- pertymcpert 2y agoSource on google sending home people?
- zjaffee 2y agoTons of critical infrastructure in the US is run on IBM zOS. It doesn't matter what operating system you use, what matters is updates aren't automatic and everything is as air gapped as possible.
- joesmogh 2y ago[dead]