13 ms·

CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes

158 points by kenjackson 2y ago

rramadass 2y ago
I had read reports of this earlier which is what makes me speculate that the Windows Crowdstrike issue is more than "just a update error" i.e. there might be some nefarious hand behind this. Given that they were already aware of the Linux issue it boggles my mind that they did not take extra precautions when it came to Windows updates. We will have to wait and see for further trustworthy info. Btw - The article mentions Dave Plummer's analysis of the issue which might be easier for people to understand and worth a watch. - https://www.youtube.com/watch?v=wAzEJxOo1ts https://www.youtube.com/watch?v=wAzEJxOo1ts
- bawolff 2y ago
  Some part of a company already aware of an issue but different part still ships is a pretty common tale and seems much more likely than some nefarious conspiracy theory. (And that is even assuming this is the same issue, which seems questionable) After all, who exactly would benefit from such a nefarious scheme to crash windows computers? Certainly not Crowdstrike.
  - tgv 2y ago
    Wrt trying to get something out of the crash: https://www.theregister.com/2024/07/19/cyber_criminals_quickly_exploit_crowdstrike/?td=rt-9cp https://www.theregister.com/2024/07/19/cyber_criminals_quick.... The conspiracy-minded could be suspicious of the quick response to this outage. "They must have known in advance!" Or they could suspect an overeager account manager at Crowdstrike, who wanted to show how important the product is. But indeed, this really sounds like it was an internal error.
  - logicchains 2y ago
    >After all, who exactly would benefit from such a nefarious scheme to crash windows computers? Russia or China would certainly benefit from the ability to do this at a time of their choosing, and it's possible they could have an agent inside Crowdstrike, especially given China's history of industrial espionage.
    - bawolff 2y ago
      You think russia or china would benefit more from this then having a stealthy rootkit on basically every important computer in the world? I think russia or china are probably the least likely purpotrators possible. Their incentives strongly disalign with this.
  - rramadass 2y ago
    > who exactly would benefit from such a nefarious scheme State Actors, given the current Geo-Political tensions. You have to take a all-in-all broader view. I remember a while ago Kaspersky was accused of data-siphoning/spying from computers it was installed on and other nefarious activities. See New Government Ban on Kaspersky Would Prevent Company from Updating Malware Signatures in U.S. - https://www.zetter-zeroday.com/new-government-ban-on-kaspersky-would-prevent-malware-signature-updates/ https://www.zetter-zeroday.com/new-government-ban-on-kaspers... As for your opening statement "Some part of a company already aware of an issue but different part still ships is a pretty common tale" is not applicable here since this code runs in kernel mode (in both OSes) and thus would be subject to far far greater scrutiny and testing than an ordinary app. As Dave Plummer points out in his analysis Microsoft Kernel Drivers are signed and certified after an exhaustive testing process. Even if Crowdstrike wrote their drivers as an interpreter and the data update files were actually programs in some p-code, Microsoft would have definitely known of it and its inherent vulnerabilities. I would bet money that Microsoft knows all about preventing threats/vulnerabilities than any other company simply because of their long experience and large userbase and thus would not have allowed Crowdstrike such a free hand.
    - bawolff 2y ago
      > State Actors, given the current Geo-Political tensions. I disagree. If state actors had this type of capability they would use it to spy on big companies. The espionage potential is huge. They wouldn't waste it on causing a minor inconvinence. > is not applicable here since this code runs in kernel mode (in both OSes) and thus would be subject to far far greater scrutiny and testing than an ordinary app Lol. What next? Politicians always tell the truth? Everyone gets a free unicorn? This is just obviously not how the world works. There is a long history of anti-virus software being kind of crap.
      - rramadass 2y ago
        > If state actors had this type of capability they would use it to spy on big companies. Who says that is not ongoing? You just don't hear about it that much because the companies downplay/hide it for obvious reasons. > They wouldn't waste it on causing a minor inconvinence. This is not a "minor" inconvenience. The losses to the Economy are already running into billions and counting. See for example https://www.youtube.com/watch?v=YedowOtznNo https://www.youtube.com/watch?v=YedowOtznNo > Lol. What next? Politicians always tell the truth? Everyone gets a free unicorn? This is just obviously not how the world works. There is a long history of anti-virus software being kind of crap. Snark/Glibness is not an argument. I have worked in Network Security and know for a fact that Kernel mode code is treated differently than User mode code in terms of scrutiny/testing/staging/release. Second, Crowdstrike is not just another anti-virus software; they are far more broader in scope/complex and hence their wide user base. Microsoft with their wide experience would have definitely processes in place to validate them comprehensively. Hence one should be cautious in taking this incident at face value and investigate everything thoroughly. I am almost sure multiple lawsuits are in the offing but am not so sure whether the full story will come out.
- rsynnott 2y ago
  I mean, what’s more likely, realistically? Shadowy saboteurs, or a cybersecurity company being poorly run, like, well, all other cybersecurity companies ever? Like, this is not new. They, as an industry, have been a byword for shoddy nonsense for literally decades.
  - michaelt 2y ago
    I mean, hypothetically you might think computer security companies would be full of passionate computer security enthusiasts. And as security tools break a lot of security norms - like sandboxing, least privilege, and running in userspace - you might think such enthusiasts would make sure they were coded with the utmost care. That this team of secure coding all-stars would be code reviewing, managing scope, fuzz testing, static analysing, formally validating and suchlike, as befits code running with the highest privilege levels. Surely huge multinational corporations wouldn't grant unlimited privileges to kernel modules written by clowns.... would they? If you believe the crowdstrike marketing, I can see how you might think shadowy saboteurs are the only plausible explanation.
    - rsynnott 2y ago
      > I mean, hypothetically you might think computer security companies would be full of passionate computer security enthusiasts. Eh, I mean, you might think that, absent any other information about the industry, but they're largely not.
- Muromec 2y ago
  Indeed, the issue could have been planted there by some nefarious occult hand
ykonstant 2y ago
Sorry to hijack this post, but for affected admins reading this: how is the recovery process going? What is your estimated time to normalcy? Also, for Linux and especially BSD admins: has this incident affected your perspective on EDR/XDR systems in the kernel? What would you suggest as an alternative to ensure regulatory compliance?
- tguvot 2y ago
  not strictly sysadmin. working on fedramp certification of really big system. we have crowdstrike in our usual production environment and i was always against it because i was always afraid of something like what happened. but security department pushed it through because this is something that they understand and can control. "security architecture of product" it's not concept that they understand. but answering your question, fedramp does requires EDR to be present. according to our fedramp advisors clamav is sufficient for passing audit.
- tgv 2y ago
  I do manage a few Linux machines (firewall passes only http and https -> nginx -> custom backend), but I'd never heard of Crowdstrike before. I don't even know what their product is supposed to do. As far as I can see, kernel level protection could only help prevent someone bypassing the firewall and trigger an exploit in nginx. But if Crowdstrike knows about such exploits, everyone does, and the firewall or nginx gets patched. What am I missing? Edit: I know it is supposed to implement "EDR", but it's always explained in the vaguest of terms.
  - Khaine 2y ago
    It is primarily aimed at workstations, although it does run and is run on servers. The idea is to be able to identify malware based on behaviour, rather than rely on signatures. EDR solutions hook into the kernel to log, and block system calls. They use this information to try and generically identify malware. For example you could detect ransomware by identifying a process that is enumerating a large number of files, reading from those files, and then saving those files. For a SOC, you can also use an EDR to identify files, hashes, connections to given IPs across your fleet of servers. This can allow you to see what devices have been compromised. The EDR can then isolate them, by blocking network syscalls and allow only the SOC to access to investigate and remediate. This is the value they provide (or at least claim to) to a cyber team
notepad0x90 2y ago
I'd like to see a proper journalistic investigation into every other EDR sensor on Linux as well. I really hate it when supposed journalists look for an angle and pursue it without critical thinking. But they could be right, they may have an issue in their engineering department recently. --- Speaking of,I wanted to mention a slightly related observation I've had recently scrolling through twitter (unfortunately) looking for information on this crisis. There are a lot of people who at least know enough technical jargon to probably work in IT or technology but they're using arguments like "it must be because of a DEI hire", for those who are unaware they're using "DEI" as a replacement for a hard-r N-word. In other words, I just learned that blatant racists are not a rarity in our corner of the world. If you're not one of them, I wanted to inform you of what they really mean. Fortunately I haven't seen this on HN so far :)
- bdjsiqoocwk 2y ago
  > it must be because of a DEI hire Of course, white people making mistakes is unheard of. Yeah it's quite frustrating how easily people latch on to the daily meme if it reinforces their preexisting prejudices. Republicans calling Kamala Harris the DEI candidate in 3, 2, 1...
  - wdlou 2y ago
    [flagged]
  - notepad0x90 2y ago
    For the record, crowdstrike isn't the type of company that does DEI hiring. For technical roles, they let technical people from the team you're interviewing for grill you and it is their opinion that decides the outcome of first round interviews.
    - rsynnott 2y ago
      I'm not sure what people think inclusive hiring _is_. In almost all cases, it's pretty much entirely about the _pipeline_, trying to attract a more diverse set of people to interview in the first place. There's generally nothing special about the interviews.
roblabla 2y ago
This is some very poor journalism. The linux issues are so, so very different from the windows BSOD issue. The redhat kernel panics were caused by a bug in the kernel ebpf implementation, likely a regression introduced by a rhel-specific patch. Blaming crowdstrike for this is stupid (just like blaming microsoft for the crowdstrike bsod is stupid). For background, I also work on a product using eBPFs, and had kernel updates cause kernel panics in my eBPF probes. In my case, the panic happened because the kernel decided to change an LSM hook interface, adding a new argument in front of the others. When the probe gets loaded, the kernel doesn’t typecheck the arguments, and so doesn’t realise the probe isn’t compatible with the new kernel. When the probe runs, shit happens and you end up with a kernel panic. eBPF probes causing kernel panics are almost always indication of a kernel bug, not a bug in the ebpf vendor. There are exceptions of course (such as an ebpf denying access to a resource causing pid1 to crash). But they’re very few.
- josefx 2y ago
  > likely a regression introduced by a rhel-specific patch. Blaming crowdstrike for this is stupid (just like blaming microsoft for the crowdstrike bsod is stupid). Yeah, it isn't as if crowdstrike was specifically advertising certified support for RedHat Linux and related products. https://www.crowdstrike.com/partners/falcon-for-red-hat/ https://www.crowdstrike.com/partners/falcon-for-red-hat/
  - dtx1 2y ago
    But being certified for RedHat Linux doesn't protect you from Bugs in the RedHat Kernel. That's on RedHat.
    - michaelt 2y ago
      Back in The Good Old Days, an OS vendor would release a beta version and software vendors would test against it and fix problems before the stable OS version was released. Obviously OS updates come out a lot more often these days than they used to - but we're also better at test automation than ever before, and beta software is easier to get than ever. It sure would be nice if companies that decide to produce kernel modules and to support certain OSes could test those kernel modules against those OSes at the beta stage.
yftsui 2y ago
Not surprising at all. My work issued MacBook top CPU time has been always `com.crowdstrike.falcon.Agent`, before Apple M1 released my Intel 2019 MacBook Pro can barely do any everyday task with that Agent running in the background. It crashed video calls, crashed the entire OS, I couldn't even smoothly type in an IDE back then.
- fernandotakai 2y ago
  yup. i worked at a company that used crowdstrike's falcon agent and it was an incredible cpu hog. nowadays i work at a place that uses a different solution and guess what: it's also a f-ing cpu (and i/o) hog -- it makes my m1 pro macbook slow to a crawl and there's no way to disable it.
  - em500 2y ago
    Part of Windows' bad reputation (for instability and poor performance) is likely due to Windows being the standard on corporate computers (outside of tech companies) where admins/management insist on installing tons of "enterprise solutions" that slow quad core PCs with lightning fast SSDs to a crawl. MacOS has the same problem as soon as they're deployed in large corporations. I had a company issued MacBook where a bad printer driver cut the battery life in half for a month or so.
    - fernandotakai 2y ago
      totally. i have a macbook air m2 and it performs better than my work m1 pro because of the bloatware. my zsh config spawns in ~90ms on my macbook air m2 while it takes 600ms in the m1 pro.
      - vips7L 2y ago
        My PowerShell startup time on my work laptop is around 4000ms. Corporate IT ruins everything.
    - acdha 2y ago
red_admiral 2y ago
In princple, yes, if you have third-party Ring 0 kernel-mode drivers, they could crash a POSIX system as well as a windows one. But that doesn't seem to be what happened here. Random idea that I haven't fully thought through: continue to run the kernel at Ring 0 and userland at Ring 3, but move "tools" like this to Ring 1.
- doikor 2y ago
  Problem with that is the tool can’t protect the system from any bad actor who gets ring 0 access. And even if it has ring 0 access it can’t really verify anything without secureboot or something like it verifying that nothing else started before it. This is also why Riots anti cheat runs as ring 0 as it has to protect the game against the owner/admin of the machine. (And after that there is still bios or firmware level exploits)
- ahazred8ta 2y ago
  Windows has an official ELAM Early Launch AntiMalware framework, which Crowdstrike complies with. The Crowdstrike driver is right where it's supposed to be, according to Microsoft.
chucke1992 2y ago
I mean fundamentally if Linux was used in the same use cases as Windows, it would have more issues like this. After all the scenarios of Windows are based on real world use cases.
ExoticPearTree 2y ago
People drink the kool-aid and believe that Linux needs antivirus. So many vendors tried and failed at scaring people into buying Linux antivirus that I've lost track over the years. My perspective is that it is a very very poor idea to have an anti-malware solution running on a Linux system. CrowdStrike is very persistent in their sales pitches and do FUD campaigns better than the competition to convinse people with decision making authority that Linux and Containers and what needs an antivirus.
- louthy 2y ago
  What makes you think Linux can't have a virus or malware? It has attack vectors like any piece of software, surely?
- nikcub 2y ago
  Recently helped somebody who accidentally left dockerd open to the internet with no auth. It was hacked, backdoored and running a crypto miner within hours. Only detected when the hosting company emailed him days later because the worm was further scanning other hosts from his machine. How would you stop, detect and/or remove this threat from this machine on a linux server without antivirus / EDR?
  - ExoticPearTree 2y ago
    You have metrics from the server which tell you that you're running 100% CPU for a period of time. If the crypto miner wasn't something very dumb, it would not be detected. And I can scan a network using nmap with an XML or grepable output format options. CrowdStrike doesn't remove threats. It would stop the process and quarantine the file. It requires knowledge on how to actually remove the threat beyond the quarantined file.
  - Ajedi32 2y ago
    Yeah, desktop Linux more obscure and generally used by more technically inclined people so malware is less common, but out of the box its just as vulnerable to viruses as Windows or any other OS that runs user-installed applications with no sandbox. If we were _really_ serious about endpoint security we'd be pushing business users towards operating systems with more modern security architectures, like Android, iOS or Chrome OS. That would be a lot of work though due to the fact that most legacy software is not compatible with those systems.
prmoustache 2y ago
I'd be interested to know how many of the companies involved in data breach / massive pwnages in the last 2 years were using crowdstrike on their devices/servers.
TacticalCoder 2y ago
> CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes How many Linux machines did crash a few days ago? How many Windows machines did crash a few days ago? Case closed.
- ranguna 2y ago
  This is not a competition or a case.
naveen99 2y ago
I want to see some proper accounting on what percent of company budget, percent of compute and memory, percent of employee time is spent in the name of security. Is it more like the usa on military 1-3% or like 10-20% ? 1% seems like a reasonable number. I suspect places that have crowdstrike installed are closer to 10-25% closer to military dictatorships.
- Aperocky 2y ago
  Might have been 4-5% in name but in between lost productivity by computers running as fast as a Pentium 4 (maybe slower?) and of course this complete dumpster fire that just happened maybe closer to 20%?
pilif 2y ago
> We're in the process of operationalizing an opt-in to this technique oh the jargon > We're making progress by the minute I better hope you do.
ChrisMarshallNY 2y ago
Oh dear. I would really hate to be anyone on the team responsible for the Falcon Sensor, right now. I suspect that their printer is churning out CVs, like nobody's business. > and will update this story if we receive substantial information. I'll lay odds that there's folks at Crowdstrike that are thinking of just responding with poop emojis.
jgalt212 2y ago
We had some hiccups over the weekend with Ubuntu Server / Hetzner. - apache2 crashed on VM (had not happened in 8 months--our entire tenure at Hetzner - another VM become entirely unresponsive--would only respond to ping. could not even access via control panel provided console. had to do a reboot. after that, the box seems to be ok. - we are still waiting on a response to our ticket from Hetzner.
deleted 2y ago
[deleted]

158 points by kenjackson 2y ago

I had read reports of this earlier which is what makes me speculate that the Windows Crowdstrike issue is more than "just a update error" i.e. there might be some nefarious hand behind this. Given that they were already aware of the Linux issue it boggles my mind that they did not take extra precautions when it came to Windows updates. We will have to wait and see for further trustworthy info. Btw - The article mentions Dave Plummer's analysis of the issue which might be easier for people to understand and worth a watch. - https://www.youtube.com/watch?v=wAzEJxOo1ts https://www.youtube.com/watch?v=wAzEJxOo1ts

bawolff 2y ago
Some part of a company already aware of an issue but different part still ships is a pretty common tale and seems much more likely than some nefarious conspiracy theory. (And that is even assuming this is the same issue, which seems questionable) After all, who exactly would benefit from such a nefarious scheme to crash windows computers? Certainly not Crowdstrike.
- tgv 2y ago
  Wrt trying to get something out of the crash: https://www.theregister.com/2024/07/19/cyber_criminals_quickly_exploit_crowdstrike/?td=rt-9cp https://www.theregister.com/2024/07/19/cyber_criminals_quick.... The conspiracy-minded could be suspicious of the quick response to this outage. "They must have known in advance!" Or they could suspect an overeager account manager at Crowdstrike, who wanted to show how important the product is. But indeed, this really sounds like it was an internal error.
- logicchains 2y ago
  >After all, who exactly would benefit from such a nefarious scheme to crash windows computers? Russia or China would certainly benefit from the ability to do this at a time of their choosing, and it's possible they could have an agent inside Crowdstrike, especially given China's history of industrial espionage.
  - bawolff 2y ago
    You think russia or china would benefit more from this then having a stealthy rootkit on basically every important computer in the world? I think russia or china are probably the least likely purpotrators possible. Their incentives strongly disalign with this.
- rramadass 2y ago
  > who exactly would benefit from such a nefarious scheme State Actors, given the current Geo-Political tensions. You have to take a all-in-all broader view. I remember a while ago Kaspersky was accused of data-siphoning/spying from computers it was installed on and other nefarious activities. See New Government Ban on Kaspersky Would Prevent Company from Updating Malware Signatures in U.S. - https://www.zetter-zeroday.com/new-government-ban-on-kaspersky-would-prevent-malware-signature-updates/ https://www.zetter-zeroday.com/new-government-ban-on-kaspers... As for your opening statement "Some part of a company already aware of an issue but different part still ships is a pretty common tale" is not applicable here since this code runs in kernel mode (in both OSes) and thus would be subject to far far greater scrutiny and testing than an ordinary app. As Dave Plummer points out in his analysis Microsoft Kernel Drivers are signed and certified after an exhaustive testing process. Even if Crowdstrike wrote their drivers as an interpreter and the data update files were actually programs in some p-code, Microsoft would have definitely known of it and its inherent vulnerabilities. I would bet money that Microsoft knows all about preventing threats/vulnerabilities than any other company simply because of their long experience and large userbase and thus would not have allowed Crowdstrike such a free hand.
  - bawolff 2y ago
    > State Actors, given the current Geo-Political tensions. I disagree. If state actors had this type of capability they would use it to spy on big companies. The espionage potential is huge. They wouldn't waste it on causing a minor inconvinence. > is not applicable here since this code runs in kernel mode (in both OSes) and thus would be subject to far far greater scrutiny and testing than an ordinary app Lol. What next? Politicians always tell the truth? Everyone gets a free unicorn? This is just obviously not how the world works. There is a long history of anti-virus software being kind of crap.
    - rramadass 2y ago
      > If state actors had this type of capability they would use it to spy on big companies. Who says that is not ongoing? You just don't hear about it that much because the companies downplay/hide it for obvious reasons. > They wouldn't waste it on causing a minor inconvinence. This is not a "minor" inconvenience. The losses to the Economy are already running into billions and counting. See for example https://www.youtube.com/watch?v=YedowOtznNo https://www.youtube.com/watch?v=YedowOtznNo > Lol. What next? Politicians always tell the truth? Everyone gets a free unicorn? This is just obviously not how the world works. There is a long history of anti-virus software being kind of crap. Snark/Glibness is not an argument. I have worked in Network Security and know for a fact that Kernel mode code is treated differently than User mode code in terms of scrutiny/testing/staging/release. Second, Crowdstrike is not just another anti-virus software; they are far more broader in scope/complex and hence their wide user base. Microsoft with their wide experience would have definitely processes in place to validate them comprehensively. Hence one should be cautious in taking this incident at face value and investigate everything thoroughly. I am almost sure multiple lawsuits are in the offing but am not so sure whether the full story will come out.
rsynnott 2y ago
I mean, what’s more likely, realistically? Shadowy saboteurs, or a cybersecurity company being poorly run, like, well, all other cybersecurity companies ever? Like, this is not new. They, as an industry, have been a byword for shoddy nonsense for literally decades.
- michaelt 2y ago
  I mean, hypothetically you might think computer security companies would be full of passionate computer security enthusiasts. And as security tools break a lot of security norms - like sandboxing, least privilege, and running in userspace - you might think such enthusiasts would make sure they were coded with the utmost care. That this team of secure coding all-stars would be code reviewing, managing scope, fuzz testing, static analysing, formally validating and suchlike, as befits code running with the highest privilege levels. Surely huge multinational corporations wouldn't grant unlimited privileges to kernel modules written by clowns.... would they? If you believe the crowdstrike marketing, I can see how you might think shadowy saboteurs are the only plausible explanation.
  - rsynnott 2y ago
    > I mean, hypothetically you might think computer security companies would be full of passionate computer security enthusiasts. Eh, I mean, you might think that, absent any other information about the industry, but they're largely not.
Muromec 2y ago
Indeed, the issue could have been planted there by some nefarious occult hand

ykonstant 2y ago

Sorry to hijack this post, but for affected admins reading this: how is the recovery process going? What is your estimated time to normalcy? Also, for Linux and especially BSD admins: has this incident affected your perspective on EDR/XDR systems in the kernel? What would you suggest as an alternative to ensure regulatory compliance?

tguvot 2y ago
not strictly sysadmin. working on fedramp certification of really big system. we have crowdstrike in our usual production environment and i was always against it because i was always afraid of something like what happened. but security department pushed it through because this is something that they understand and can control. "security architecture of product" it's not concept that they understand. but answering your question, fedramp does requires EDR to be present. according to our fedramp advisors clamav is sufficient for passing audit.
tgv 2y ago
I do manage a few Linux machines (firewall passes only http and https -> nginx -> custom backend), but I'd never heard of Crowdstrike before. I don't even know what their product is supposed to do. As far as I can see, kernel level protection could only help prevent someone bypassing the firewall and trigger an exploit in nginx. But if Crowdstrike knows about such exploits, everyone does, and the firewall or nginx gets patched. What am I missing? Edit: I know it is supposed to implement "EDR", but it's always explained in the vaguest of terms.
- Khaine 2y ago
  It is primarily aimed at workstations, although it does run and is run on servers. The idea is to be able to identify malware based on behaviour, rather than rely on signatures. EDR solutions hook into the kernel to log, and block system calls. They use this information to try and generically identify malware. For example you could detect ransomware by identifying a process that is enumerating a large number of files, reading from those files, and then saving those files. For a SOC, you can also use an EDR to identify files, hashes, connections to given IPs across your fleet of servers. This can allow you to see what devices have been compromised. The EDR can then isolate them, by blocking network syscalls and allow only the SOC to access to investigate and remediate. This is the value they provide (or at least claim to) to a cyber team

notepad0x90 2y ago

I'd like to see a proper journalistic investigation into every other EDR sensor on Linux as well. I really hate it when supposed journalists look for an angle and pursue it without critical thinking. But they could be right, they may have an issue in their engineering department recently. --- Speaking of,I wanted to mention a slightly related observation I've had recently scrolling through twitter (unfortunately) looking for information on this crisis. There are a lot of people who at least know enough technical jargon to probably work in IT or technology but they're using arguments like "it must be because of a DEI hire", for those who are unaware they're using "DEI" as a replacement for a hard-r N-word. In other words, I just learned that blatant racists are not a rarity in our corner of the world. If you're not one of them, I wanted to inform you of what they really mean. Fortunately I haven't seen this on HN so far :)

bdjsiqoocwk 2y ago
> it must be because of a DEI hire Of course, white people making mistakes is unheard of. Yeah it's quite frustrating how easily people latch on to the daily meme if it reinforces their preexisting prejudices. Republicans calling Kamala Harris the DEI candidate in 3, 2, 1...
- wdlou 2y ago
  [flagged]
- notepad0x90 2y ago
  For the record, crowdstrike isn't the type of company that does DEI hiring. For technical roles, they let technical people from the team you're interviewing for grill you and it is their opinion that decides the outcome of first round interviews.
  - rsynnott 2y ago
    I'm not sure what people think inclusive hiring _is_. In almost all cases, it's pretty much entirely about the _pipeline_, trying to attract a more diverse set of people to interview in the first place. There's generally nothing special about the interviews.

roblabla 2y ago

This is some very poor journalism. The linux issues are so, so very different from the windows BSOD issue. The redhat kernel panics were caused by a bug in the kernel ebpf implementation, likely a regression introduced by a rhel-specific patch. Blaming crowdstrike for this is stupid (just like blaming microsoft for the crowdstrike bsod is stupid). For background, I also work on a product using eBPFs, and had kernel updates cause kernel panics in my eBPF probes. In my case, the panic happened because the kernel decided to change an LSM hook interface, adding a new argument in front of the others. When the probe gets loaded, the kernel doesn’t typecheck the arguments, and so doesn’t realise the probe isn’t compatible with the new kernel. When the probe runs, shit happens and you end up with a kernel panic. eBPF probes causing kernel panics are almost always indication of a kernel bug, not a bug in the ebpf vendor. There are exceptions of course (such as an ebpf denying access to a resource causing pid1 to crash). But they’re very few.

josefx 2y ago
> likely a regression introduced by a rhel-specific patch. Blaming crowdstrike for this is stupid (just like blaming microsoft for the crowdstrike bsod is stupid). Yeah, it isn't as if crowdstrike was specifically advertising certified support for RedHat Linux and related products. https://www.crowdstrike.com/partners/falcon-for-red-hat/ https://www.crowdstrike.com/partners/falcon-for-red-hat/
- dtx1 2y ago
  But being certified for RedHat Linux doesn't protect you from Bugs in the RedHat Kernel. That's on RedHat.
  - michaelt 2y ago
    Back in The Good Old Days, an OS vendor would release a beta version and software vendors would test against it and fix problems before the stable OS version was released. Obviously OS updates come out a lot more often these days than they used to - but we're also better at test automation than ever before, and beta software is easier to get than ever. It sure would be nice if companies that decide to produce kernel modules and to support certain OSes could test those kernel modules against those OSes at the beta stage.

yftsui 2y ago

Not surprising at all. My work issued MacBook top CPU time has been always `com.crowdstrike.falcon.Agent`, before Apple M1 released my Intel 2019 MacBook Pro can barely do any everyday task with that Agent running in the background. It crashed video calls, crashed the entire OS, I couldn't even smoothly type in an IDE back then.

fernandotakai 2y ago
yup. i worked at a company that used crowdstrike's falcon agent and it was an incredible cpu hog. nowadays i work at a place that uses a different solution and guess what: it's also a f-ing cpu (and i/o) hog -- it makes my m1 pro macbook slow to a crawl and there's no way to disable it.
- em500 2y ago
  Part of Windows' bad reputation (for instability and poor performance) is likely due to Windows being the standard on corporate computers (outside of tech companies) where admins/management insist on installing tons of "enterprise solutions" that slow quad core PCs with lightning fast SSDs to a crawl. MacOS has the same problem as soon as they're deployed in large corporations. I had a company issued MacBook where a bad printer driver cut the battery life in half for a month or so.
  - fernandotakai 2y ago
    totally. i have a macbook air m2 and it performs better than my work m1 pro because of the bloatware. my zsh config spawns in ~90ms on my macbook air m2 while it takes 600ms in the m1 pro.
    - vips7L 2y ago
      My PowerShell startup time on my work laptop is around 4000ms. Corporate IT ruins everything.
  - acdha 2y ago

red_admiral 2y ago

In princple, yes, if you have third-party Ring 0 kernel-mode drivers, they could crash a POSIX system as well as a windows one. But that doesn't seem to be what happened here. Random idea that I haven't fully thought through: continue to run the kernel at Ring 0 and userland at Ring 3, but move "tools" like this to Ring 1.

doikor 2y ago
Problem with that is the tool can’t protect the system from any bad actor who gets ring 0 access. And even if it has ring 0 access it can’t really verify anything without secureboot or something like it verifying that nothing else started before it. This is also why Riots anti cheat runs as ring 0 as it has to protect the game against the owner/admin of the machine. (And after that there is still bios or firmware level exploits)
ahazred8ta 2y ago
Windows has an official ELAM Early Launch AntiMalware framework, which Crowdstrike complies with. The Crowdstrike driver is right where it's supposed to be, according to Microsoft.

chucke1992 2y ago

I mean fundamentally if Linux was used in the same use cases as Windows, it would have more issues like this. After all the scenarios of Windows are based on real world use cases.

ExoticPearTree 2y ago

People drink the kool-aid and believe that Linux needs antivirus. So many vendors tried and failed at scaring people into buying Linux antivirus that I've lost track over the years. My perspective is that it is a very very poor idea to have an anti-malware solution running on a Linux system. CrowdStrike is very persistent in their sales pitches and do FUD campaigns better than the competition to convinse people with decision making authority that Linux and Containers and what needs an antivirus.

louthy 2y ago
What makes you think Linux can't have a virus or malware? It has attack vectors like any piece of software, surely?
nikcub 2y ago
Recently helped somebody who accidentally left dockerd open to the internet with no auth. It was hacked, backdoored and running a crypto miner within hours. Only detected when the hosting company emailed him days later because the worm was further scanning other hosts from his machine. How would you stop, detect and/or remove this threat from this machine on a linux server without antivirus / EDR?
- ExoticPearTree 2y ago
  You have metrics from the server which tell you that you're running 100% CPU for a period of time. If the crypto miner wasn't something very dumb, it would not be detected. And I can scan a network using nmap with an XML or grepable output format options. CrowdStrike doesn't remove threats. It would stop the process and quarantine the file. It requires knowledge on how to actually remove the threat beyond the quarantined file.
- Ajedi32 2y ago
  Yeah, desktop Linux more obscure and generally used by more technically inclined people so malware is less common, but out of the box its just as vulnerable to viruses as Windows or any other OS that runs user-installed applications with no sandbox. If we were _really_ serious about endpoint security we'd be pushing business users towards operating systems with more modern security architectures, like Android, iOS or Chrome OS. That would be a lot of work though due to the fact that most legacy software is not compatible with those systems.

prmoustache 2y ago

I'd be interested to know how many of the companies involved in data breach / massive pwnages in the last 2 years were using crowdstrike on their devices/servers.

TacticalCoder 2y ago

> CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes How many Linux machines did crash a few days ago? How many Windows machines did crash a few days ago? Case closed.

ranguna 2y ago
This is not a competition or a case.

naveen99 2y ago

I want to see some proper accounting on what percent of company budget, percent of compute and memory, percent of employee time is spent in the name of security. Is it more like the usa on military 1-3% or like 10-20% ? 1% seems like a reasonable number. I suspect places that have crowdstrike installed are closer to 10-25% closer to military dictatorships.

Aperocky 2y ago
Might have been 4-5% in name but in between lost productivity by computers running as fast as a Pentium 4 (maybe slower?) and of course this complete dumpster fire that just happened maybe closer to 20%?

pilif 2y ago

> We're in the process of operationalizing an opt-in to this technique oh the jargon > We're making progress by the minute I better hope you do.

ChrisMarshallNY 2y ago

Oh dear. I would really hate to be anyone on the team responsible for the Falcon Sensor, right now. I suspect that their printer is churning out CVs, like nobody's business. > and will update this story if we receive substantial information. I'll lay odds that there's folks at Crowdstrike that are thinking of just responding with poop emojis.

jgalt212 2y ago

We had some hiccups over the weekend with Ubuntu Server / Hetzner. - apache2 crashed on VM (had not happened in 8 months--our entire tenure at Hetzner - another VM become entirely unresponsive--would only respond to ping. could not even access via control panel provided console. had to do a reboot. after that, the box seems to be ok. - we are still waiting on a response to our ticket from Hetzner.

deleted 2y ago

[deleted]