9 ms·

CrowdStrike broke Debian and Rocky Linux months ago

294 points by taubek 2y ago

jsheard 2y ago
There was also this report of CrowdStrike injecting a buggy DLL into Windows applications, which could cause the app to crash through no fault of its own: https://x.com/molecularmusing/status/1808756095860543916 https://x.com/molecularmusing/status/1808756095860543916
- FreakLegion 2y ago
  That's completely different. The injected DLL is an optional, off-by-default feature that admins are specifically warned to check for compatibility against their fleet before rolling out. It has four levels of increasingly unstable hooks it can apply, and there are repeated warnings about using it in the UI and documentation. E.g.: > Extended User Mode Data (XUMD) allows the sensor to monitor information in running processes by loading a library that can hook various user-mode APIs. Some endpoint telemetry can be gathered only through user-mode hooking. XUMD provides a flexible way to provide information about which APIs a process is leveraging. This information feeds a variety of prevention mechanisms that are available to the sensor based on the accumulated behavior observed. Unlike Additional User Mode Data (AUMD), the cloud can dynamically modify XUMD visibility without a sensor update. Supported prevention policy settings for XUMD: Disabled: The extended visibility, detection, and prevention capabilities of XUMD are disabled. The hooking library is not loaded into processes. Cautious: XUMD is enabled with high-confidence hooks that are accessible to detection and prevention logic. Performance and compatibility impact at this setting is expected to be negligible, but we recommend testing this setting in a staging environment before deploying it to production. Moderate: XUMD is enabled with high- and medium-confidence hooks that are accessible to detection and prevention logic. This setting can result in performance or application-compatibility impact but provides expanded visibility. Performance impact at this setting is expected to be negligible, but we recommend testing this setting in a staging environment before deploying it to production. Aggressive: XUMD is enabled with high-, medium-, and low-confidence hooks that are accessible to detection and prevention logic. This setting can result in significant performance or application-compatibility problems. This setting is not recommended for production environments without significant prior testing in a staging environment. Extra Aggressive: XUMD is enabled with high-, medium-, low-, and experimental-confidence hooks that are accessible to detection and prevention logic. This setting can result in significant performance problems or application compatibility problems. This setting is not recommended for any production environment but might be appropriate for penetration and stress testing in specific limited deployments. Because XUMD is loaded in user processes that were not developed with it, negative interactions with other software might occur. This is most common when other security products are installed. In certain software environments, conflicting software might crash, fail to start, or suffer degraded performance. In these scenarios, move a test system into a policy where XUMD is disabled, reboot the host, and then retry the software. If the issue is resolved, open a Support case and request assistance in resolving the conflict. Support can assist in diagnosing and resolving these issues between XUMD and specific software. To determine which processes have loaded the XUMD DLL, run the following command at the command line: tasklist /m csxumd*
andyjohnson0 2y ago
Relevant comment from yesterday's Crowdstrike mega-thread: "Crowdstrike did this to our production linux fleet back on April 19th, and I've been dying to rant about it." [1] Continues with a multi-para rant. [1] https://news.ycombinator.com/item?id=41005936 https://news.ycombinator.com/item?id=41005936
- rkagerer 2y ago
  > ...weeks later had a root cause analysis that didn't cover our scenario (Debian stable running version n-1, I think, which is a supported configuration) in their test matrix. In our own post mortem there was no real ability to prevent the same thing from happening again -- "we push software to your machines any time we want, whether or not it's urgent, without testing it"
- ufo 2y ago
  Not just a relevant comment; this HN thread serms to be a primary source for the article.
can16358p 2y ago
Product quality is on freefall: from aircraft to software. Lack of QA is the norm nowadays as everyone just care about the extra penny.
- bloopernova 2y ago
  The economic consequences for doing things wrong are less than the profit made. Until that changes, nothing else will.
  - fifteen1506 2y ago
    Penalty 1: stock buybacks forbidden Penalty 2: separate the company into two, separating financialization procedures from manufacturing ones. Penalty 3: greenlight a union by default.
    - rbanffy 2y ago
      > Penalty 3: greenlight a union by default. This should be a fundamental feature in any functioning society that expects (or wishes) to remain functional.
    - photonbeam 2y ago
      Penalty 2 is the same as closing the company down
      - AnimalMuppet 2y ago
        No, why? If there's a company that provides actual value, why should splitting off the financialization part kill the part that provides real value? It's maybe the same as closing down the financialization part of the company, but if so, what loss to society?
        immibis 2y ago
        The financialization (profit-making) is the only reason anyone with any power bothers to keep any of it running.
xyst 2y ago
Need a “crowdstrike.sucks” or maybe a general “it.sucks/{company}” to gather all of these company misgivings. Avoid this company at all costs. Move to competitors which offer the same “audit passing” requirements
- jsheard 2y ago
  Funnily enough crowdstrike.sucks is already registered through CSC, the same fancy corporate registrar which handles crowdstrike.com. The people behind .sucks have a great racket going, every big brand has to pay $300/year for their name dot sucks just so nobody else can get it.
  - rbanffy 2y ago
    They also own “clownstrike.com”. The fact they proactively registered it says a lot.
    - j-bos 2y ago
      Proative marketing culture > proactive testing culture
      - svnt 2y ago
        Probably the relationship is more causative: A decided lack of a proactive testing culture feeds back into the org through sales interactions but only makes it as far as the discretionary credit card of someone in the marketing group
    - smsm42 2y ago
      They've been called that way before they made Jun 19 the International Bluescreen Day.
  - xyst 2y ago
    Wow, $300/year is wild. In this influencer age, negative attention is the cash grab. <researching how to setup TLD, maybe set it up as a DAO>
righthand 2y ago
“No one noticed” which is a cute way to say that Crowdstrike suppressed the media noticing. The day of the bug, the HN post had comments about how people tried reporting the issue months ago. Even the article is written as people noticing. So who didn’t notice? Or were the issues not popular enough to be not ignored?
- rbanffy 2y ago
  I remember something about it, but, at the time I thought that very few people would install such software in Linux (and, indeed, very few companies do). The blast radius was minimal, probably smaller than a bad Nvidia driver.
  - LtWorf 2y ago
    My company installs falcon on servers! I know because our apt mirror takes much longer now to sync. That's because the crowdstrike agent is using all the CPU to scan every .deb package. And they're ar-tar packed, so unless there's a special algorithm to scan for them, nothing will ever be found in them anyway.
- WillPostForFood 2y ago
  Crowdstrike suppressed the media noticing You think they sent checks to the NY Times, WaPo, and major networks or something? Media doesn't care if some servers crash unless it is noticeable in the world at large (like the airline groundings).
  - oddthaty3 2y ago
    [flagged]
neilwilson 2y ago
This is all a consequence of firms being able to contract out of consequential liability. Perhaps we should render such clauses unenforceable, as we do with contracting out of consequential loss of life. Or at least limit them.
- immibis 2y ago
  That's what that European cyber security directive tried to do, right? The one that everyone was mad about until they carved out an exception for FOSS?
lkdfjlkdfjlg 2y ago
> ( ... ) experienced significant disruptions as a result of CrowdStrike updates, raising serious concerns about the company's software update and testing procedures To me the issue isn't CrowdStrike's testing procedures. To me the issue is why does Debian depend on CrowdStrike? Does anyone understand this?
- Sakos 2y ago
  Debian doesn't depend on CrowdStrike. CS provides Linux clients for organizations that want to deploy it with their Linux workstations/servers. From the article: > In April, a CrowdStrike update caused all Debian Linux servers in a civic tech lab to crash simultaneously and refuse to boot. The update proved incompatible with the latest stable version of Debian, despite the specific Linux configuration being supposedly supported. The lab's IT team discovered that removing CrowdStrike allowed the machines to boot and reported the incident. Debian is as at fault for CrowdStrike's incompetence and negligence as Microsoft is. That is to say, not at all.
  - lkdfjlkdfjlg 2y ago
    Right, thanks for that. My fault for skipping every other word. This had really soured my day for a bit there, but you brought it back. Debian is one of those things that I consider a source of stability in my software life, together with git, wikipedia, and openstreetmap. Believing that it depends on some dodgy company really put me in a bad mood.
    - rbanffy 2y ago
      Indeed. Debian is the BSD of the Linux world, CentOS/RHEL being the Solaris. As usual, this was a self-inflicted pain by companies wishing to check yet another box to externalise liability.
- jccooper 2y ago
  Debian doesn't depend on CrowdStrike. That's about people who installed it on such systems.
paholg 2y ago
I used to work in this space, and I always had the nagging question of "is any of this stuff actually useful?" It seems a hard question to answer, but are there any third party studies of the effectiveness of Crowdstrike et al. or are we all making our lives worse for some security theater?
- marcosdumay 2y ago
  Have you seen it actually stop anything? (I'm sure the company that made the tool used it too, right?) If I make a WWW-wide question of "has anybody seen it?", somebody will appear. But the number of people that got a security flaw caused by those tools is huge, and the people that got stability and availability problems because of them is basically the number of people that use them.
  - paholg 2y ago
    I worked on something different, but we integrated with Crowdstrike and such. Maybe someone could do a study of like breaches in Fortune 500 companies that use an EDR vs. those that don't, but they probably all do at this point.
    - sam_bristow 2y ago
      I would imagine any study like that would also be just packed with confounding factors.
- nocsi 2y ago
  It’s like trying to study the effectiveness of antivirus. But you already said it. As long as it produces consumable metrics a c-level can ingest, then it’s worth it. Because really, how does it make sense to add something so invasive? Anyways in the 90s, antivirus makers also wrote viruses. They’d go on to flood networks with their creations, but magically block infection for their subscribers.
lambdaone 2y ago
What gets me is that much of the OSS/Linux ecosystem consists of thousandas of lashed together piles of code written by independent and only very loosely coordinated groups, much of it code and lashed together by amateurs for free, and it is still more robust than software created by multi-billion dollar corporations. Perhaps one reason is that OSS system programmers are washing their dirty linen in public; not a matter of "many eyes make bugs shallow", but that "any eyes make bad code embarassing". Just for example, I'm planning to make one of my commercial projects open source, and I am going to have to do a lot of fixing up before I'm willing to show the source code in public. It's not terrible code, and it works perfectly well, but it's not the sort of code I'd be willing to show to the world in general. Better documentation, TODO and FIXME fixing, checking comments still reflect the code, etc. etc. But for all my sense of shame for this (perfectly good and working) software, I've seen the insides of several closed-source commercial code bases and seen far, far worse. I would imagine most "enterprise" software is written to a similar standard.
- teeheelol 2y ago
  Yep. Some of the garbage I've seen out there is shocking. It scares me. Then I try and get fractional scaling working on Wayland with my NVidia card and want to gouge my eyes out with frustration that after a decade I still can't do what I can do on a closed source thing that came free with my computer. Actually make that 25 years now. The enterprise crap while horrible, actually mostly does work reasonably well. Sometimes I feel dirty about this. Quality is therefore relative to the consumer. The attention is on what the engineers care about with Linux, not the users I find. Where there's an impedance mismatch there are a lot of unhappy users.
  - takluyver 2y ago
    > after a decade I still can't do what I can do on a closed source thing that came free with my computer I don't know the specifics, but there's a good chance that your issue is ultimately because Nvidia wants to keep stuff closed, and Linux is not their main market - at least for actual graphics, I guess these days it's a big market for GPU computing. So it's the interface between closed and open source that's giving you grief.
pipes 2y ago
What is cloud strikes unique selling point? Genuine question, because I'd never heard of them before this.
- CSMastermind 2y ago
  At face value they sell it as a security product. In reality it's a tool for employers to spy on employees, control what they do on the device, etc. The unique selling point in a sense is plausible deniability about the true purpose of the software.
- rbanffy 2y ago
  “Dear CTO, do you want to be seen as the person who didn’t take every measure to stop hackers from stealing your data? Then buy our stuff” Something like that.
  - teeray 2y ago
    Close, but instead of an email, it’s probably a conversation during a game of golf at an exclusive private club.
    - rbanffy 2y ago
      “It’s a nice IT infrastructure you have at XCorp. Would be a shame if something happened to it”… Or “You really don’t want to be the last one in the finger pointing chain. Come on, sign here and you can point to us. Our lawyers will deal with the mess”.
      - Am4TIfIsER0ppos 2y ago
        > “It’s a nice IT infrastructure you have at XCorp. Would be a shame if something happened to it”… That happens to be the Cloudflare sales pitch too.
    - marcosdumay 2y ago
      It's an auditor, from insurance or some financial service, with a list of 2 or 3 companies, 1 of them being famously invaded since the turn of the century (once, continually).
yolo3000 2y ago
Is anyone here using Crowdstrike, what does it do? I see it referred to as an 'anti-virus'? I have it installed on my work laptops and I see it as a keylogger and activity monitor. "I got nothing to hide", but still bothers me when some corporate super users spy on me.
- surfingdino 2y ago
  It gets you a sign-off from the security, compliance, and legal teams.
  - jdgoesmarching 2y ago
    This right here
- kchr 2y ago
  Instead of calling out AV/EDR solutions as malware and spyware, I have a better solution: Stop using your workstations for private stuff. They belong to the company, and they are a liability since you use them to access the company environment and could cause damage if actual malware would find its way onto it. Use a separate device for private stuff if you want privacy.
  - josephcsible 2y ago
    Yes, you should be using a separate computer for personal stuff, but you should still be calling out that spyware for what it is too.
deleted 2y ago
[deleted]
bigcat12345678 2y ago
I think someone noticed. And was thinking: People wont be happy to fix this and I am not allowed to fix either. Well, it might be just like the 3000 other rare issues that would one day break the world's IT. Who cares...
kermatt 2y ago
Is it possible that these events had less impact because the damage was less / more easily fixed due to the nature of the OS? Or perhaps because the admins of Linux systems are typically more knowledable about how to run their platforms, and not just install them? Or is it due to sheer numbers of enterprise software running on Windows?
kayo_20211030 2y ago
In the end, because of regulatory pressure, the only pressure that matters in a commercial environment, there will be three supported OS's: a windows one, a mac one, and probably one, and only one, Linux distribution, or flavor thereof. Everything else will be toast in a commercial environment. For Linux there might be this AWS one, and that Google one, but they'll be close. And, in order to satisfy regulatory requirements, they'll be very, very close. Commercial organizations have bosses and, more ominously, regulators. We, and they, need a checkbox checked. So let's not fool ourselves with thoughts of freedom and liberty. There's a real world out there. CrowdStrike screwed up, but there's more chance that a 1000 linux's go to one than 1 CrowdStrike goes to zero.
deleted 2y ago
[deleted]
utensil4778 2y ago
Huh, this story sounds familiar. I read a HN comment the other day telling this same story. They didn't just turn a random HN comment into a news article, did they? Yup. They did. At least they cited it I suppose.
smsm42 2y ago
> The update proved incompatible with the latest stable version of Debian, despite the specific Linux configuration being supposedly supported. > The analysis revealed that the Debian Linux configuration was not included in their test matrix. This is suspiciously close to actual fraud. They declare they support configuration X, but they actually do not do any testing on configuration X. That's like telling me my car will have seatbelts, but in no place in manufacturing it is ensured the seatbelts are actually installed and work. I think a car maker that does something like that would be prosecuted. Why Crowdstrike isn't? I mean, one thing if they don't support some version of Linux - ok, too many of them, I can get it. But if you advertise support for it without even bothering to test on it - that's at best willful negligence and possibly outright fraud.
3np 2y ago
Anecdote from SWIM: Employer corp supposedly has CS deployed on all endpoints. Been getting away with just running it in a VM with restricted resources and not hearing anything about it. Did notice the VM failing around that time. Also heard from an ex-coworker in another large corp where IT just gave up on enforcing compliance for Linux endpoints. I wouldn't be surprised if some IT admins effectively adopt a "don't ask don't tell" policy here: If you can figure out the alternative stack by yourself without causing noise or lying about it, you're on your own. It'd certainly make sense if the motivation for enforcement is largely checkbox- compliance. I wonder just how widespread this kind of admittedly malicious compliance is and how much it contributed to the April incident not being bigger news...
romaniitedomum 2y ago
Not just Debian and Rocky, but RHEL too. https://access.redhat.com/solutions/7068083 https://access.redhat.com/solutions/7068083 I ran into this doing CentOS7 to Alma9 upgrades. The bug was in RHEL, Alma and Rocky and any other distro derived from RHEL. I had a VM go into an endless reboot cycle and the only way to get back in was to boot to an emergency rescue console and disable falcon-sensor. The problem was something to do with eBPF, and one of the workarounds was to tell falcon sensor to use kernel mode and not auto or user (bpf) mode. We don't allow automatic updates on hosts, however, so thankfully this was contained, but it certainly begs the question of just what testing Crowdstrike are doing.
egorfine 2y ago
> CrowdStrike should prioritize rigorous testing across all supported They should not. Testing costs money and they aren't selling their product to a company that needs or wants it on a competitive market. Their business model is based on shoving the product down the throat of enterprises due to compliance and therefore they have zero incentive to invest any money into quality.

294 points by taubek 2y ago

There was also this report of CrowdStrike injecting a buggy DLL into Windows applications, which could cause the app to crash through no fault of its own: https://x.com/molecularmusing/status/1808756095860543916 https://x.com/molecularmusing/status/1808756095860543916

FreakLegion 2y ago
That's completely different. The injected DLL is an optional, off-by-default feature that admins are specifically warned to check for compatibility against their fleet before rolling out. It has four levels of increasingly unstable hooks it can apply, and there are repeated warnings about using it in the UI and documentation. E.g.: > Extended User Mode Data (XUMD) allows the sensor to monitor information in running processes by loading a library that can hook various user-mode APIs. Some endpoint telemetry can be gathered only through user-mode hooking. XUMD provides a flexible way to provide information about which APIs a process is leveraging. This information feeds a variety of prevention mechanisms that are available to the sensor based on the accumulated behavior observed. Unlike Additional User Mode Data (AUMD), the cloud can dynamically modify XUMD visibility without a sensor update. Supported prevention policy settings for XUMD: Disabled: The extended visibility, detection, and prevention capabilities of XUMD are disabled. The hooking library is not loaded into processes. Cautious: XUMD is enabled with high-confidence hooks that are accessible to detection and prevention logic. Performance and compatibility impact at this setting is expected to be negligible, but we recommend testing this setting in a staging environment before deploying it to production. Moderate: XUMD is enabled with high- and medium-confidence hooks that are accessible to detection and prevention logic. This setting can result in performance or application-compatibility impact but provides expanded visibility. Performance impact at this setting is expected to be negligible, but we recommend testing this setting in a staging environment before deploying it to production. Aggressive: XUMD is enabled with high-, medium-, and low-confidence hooks that are accessible to detection and prevention logic. This setting can result in significant performance or application-compatibility problems. This setting is not recommended for production environments without significant prior testing in a staging environment. Extra Aggressive: XUMD is enabled with high-, medium-, low-, and experimental-confidence hooks that are accessible to detection and prevention logic. This setting can result in significant performance problems or application compatibility problems. This setting is not recommended for any production environment but might be appropriate for penetration and stress testing in specific limited deployments. Because XUMD is loaded in user processes that were not developed with it, negative interactions with other software might occur. This is most common when other security products are installed. In certain software environments, conflicting software might crash, fail to start, or suffer degraded performance. In these scenarios, move a test system into a policy where XUMD is disabled, reboot the host, and then retry the software. If the issue is resolved, open a Support case and request assistance in resolving the conflict. Support can assist in diagnosing and resolving these issues between XUMD and specific software. To determine which processes have loaded the XUMD DLL, run the following command at the command line: tasklist /m csxumd*

andyjohnson0 2y ago

Relevant comment from yesterday's Crowdstrike mega-thread: "Crowdstrike did this to our production linux fleet back on April 19th, and I've been dying to rant about it." [1] Continues with a multi-para rant. [1] https://news.ycombinator.com/item?id=41005936 https://news.ycombinator.com/item?id=41005936

rkagerer 2y ago
> ...weeks later had a root cause analysis that didn't cover our scenario (Debian stable running version n-1, I think, which is a supported configuration) in their test matrix. In our own post mortem there was no real ability to prevent the same thing from happening again -- "we push software to your machines any time we want, whether or not it's urgent, without testing it"
ufo 2y ago
Not just a relevant comment; this HN thread serms to be a primary source for the article.

can16358p 2y ago

Product quality is on freefall: from aircraft to software. Lack of QA is the norm nowadays as everyone just care about the extra penny.

bloopernova 2y ago
The economic consequences for doing things wrong are less than the profit made. Until that changes, nothing else will.
- fifteen1506 2y ago
  Penalty 1: stock buybacks forbidden Penalty 2: separate the company into two, separating financialization procedures from manufacturing ones. Penalty 3: greenlight a union by default.
  - rbanffy 2y ago
    > Penalty 3: greenlight a union by default. This should be a fundamental feature in any functioning society that expects (or wishes) to remain functional.
  - photonbeam 2y ago
    Penalty 2 is the same as closing the company down
    - AnimalMuppet 2y ago
      No, why? If there's a company that provides actual value, why should splitting off the financialization part kill the part that provides real value? It's maybe the same as closing down the financialization part of the company, but if so, what loss to society?
      - immibis 2y ago
        The financialization (profit-making) is the only reason anyone with any power bothers to keep any of it running.

xyst 2y ago

Need a “crowdstrike.sucks” or maybe a general “it.sucks/{company}” to gather all of these company misgivings. Avoid this company at all costs. Move to competitors which offer the same “audit passing” requirements

jsheard 2y ago
Funnily enough crowdstrike.sucks is already registered through CSC, the same fancy corporate registrar which handles crowdstrike.com. The people behind .sucks have a great racket going, every big brand has to pay $300/year for their name dot sucks just so nobody else can get it.
- rbanffy 2y ago
  They also own “clownstrike.com”. The fact they proactively registered it says a lot.
  - j-bos 2y ago
    Proative marketing culture > proactive testing culture
    - svnt 2y ago
      Probably the relationship is more causative: A decided lack of a proactive testing culture feeds back into the org through sales interactions but only makes it as far as the discretionary credit card of someone in the marketing group
  - smsm42 2y ago
    They've been called that way before they made Jun 19 the International Bluescreen Day.
- xyst 2y ago
  Wow, $300/year is wild. In this influencer age, negative attention is the cash grab. <researching how to setup TLD, maybe set it up as a DAO>

righthand 2y ago

“No one noticed” which is a cute way to say that Crowdstrike suppressed the media noticing. The day of the bug, the HN post had comments about how people tried reporting the issue months ago. Even the article is written as people noticing. So who didn’t notice? Or were the issues not popular enough to be not ignored?

rbanffy 2y ago
I remember something about it, but, at the time I thought that very few people would install such software in Linux (and, indeed, very few companies do). The blast radius was minimal, probably smaller than a bad Nvidia driver.
- LtWorf 2y ago
  My company installs falcon on servers! I know because our apt mirror takes much longer now to sync. That's because the crowdstrike agent is using all the CPU to scan every .deb package. And they're ar-tar packed, so unless there's a special algorithm to scan for them, nothing will ever be found in them anyway.
WillPostForFood 2y ago
Crowdstrike suppressed the media noticing You think they sent checks to the NY Times, WaPo, and major networks or something? Media doesn't care if some servers crash unless it is noticeable in the world at large (like the airline groundings).
- oddthaty3 2y ago
  [flagged]

neilwilson 2y ago

This is all a consequence of firms being able to contract out of consequential liability. Perhaps we should render such clauses unenforceable, as we do with contracting out of consequential loss of life. Or at least limit them.

immibis 2y ago
That's what that European cyber security directive tried to do, right? The one that everyone was mad about until they carved out an exception for FOSS?

lkdfjlkdfjlg 2y ago

> ( ... ) experienced significant disruptions as a result of CrowdStrike updates, raising serious concerns about the company's software update and testing procedures To me the issue isn't CrowdStrike's testing procedures. To me the issue is why does Debian depend on CrowdStrike? Does anyone understand this?

Sakos 2y ago
Debian doesn't depend on CrowdStrike. CS provides Linux clients for organizations that want to deploy it with their Linux workstations/servers. From the article: > In April, a CrowdStrike update caused all Debian Linux servers in a civic tech lab to crash simultaneously and refuse to boot. The update proved incompatible with the latest stable version of Debian, despite the specific Linux configuration being supposedly supported. The lab's IT team discovered that removing CrowdStrike allowed the machines to boot and reported the incident. Debian is as at fault for CrowdStrike's incompetence and negligence as Microsoft is. That is to say, not at all.
- lkdfjlkdfjlg 2y ago
  Right, thanks for that. My fault for skipping every other word. This had really soured my day for a bit there, but you brought it back. Debian is one of those things that I consider a source of stability in my software life, together with git, wikipedia, and openstreetmap. Believing that it depends on some dodgy company really put me in a bad mood.
  - rbanffy 2y ago
    Indeed. Debian is the BSD of the Linux world, CentOS/RHEL being the Solaris. As usual, this was a self-inflicted pain by companies wishing to check yet another box to externalise liability.
jccooper 2y ago
Debian doesn't depend on CrowdStrike. That's about people who installed it on such systems.

paholg 2y ago

I used to work in this space, and I always had the nagging question of "is any of this stuff actually useful?" It seems a hard question to answer, but are there any third party studies of the effectiveness of Crowdstrike et al. or are we all making our lives worse for some security theater?

marcosdumay 2y ago
Have you seen it actually stop anything? (I'm sure the company that made the tool used it too, right?) If I make a WWW-wide question of "has anybody seen it?", somebody will appear. But the number of people that got a security flaw caused by those tools is huge, and the people that got stability and availability problems because of them is basically the number of people that use them.
- paholg 2y ago
  I worked on something different, but we integrated with Crowdstrike and such. Maybe someone could do a study of like breaches in Fortune 500 companies that use an EDR vs. those that don't, but they probably all do at this point.
  - sam_bristow 2y ago
    I would imagine any study like that would also be just packed with confounding factors.
nocsi 2y ago
It’s like trying to study the effectiveness of antivirus. But you already said it. As long as it produces consumable metrics a c-level can ingest, then it’s worth it. Because really, how does it make sense to add something so invasive? Anyways in the 90s, antivirus makers also wrote viruses. They’d go on to flood networks with their creations, but magically block infection for their subscribers.

lambdaone 2y ago

What gets me is that much of the OSS/Linux ecosystem consists of thousandas of lashed together piles of code written by independent and only very loosely coordinated groups, much of it code and lashed together by amateurs for free, and it is still more robust than software created by multi-billion dollar corporations. Perhaps one reason is that OSS system programmers are washing their dirty linen in public; not a matter of "many eyes make bugs shallow", but that "any eyes make bad code embarassing". Just for example, I'm planning to make one of my commercial projects open source, and I am going to have to do a lot of fixing up before I'm willing to show the source code in public. It's not terrible code, and it works perfectly well, but it's not the sort of code I'd be willing to show to the world in general. Better documentation, TODO and FIXME fixing, checking comments still reflect the code, etc. etc. But for all my sense of shame for this (perfectly good and working) software, I've seen the insides of several closed-source commercial code bases and seen far, far worse. I would imagine most "enterprise" software is written to a similar standard.

teeheelol 2y ago
Yep. Some of the garbage I've seen out there is shocking. It scares me. Then I try and get fractional scaling working on Wayland with my NVidia card and want to gouge my eyes out with frustration that after a decade I still can't do what I can do on a closed source thing that came free with my computer. Actually make that 25 years now. The enterprise crap while horrible, actually mostly does work reasonably well. Sometimes I feel dirty about this. Quality is therefore relative to the consumer. The attention is on what the engineers care about with Linux, not the users I find. Where there's an impedance mismatch there are a lot of unhappy users.
- takluyver 2y ago
  > after a decade I still can't do what I can do on a closed source thing that came free with my computer I don't know the specifics, but there's a good chance that your issue is ultimately because Nvidia wants to keep stuff closed, and Linux is not their main market - at least for actual graphics, I guess these days it's a big market for GPU computing. So it's the interface between closed and open source that's giving you grief.

pipes 2y ago

What is cloud strikes unique selling point? Genuine question, because I'd never heard of them before this.

CSMastermind 2y ago
At face value they sell it as a security product. In reality it's a tool for employers to spy on employees, control what they do on the device, etc. The unique selling point in a sense is plausible deniability about the true purpose of the software.
rbanffy 2y ago
“Dear CTO, do you want to be seen as the person who didn’t take every measure to stop hackers from stealing your data? Then buy our stuff” Something like that.
- teeray 2y ago
  Close, but instead of an email, it’s probably a conversation during a game of golf at an exclusive private club.
  - rbanffy 2y ago
    “It’s a nice IT infrastructure you have at XCorp. Would be a shame if something happened to it”… Or “You really don’t want to be the last one in the finger pointing chain. Come on, sign here and you can point to us. Our lawyers will deal with the mess”.
    - Am4TIfIsER0ppos 2y ago
      > “It’s a nice IT infrastructure you have at XCorp. Would be a shame if something happened to it”… That happens to be the Cloudflare sales pitch too.
  - marcosdumay 2y ago
    It's an auditor, from insurance or some financial service, with a list of 2 or 3 companies, 1 of them being famously invaded since the turn of the century (once, continually).

yolo3000 2y ago

Is anyone here using Crowdstrike, what does it do? I see it referred to as an 'anti-virus'? I have it installed on my work laptops and I see it as a keylogger and activity monitor. "I got nothing to hide", but still bothers me when some corporate super users spy on me.

surfingdino 2y ago
It gets you a sign-off from the security, compliance, and legal teams.
- jdgoesmarching 2y ago
  This right here
kchr 2y ago
Instead of calling out AV/EDR solutions as malware and spyware, I have a better solution: Stop using your workstations for private stuff. They belong to the company, and they are a liability since you use them to access the company environment and could cause damage if actual malware would find its way onto it. Use a separate device for private stuff if you want privacy.
- josephcsible 2y ago
  Yes, you should be using a separate computer for personal stuff, but you should still be calling out that spyware for what it is too.

deleted 2y ago

[deleted]

bigcat12345678 2y ago

I think someone noticed. And was thinking: People wont be happy to fix this and I am not allowed to fix either. Well, it might be just like the 3000 other rare issues that would one day break the world's IT. Who cares...

kermatt 2y ago

Is it possible that these events had less impact because the damage was less / more easily fixed due to the nature of the OS? Or perhaps because the admins of Linux systems are typically more knowledable about how to run their platforms, and not just install them? Or is it due to sheer numbers of enterprise software running on Windows?

kayo_20211030 2y ago

In the end, because of regulatory pressure, the only pressure that matters in a commercial environment, there will be three supported OS's: a windows one, a mac one, and probably one, and only one, Linux distribution, or flavor thereof. Everything else will be toast in a commercial environment. For Linux there might be this AWS one, and that Google one, but they'll be close. And, in order to satisfy regulatory requirements, they'll be very, very close. Commercial organizations have bosses and, more ominously, regulators. We, and they, need a checkbox checked. So let's not fool ourselves with thoughts of freedom and liberty. There's a real world out there. CrowdStrike screwed up, but there's more chance that a 1000 linux's go to one than 1 CrowdStrike goes to zero.

deleted 2y ago

[deleted]

utensil4778 2y ago

Huh, this story sounds familiar. I read a HN comment the other day telling this same story. They didn't just turn a random HN comment into a news article, did they? Yup. They did. At least they cited it I suppose.

smsm42 2y ago

> The update proved incompatible with the latest stable version of Debian, despite the specific Linux configuration being supposedly supported. > The analysis revealed that the Debian Linux configuration was not included in their test matrix. This is suspiciously close to actual fraud. They declare they support configuration X, but they actually do not do any testing on configuration X. That's like telling me my car will have seatbelts, but in no place in manufacturing it is ensured the seatbelts are actually installed and work. I think a car maker that does something like that would be prosecuted. Why Crowdstrike isn't? I mean, one thing if they don't support some version of Linux - ok, too many of them, I can get it. But if you advertise support for it without even bothering to test on it - that's at best willful negligence and possibly outright fraud.

3np 2y ago

Anecdote from SWIM: Employer corp supposedly has CS deployed on all endpoints. Been getting away with just running it in a VM with restricted resources and not hearing anything about it. Did notice the VM failing around that time. Also heard from an ex-coworker in another large corp where IT just gave up on enforcing compliance for Linux endpoints. I wouldn't be surprised if some IT admins effectively adopt a "don't ask don't tell" policy here: If you can figure out the alternative stack by yourself without causing noise or lying about it, you're on your own. It'd certainly make sense if the motivation for enforcement is largely checkbox- compliance. I wonder just how widespread this kind of admittedly malicious compliance is and how much it contributed to the April incident not being bigger news...

romaniitedomum 2y ago

Not just Debian and Rocky, but RHEL too. https://access.redhat.com/solutions/7068083 https://access.redhat.com/solutions/7068083 I ran into this doing CentOS7 to Alma9 upgrades. The bug was in RHEL, Alma and Rocky and any other distro derived from RHEL. I had a VM go into an endless reboot cycle and the only way to get back in was to boot to an emergency rescue console and disable falcon-sensor. The problem was something to do with eBPF, and one of the workarounds was to tell falcon sensor to use kernel mode and not auto or user (bpf) mode. We don't allow automatic updates on hosts, however, so thankfully this was contained, but it certainly begs the question of just what testing Crowdstrike are doing.

egorfine 2y ago

> CrowdStrike should prioritize rigorous testing across all supported They should not. Testing costs money and they aren't selling their product to a company that needs or wants it on a competitive market. Their business model is based on shoving the product down the throat of enterprises due to compliance and therefore they have zero incentive to invest any money into quality.