10 ms·

It's not just CrowdStrike – the cyber sector is vulnerable

123 points by jmsflknr 2y ago

mrkramer 2y ago
Don't do automatic updates....roll updates manually. That would be a nice thing for the beginning.
- ygjb 2y ago
  It's easy to make comments like this against automatic updates, but then you get popped because something that would have been automatically updated misses a patch because it was too critical to risk automatic updates. In practice, failing closed (or crashing) is probably fine for most businesses, and lower cost than a breach, but the correct solution is automated testing across a broad spectrum of devices, staged and rolling updates to prevent entire fleets going down at once, and ensuring that there is an effective, tested rollback mechanism. But that shit's expensive, so shrug :/
  - dataflow 2y ago
    > but then you get popped because something that would have been automatically updated misses a patch because it was too critical to risk automatic updates. This kind of logic only works if you ignore any kind of possible nuances in the problem and just insist on throwing the baby out with the bathwater. Just because someone let you do automatic updates (or let's be real, you probably didn't give them much of an option) that doesn't mean you should use it for everything. Automatic update of data (like virus definitions) != automatic update of code (like kernel driver) And really, the only time you could justify doing automatic updates on other people's machines is when have reason to believe the risk of waiting for the user to get around to it is larger than the damage you might do in the process... which doesn't seem to have been the case here.
    - lucianbr 2y ago
      From what I read they automatically updated data. But the pre-existing code had a bug, which crashed on reading the updated data. Even if this is not what happened, it is possible, and shows the data/code update separation does not prevent problems.
      - dataflow 2y ago
        > shows the data/code update separation does not prevent problems. Sure they do? This is like saying seatbelts don't prevent injuries because people still die even while wearing them. I never said that one weird trick would solve every problem, or even this particular one for that matter. What I was saying was that if you look for ways to add nuance... you can find better solutions than if you throw the baby out with the bathwater. I just gave two examples of how you could do that in this problem space. That doesn't mean those are the only two things you can do, or that either would've single handedly solved this problem. The problem in your scenario is that kernel mode behavior is being auto updated globally (via data or code is irrelevant), and that should require a damn high bar. You don't do it just because you can. There's got to be a lower bar for user mode updates than kernel, etc.
        lucianbr 2y ago
        That's definitely what I meant - just because you wear a seatbelt does not mean it is now impossible to get hurt. You still need to drive carefully. To me, it looks like these people relied on the safety of seatbelts and drove really fast, and there was, predictably, a horrible crash with a lot of damage. Crowdstrike themselves seem to have missed the nuance.
    - ygjb 2y ago
      > This kind of logic only works if you ignore any kind of possible nuances in the problem and just insist on throwing the baby out with the bathwater. Just because someone let you do automatic updates (or let's be real, you probably didn't give them much or an option) that doesn't mean you should do use for everything. Oh, I agree - automatic updates are nuanced in many cases. Generally speaking, automatic updates are a good thing, but they offer trade-offs; the main trade-off is rapidly receiving security updates, at the risk of encountering new features, which can include new bugs. This is kind of a big reason why folks who buy systems should be requiring that updates offer a distinction between Security/Long Term Support, and Feature updates. It allows the person who buys the product to make an effective decision about the level of risk they want to assume from those updates. > Automatic update of data (like virus definitions) != automatic update of code (like kernel driver) Yep, absolutely, except for the case where the virus definitions (or security checks) are written in a language that gets interpreted in a kernel driver, presumably in languages that don't necessarily have memory safety guarantees. It really depends on how the security technology implements it's checks, and the facilities that the operating system provides for instrumentation and monitoring.
- winternett 2y ago
  Testing in software dev is taken completely for granted by many companies on mission-critical updates... Back in the day, our deployments would get tested on configs new & old and with several different variables, we always made sure deployments went smoothly. Now it seems as if most of these companies hire junior devs and skip testing to cut cost and then just put the blame of failure all on them. Burnout levels are high in these settings. This whole incident would have not happened if just a basic deployment test was conducted. It's so widespread, it would have been impossible to miss detecting the issue.
- rs999gti 2y ago
  Can auto updates be turned off on the crowdstrike falcon client?
- Johnny555 2y ago
  In any sizable organization, you can't get around automatic updates. But updates should be rolled out slowly and you need enough telemetry to detect problems as it's rolled out. Reboots, crashes, cpu/memory use, end user reports, etc should all be used to detect issues and pause the rollout.
- HdS84 2y ago
  Then you end up like some of our customers with log4j. We are consultants and notice that a cave for log4j comes out. We inform our customers that we have detected an issue under active exploit, and we performed an update to non vulnerable versions and want to deploy. Customer waffles for days and gets exploited before he decides to upgrade. Threats are often only minutes away. We are currently away to slow and manual updates are slowing you down even more.
  - galdosdi 2y ago
    So do automated canarying! Deploy to 1000 machines, wait 60 seconds, deploy to 10,000, etc. All done in under 5 minutes while automatically rolling back or at least halting if any metrics look bad. Canarying is by now not a very new practice. This is like doctors not washing their hands.
udev4096 2y ago
https://archive.ph/sKcp3 https://archive.ph/sKcp3
cs702 2y ago
For most corporations, security and robustness are -- and for a long time have been -- an afterthought. Making systems hard to hack and robust to rare events: * is really hard, * costs a lot of money, and * reduces earnings in the short term. Faced with these inconvenient facts, many executives who want to see stock prices go up prioritize... other things.
- hypeatei 2y ago
  To them, they are thinking about it though. They installed this ultra secure thing called CrowdStrike that checked a regulatory box for cybersecurity
  - cs702 2y ago
    CrowdStrike is a textbook example of a single point of failure: https://en.wikipedia.org/wiki/Single_point_of_failure https://en.wikipedia.org/wiki/Single_point_of_failure
- prasadjoglekar 2y ago
  To be fair, most corporations signed up for Crowdstrike as a way to address some issues. I'm sure it wasn't cheap and CS was probably better at security than an IT admin at a 50 person shop.
  - cs702 2y ago
    Yeah, it wasn't cheap. It's still not enough.
  - Damogran6 2y ago
    Much like the RSA attack, now we get to see how Crowdstrike handles damage control.
  - croes 2y ago
    But what's worse, hundreds of maybe insecure companies or creating a big single point of failure?
monero-xmr 2y ago
The real financial problem is that cybersecurity is mostly box checking. It's an industry that is open to commoditization, as startups in lower-cost global regions manage to check the box as well as the next-most-expensive region, and cost conscious companies keep migrating. But the power of the box checking is strong. I do not invest in cybersecurity companies, it is very risky IMO
- mrkramer 2y ago
  The problem with cybersecurity is that there are hundreds of attack vectors; you can get pwned by supply chain attack or by some random zero-day exploit or by an insider....It is literally impossible to 100% prevent breaching of your computer network.
  - lucianbr 2y ago
    It is impossible to write bug-free/exploit-free code. But some companies are using this as an excuse to not care about the chances of an exploit at all, and just write code in a cheaper way. We need a middle ground, where there is at least a reasonable effort towards security.
    - StillBored 2y ago
      "It is impossible to write bug-free/exploit-free code." Right, and this should be the single deciding factor for most system programming and core infrastructure development. One doesn't throw away 20+-year-old battle-tested code simply because it's grown ugly bug fixes for edge conditions no one wants to worry about. The idea that it's possible to throw away, say 30-year-old font rendering code and replace it without revisiting a lot of the problems along the way is peak hubris. And the same goes for choosing and building internal IT systems, KISS should rule those choices because each layer adds additional code, additional updating, etc. Monolithic general-purpose software is not only a waste of resources (having software that 9/10th is just taking up disk/memory/cache space because only 10% of its features are used), but it's a maintenance and security nightmare. This is the problem with much of the open-source world, too. Having 20 different Linux filesystem drivers or whatever is just adding code that will contain bugs, exploits, and a monthly kernel update containing 80 KLOC of changes is just asking for problems. Faster processes, updates, and development velocity in projects that were "solved" decades ago are just a playground for bad actors. So, to go back to Andrew Tanenbaum and many others, no one in their right mind should be writing or using OSs and software that aren't built from first principles with clearly defined modularity and security boundaries. A disk driver update should be 100% separate and compatible with not just the latest OS kernel but ones from 10+ years ago. A database update shouldn't require the latest version of python "just because". Most software is garbage quality written by a bunch of people who are all convinced they are better than their peers. And yet another code review, or CI loop, isn't going to solve this, although it might stop a maintainer from throwing poorly tested code over the fence instead of subjecting it to the same levels of scrutiny they give 3rd party contributors.
lukev 2y ago
It's rapidly getting to the point where the cure is worse than the disease, when it comes to this kind of product.
- croes 2y ago
  It isn't a cure in the first place it just fights the symptoms.
  - type0 2y ago
    it's a kind of palliative care if you extend the analogy
- deleted 2y ago
  [deleted]
Damogran6 2y ago
It's almost as if we're seeing the downsides to our cloud based decisions. Uncontrolled costs, lack of visibility, placing control of critical processes in the hands of other groups...that also have control of critical processes globally. Am I bitter at losing the business decisions that push ease of management by sending control to service providers? Not really. It's been dozens of times, and I lose every time. I can raise the concerns to make sure the decisions are educated ones, and then let the decisions be made.
- altdataseller 2y ago
  If you managed your own servers, you would still need some sort of endpoint security solution too tho…
  - Damogran6 2y ago
    Not saying it outages wouldn't happen, saying it might not happen on a _global_ scale....and it's not the only downside. There are pros and cons to each solution.
Buttons840 2y ago
Cyber Security is a matter of national security, but currently we sacrifice our national security for the convenience of companies. The disconnect is that companies are both (1) the only entity in control of their system and how it is tested and (2) not liable if a security breach does happen. I believe we need to enable red teams (security researchers) to test the security of any system, with or without permission, so long as they report responsibly and avoid obviously destructive behavior such as sustained DDoS attacks. A branch of the government, possibly of the military (the Space Force?) could constantly be trying to hack the most important systems in our nation (individuals and private companies too). The bad guys are doing this anyway, but hopefully the good guys could find the security holes first and report them responsibly. Again, currently this doesn't happen because it would be embarrassing and inconvenient for powerful companies. We threaten researchers who do nothing more than press F12 (view HTML source) with jail time and then have our best surprised Pikachu faces ready for when half the nations data is stolen every week or major systems go down. Actually, we don't make faces at all, half the nation's data is stolen every week--no, actually we don't even take notice, we just accept it as the way things have to be. Because, after all, we can't expect companies to be liable, but we can trust companies to have exclusive control over the testing of their security. How convenient for them.
- frenchy 2y ago
  Isn't this what the NSA is for? Also, I think we have plenty of reason to believe they regularly try to penetrate powerful companies, they just don't necessarily tell us when they do.
  - Buttons840 2y ago
    I've never heard anything about the NSA telling a company they have a security vulnerability. Have you?
    - orr94 2y ago
      Not the NSA, but I know of at least one time the FBI did: https://arstechnica.com/security/2024/01/chinese-malware-removed-from-soho-routers-after-fbi-issues-covert-commands/ https://arstechnica.com/security/2024/01/chinese-malware-rem...
freitzkriesler2 2y ago
Cyber was a 90s buzz word that died out and became vogue when cyber security became cool. I cringe every time I hear it drop.
- jabroni_salad 2y ago
  government and military loves that word and will probably never let it go.
  - whycome 2y ago
    Air Force. Space Force. ...Cyber Force? Inevitable.
    - jabroni_salad 2y ago
      We do actually have US CYBER COMMAND but it isn't a branch of the military, it's just a unit inside the DoD.
  - freitzkriesler2 2y ago
    I roasted some Booz Allen booth people when they asked me about cyber with a," the 90s called and wants its buzz word back." The look they gave was priceless.
- bluedino 2y ago
  I still think of how 'cyber' was used in AOL Chatrooms back in the 90's...
  - whycome 2y ago
    I forgot about that until this mention. Definitely not relegated to AOL chats.
- jijijijij 2y ago
  "Cyber sector" is reeeeally pushing it. Full body cringe.
PreInternet01 2y ago
The "cyber sector" is... awful? Nah... irresponsible? Nah... immature? Yeah, probably! Right now, pretty much everyone is looking to outsource their "security" to a single vendor, disregarding the fact that security is not a product, but a process. That... won't change! And incumbents will get less-awful about their impact on "protected" systems. And yet, there's an opportunity here! Do you truly understand Windows? And whatever happens on that platform? And how to monitor that activity for adverse actions? Without taking down your customers on a regular/observable basis? Step right up! There are a lot of incumbents facing imminent replacement...
- silverquiet 2y ago
  There can't be a person alive who "truly understands" Windows; though made by humans (allegedly), any modern OS is going to be beyond the understanding of any individual. This is the fundamental problem of managing modern systems.
  - hypeatei 2y ago
    Yes, we're in a complexity crisis.
  - nick__m 2y ago
    I can name two: Raymond Chen and Mark Russinovich ! I don't know if Mark is still up-to-date into the latest Windows internals now that he is the CTO of Azure but Mr Chen sure is.
    - silverquiet 2y ago
      I was thinking Linus for Linux, but in spite of how talented these people are, it's still hard to imagine them having a detailed grasp of the entire codebase at this point.
      - pxc 2y ago
        Linus would definitely defer to lieutenants when it comes to understanding the specifics of code in specialized parts of the kernel. Even these experts would probably want to consult with others instead of trying to just grow what's going on on their own.
hcfman 2y ago
Probably get a lot more of this when the full force of the cyber resilience act kicks in.
ramesh31 2y ago
I remember being proud of the fact that I had an intimate knowledge and understanding of every single process running on my dev machine. Things felt sane. I could fully comprehend what was happening on my system at all times. Then the button pusher configurator class got called a new name, "DevOps" and started pushing all this crap on us. I'm ready to just start doing work on a private machine at this point.
blibble 2y ago
they're correct, all the others are similarly shit sentinelone, tanium, guardicore, defender endpoint, delina all running as root (or worse), sucking up absurd amounts of resources, often more than the software running on the machine (but advertised as "LOW IMPACT") they also cause reliable software to break due to bugs in e.g. their EBPF also often serialises all network and disk on the machine through to one single thread (so much for multi-queue NVMe/NICs) the risk and compliance attitude that results in this corporate mandated malware being required needs to go this software creates more risk than it prevents
- altdataseller 2y ago
  So whats the alternative? Have no endpoint protection? Have nothing in place to warn you when malware ends up in your system? (Just playing devils advocate. I hate Crowdstrike as much as anyone here :)
  - iwwr 2y ago
    Or maybe switch to an operating system that isn't a security dumpster fire?
    - altdataseller 2y ago
      And what if this bug happened to affect Linux somehow too? What then?
    - esafak 2y ago
      How do you objectively assess an operating system's security? I wanted to convince friends that Windows is insecure but I couldn't find unassailable evidence. Got some? There are confounding variables like the age of the operating system and size of the userbase (distorting the event volume), its attractiveness to attackers, and the tendency of organizations of different levels of technical ability to prefer different operating systems...
      - freedomben 2y ago
UweSchmidt 2y ago
So how do you actually cybersecure a company in a compliant and practical way?
- rawgabbit 2y ago
  Today, we comply by ticking all the boxes in a checklist; it takes care of the most obvious hacks. Is it good enough? Today, we got our answer. Practically speaking, that is all the end-user can do with Windows machines. My point is Windows is fundamentally unsecure. It is a dike with thousands of holes some of which are not even visible to Microsoft themselves. The reason of that is security has been after-thought. It is band aids/plasters put on top of other plasters.
  - UweSchmidt 2y ago
    Because security experts seem to have this slight dismissive attitude about companies' and individuals' attempts to do security, while not usually having answers or providing secure systems.
    - bostik 2y ago
      Sadly most so-called security experts are not hands-on professionals but hands-off "cybersecurity persons". They do not do any real work themselves, they only generate useless busywork for others. There are people in that category who are not hands-on themselves but still have sufficiently deep understanding of technical details. But as one might guess, they are about as common as four-leaf clovers.
    - type0 2y ago
      windows security experts are as useful as astrologists, but astrologists are cheaper
- jabroni_salad 2y ago
  I firmly believe that most routine security issues are really just operations issues and vulns are just bugs and security largely doesnt need to be its own category at all. I know everybody hates the C-word but if I look at 27001 requirements or the CIS benchmarks, there is nothing in there that I do not want for myself. If you can keep a list of the products and services you are running, have actually put the time into implementing it correctly, and have an ongoing maintenance plan then you are probably in the top 1% of networks.
ehPReth 2y ago
I've been trialling application allowlisting, but wow is it ever frustrating. So much stuff isn't signed, and when it is the accompanying DLLs aren't. or the signature is invalid. or some of Windows' own executables/dlls aren't signed (why?? you make applocker??) or the installer is, but none of the actual resultant end files Is it just me?
- Avamander 2y ago
  It's not just you. Windows software management sucks and people just find excuses for it. WDAC is really difficult to use directly because of it. OEM software is usually the worst offender here, all these installers and support utilities should be fully abolished. Drivers should stop loading if they aren't coming from Windows Update and haven't passed some quality control.
bluedino 2y ago
Security: You must install Microsoft Defender on all Linux VM's Devs: Ugh...why? Security: For safety! Devs: Fine, we won't argue. Deploy it if you may. A few moments later... Devs: All of our VM's are slow as crap! Defender is using 100% of the CPU! Security: Add another core to your VM's. ticket closed Management: Why are our developers up 30% on their cloud spend!?
- lanstin 2y ago
  Even worse, at least aesthetically, than the AV is the pacct stuff to log 1 line for every single syscall. Talk about sublinear scaling. And for what, no one can say. Actually, even worse than that was we had to install AV on all the images that the ephemeral map-reduce/Hadoop clusters we spun up, but the way the AV stuff worked, the computes were gone by the time the registration for the new compute had gone thru whole process. And, in AWS accounts where there were maybe 63 IPs and say 600 EC2s/day they used IP as the primary key for the "list of compute in the VPC." So they stitched together totally unrelated stuff as if it was the same continuous compute. I guess it would be eventually fixed, but the bad security data was not a real concern of the devops team that was building out stuff as rapidly as possible, nor were the EC2 made in a sealed off VPC and only lived for a few tens of minutes or hours at best a serious security concern to the actual security people. Just a check list solution hitting a novel environment.
- blibble 2y ago
  and all microsoft have to do to increase cloud revenue is make defender chew up 5% more CPU every few months directly incentivised to make shit software
  - 01HNNWZ0MV43FF 2y ago
    To a point. After a while they're wasting hardware and another vendor will undercut them. After some decades people will start leaving
    - sebazzz 2y ago
      That is very long term. Not interesting to shareholders.
guru4consulting 2y ago
For a business that relies on SaaS applications over cloud and uses dumb machines (windows, iPad, whatever) as client terminals, can someone please explain what are the actual threat factors that these EDR tools like Crowdstrike Falcon address? And if SaaS applications can restrict access, detect anomalies with user behavior, have MFA for auth, etc.. will that mitigate these risks? I guess common issues like key loggers, malwares, virus attacks have much simpler solutions than a complex EDR which seems to need root access!! Someone, please educate.
mikewarot 2y ago
We build our "cyber fortress" out of the Turing Complete analog of Crates of C4... and wonder why things go wrong all the time. As I say every time this happens (and it will keep happening for the next decade or so)... Ambient Authority systems can't be secured, we need to switch to Operating Systems designed around Capability Based Security. We need at least 2 of them, from competing projects.
- deleted 2y ago
  [deleted]
1vuio0pswjnm7 2y ago
Works where archive.ph is blocked: https://webcache.googleusercontent.com/search?q=cache:https://www.ft.com/content/d2f529fe-3050-4ef5-a8c7-6487d533e57a https://webcache.googleusercontent.com/search?q=cache:https:...
exabrial 2y ago
Honestly until we can get rid of the perception that SOC2/Sables-Oxly/HiTrust provides any meaningful security we’re stuck.
deleted 2y ago
[deleted]
1vuio0pswjnm7 2y ago
Works where archive.ph is blocked: https://cc.bingj.com/cache.aspx?d=464600016483&w=i7yXBm7GwofOVaoKk5qaOqy633_CL224 https://cc.bingj.com/cache.aspx?d=464600016483&w=i7yXBm7Gwof...
sebazzz 2y ago
Centralisation is really the core of the problem here. Take ZScaler, which is a service that proxies all network connections of a computer to a central cloud proxy server, mitms it (decrypt, inspect, log, and encrypt), and then forwards it to the target server. Imagine that this is hacked, and this isn't immediately discovered. Hackers listening in and being able to tap off cookies, bearer tokens and other confidential information for weeks. That would affect so many companies. And if they would want to cause a DoS, many computers and servers would be left without an operational internet connection.
- sconi 2y ago
  Yes, Zscaler or any other Zscaler clone (e.g., Netskope, Cato, etc) -- they're all just sitting ducks, and once they are compromised, what happens to all the customers? It doesn't make any sense and shows how much we're willing to give up for convenience.
craxdevil7 2y ago
thats true, it is disheartening when security controls are only seen as a checklist to comply with some framework, and not actually implemented. https://medium.com/@confusedcyberwarrior/what-is-soc2-how-to-do-it-wrong-c72d7374fc31 https://medium.com/@confusedcyberwarrior/what-is-soc2-how-to... This gives a false sense of security, which is further bad for cyber space. Crowdstrike incident on other hand shows that how we still have single points of failure on our supposedly secure and safe systems. https://medium.com/@confusedcyberwarrior/when-security-becomes-the-threat-the-crowdstrike-incident-9bbaeab9db9d https://medium.com/@confusedcyberwarrior/when-security-becom...

123 points by jmsflknr 2y ago

mrkramer 2y ago

Don't do automatic updates....roll updates manually. That would be a nice thing for the beginning.

ygjb 2y ago
It's easy to make comments like this against automatic updates, but then you get popped because something that would have been automatically updated misses a patch because it was too critical to risk automatic updates. In practice, failing closed (or crashing) is probably fine for most businesses, and lower cost than a breach, but the correct solution is automated testing across a broad spectrum of devices, staged and rolling updates to prevent entire fleets going down at once, and ensuring that there is an effective, tested rollback mechanism. But that shit's expensive, so shrug :/
- dataflow 2y ago
  > but then you get popped because something that would have been automatically updated misses a patch because it was too critical to risk automatic updates. This kind of logic only works if you ignore any kind of possible nuances in the problem and just insist on throwing the baby out with the bathwater. Just because someone let you do automatic updates (or let's be real, you probably didn't give them much of an option) that doesn't mean you should use it for everything. Automatic update of data (like virus definitions) != automatic update of code (like kernel driver) And really, the only time you could justify doing automatic updates on other people's machines is when have reason to believe the risk of waiting for the user to get around to it is larger than the damage you might do in the process... which doesn't seem to have been the case here.
  - lucianbr 2y ago
    From what I read they automatically updated data. But the pre-existing code had a bug, which crashed on reading the updated data. Even if this is not what happened, it is possible, and shows the data/code update separation does not prevent problems.
    - dataflow 2y ago
      > shows the data/code update separation does not prevent problems. Sure they do? This is like saying seatbelts don't prevent injuries because people still die even while wearing them. I never said that one weird trick would solve every problem, or even this particular one for that matter. What I was saying was that if you look for ways to add nuance... you can find better solutions than if you throw the baby out with the bathwater. I just gave two examples of how you could do that in this problem space. That doesn't mean those are the only two things you can do, or that either would've single handedly solved this problem. The problem in your scenario is that kernel mode behavior is being auto updated globally (via data or code is irrelevant), and that should require a damn high bar. You don't do it just because you can. There's got to be a lower bar for user mode updates than kernel, etc.
      - lucianbr 2y ago
        That's definitely what I meant - just because you wear a seatbelt does not mean it is now impossible to get hurt. You still need to drive carefully. To me, it looks like these people relied on the safety of seatbelts and drove really fast, and there was, predictably, a horrible crash with a lot of damage. Crowdstrike themselves seem to have missed the nuance.
  - ygjb 2y ago
    > This kind of logic only works if you ignore any kind of possible nuances in the problem and just insist on throwing the baby out with the bathwater. Just because someone let you do automatic updates (or let's be real, you probably didn't give them much or an option) that doesn't mean you should do use for everything. Oh, I agree - automatic updates are nuanced in many cases. Generally speaking, automatic updates are a good thing, but they offer trade-offs; the main trade-off is rapidly receiving security updates, at the risk of encountering new features, which can include new bugs. This is kind of a big reason why folks who buy systems should be requiring that updates offer a distinction between Security/Long Term Support, and Feature updates. It allows the person who buys the product to make an effective decision about the level of risk they want to assume from those updates. > Automatic update of data (like virus definitions) != automatic update of code (like kernel driver) Yep, absolutely, except for the case where the virus definitions (or security checks) are written in a language that gets interpreted in a kernel driver, presumably in languages that don't necessarily have memory safety guarantees. It really depends on how the security technology implements it's checks, and the facilities that the operating system provides for instrumentation and monitoring.
winternett 2y ago
Testing in software dev is taken completely for granted by many companies on mission-critical updates... Back in the day, our deployments would get tested on configs new & old and with several different variables, we always made sure deployments went smoothly. Now it seems as if most of these companies hire junior devs and skip testing to cut cost and then just put the blame of failure all on them. Burnout levels are high in these settings. This whole incident would have not happened if just a basic deployment test was conducted. It's so widespread, it would have been impossible to miss detecting the issue.
rs999gti 2y ago
Can auto updates be turned off on the crowdstrike falcon client?
Johnny555 2y ago
In any sizable organization, you can't get around automatic updates. But updates should be rolled out slowly and you need enough telemetry to detect problems as it's rolled out. Reboots, crashes, cpu/memory use, end user reports, etc should all be used to detect issues and pause the rollout.
HdS84 2y ago
Then you end up like some of our customers with log4j. We are consultants and notice that a cave for log4j comes out. We inform our customers that we have detected an issue under active exploit, and we performed an update to non vulnerable versions and want to deploy. Customer waffles for days and gets exploited before he decides to upgrade. Threats are often only minutes away. We are currently away to slow and manual updates are slowing you down even more.
- galdosdi 2y ago
  So do automated canarying! Deploy to 1000 machines, wait 60 seconds, deploy to 10,000, etc. All done in under 5 minutes while automatically rolling back or at least halting if any metrics look bad. Canarying is by now not a very new practice. This is like doctors not washing their hands.

udev4096 2y ago

https://archive.ph/sKcp3 https://archive.ph/sKcp3

cs702 2y ago

For most corporations, security and robustness are -- and for a long time have been -- an afterthought. Making systems hard to hack and robust to rare events: * is really hard, * costs a lot of money, and * reduces earnings in the short term. Faced with these inconvenient facts, many executives who want to see stock prices go up prioritize... other things.

hypeatei 2y ago
To them, they are thinking about it though. They installed this ultra secure thing called CrowdStrike that checked a regulatory box for cybersecurity
- cs702 2y ago
  CrowdStrike is a textbook example of a single point of failure: https://en.wikipedia.org/wiki/Single_point_of_failure https://en.wikipedia.org/wiki/Single_point_of_failure
prasadjoglekar 2y ago
To be fair, most corporations signed up for Crowdstrike as a way to address some issues. I'm sure it wasn't cheap and CS was probably better at security than an IT admin at a 50 person shop.
- cs702 2y ago
  Yeah, it wasn't cheap. It's still not enough.
- Damogran6 2y ago
  Much like the RSA attack, now we get to see how Crowdstrike handles damage control.
- croes 2y ago
  But what's worse, hundreds of maybe insecure companies or creating a big single point of failure?

monero-xmr 2y ago

The real financial problem is that cybersecurity is mostly box checking. It's an industry that is open to commoditization, as startups in lower-cost global regions manage to check the box as well as the next-most-expensive region, and cost conscious companies keep migrating. But the power of the box checking is strong. I do not invest in cybersecurity companies, it is very risky IMO

mrkramer 2y ago
The problem with cybersecurity is that there are hundreds of attack vectors; you can get pwned by supply chain attack or by some random zero-day exploit or by an insider....It is literally impossible to 100% prevent breaching of your computer network.
- lucianbr 2y ago
  It is impossible to write bug-free/exploit-free code. But some companies are using this as an excuse to not care about the chances of an exploit at all, and just write code in a cheaper way. We need a middle ground, where there is at least a reasonable effort towards security.
  - StillBored 2y ago
    "It is impossible to write bug-free/exploit-free code." Right, and this should be the single deciding factor for most system programming and core infrastructure development. One doesn't throw away 20+-year-old battle-tested code simply because it's grown ugly bug fixes for edge conditions no one wants to worry about. The idea that it's possible to throw away, say 30-year-old font rendering code and replace it without revisiting a lot of the problems along the way is peak hubris. And the same goes for choosing and building internal IT systems, KISS should rule those choices because each layer adds additional code, additional updating, etc. Monolithic general-purpose software is not only a waste of resources (having software that 9/10th is just taking up disk/memory/cache space because only 10% of its features are used), but it's a maintenance and security nightmare. This is the problem with much of the open-source world, too. Having 20 different Linux filesystem drivers or whatever is just adding code that will contain bugs, exploits, and a monthly kernel update containing 80 KLOC of changes is just asking for problems. Faster processes, updates, and development velocity in projects that were "solved" decades ago are just a playground for bad actors. So, to go back to Andrew Tanenbaum and many others, no one in their right mind should be writing or using OSs and software that aren't built from first principles with clearly defined modularity and security boundaries. A disk driver update should be 100% separate and compatible with not just the latest OS kernel but ones from 10+ years ago. A database update shouldn't require the latest version of python "just because". Most software is garbage quality written by a bunch of people who are all convinced they are better than their peers. And yet another code review, or CI loop, isn't going to solve this, although it might stop a maintainer from throwing poorly tested code over the fence instead of subjecting it to the same levels of scrutiny they give 3rd party contributors.

lukev 2y ago

It's rapidly getting to the point where the cure is worse than the disease, when it comes to this kind of product.

croes 2y ago
It isn't a cure in the first place it just fights the symptoms.
- type0 2y ago
  it's a kind of palliative care if you extend the analogy
deleted 2y ago
[deleted]

Damogran6 2y ago

It's almost as if we're seeing the downsides to our cloud based decisions. Uncontrolled costs, lack of visibility, placing control of critical processes in the hands of other groups...that also have control of critical processes globally. Am I bitter at losing the business decisions that push ease of management by sending control to service providers? Not really. It's been dozens of times, and I lose every time. I can raise the concerns to make sure the decisions are educated ones, and then let the decisions be made.

altdataseller 2y ago
If you managed your own servers, you would still need some sort of endpoint security solution too tho…
- Damogran6 2y ago
  Not saying it outages wouldn't happen, saying it might not happen on a _global_ scale....and it's not the only downside. There are pros and cons to each solution.

Buttons840 2y ago

Cyber Security is a matter of national security, but currently we sacrifice our national security for the convenience of companies. The disconnect is that companies are both (1) the only entity in control of their system and how it is tested and (2) not liable if a security breach does happen. I believe we need to enable red teams (security researchers) to test the security of any system, with or without permission, so long as they report responsibly and avoid obviously destructive behavior such as sustained DDoS attacks. A branch of the government, possibly of the military (the Space Force?) could constantly be trying to hack the most important systems in our nation (individuals and private companies too). The bad guys are doing this anyway, but hopefully the good guys could find the security holes first and report them responsibly. Again, currently this doesn't happen because it would be embarrassing and inconvenient for powerful companies. We threaten researchers who do nothing more than press F12 (view HTML source) with jail time and then have our best surprised Pikachu faces ready for when half the nations data is stolen every week or major systems go down. Actually, we don't make faces at all, half the nation's data is stolen every week--no, actually we don't even take notice, we just accept it as the way things have to be. Because, after all, we can't expect companies to be liable, but we can trust companies to have exclusive control over the testing of their security. How convenient for them.

frenchy 2y ago
Isn't this what the NSA is for? Also, I think we have plenty of reason to believe they regularly try to penetrate powerful companies, they just don't necessarily tell us when they do.
- Buttons840 2y ago
  I've never heard anything about the NSA telling a company they have a security vulnerability. Have you?
  - orr94 2y ago
    Not the NSA, but I know of at least one time the FBI did: https://arstechnica.com/security/2024/01/chinese-malware-removed-from-soho-routers-after-fbi-issues-covert-commands/ https://arstechnica.com/security/2024/01/chinese-malware-rem...

freitzkriesler2 2y ago

Cyber was a 90s buzz word that died out and became vogue when cyber security became cool. I cringe every time I hear it drop.

jabroni_salad 2y ago
government and military loves that word and will probably never let it go.
- whycome 2y ago
  Air Force. Space Force. ...Cyber Force? Inevitable.
  - jabroni_salad 2y ago
    We do actually have US CYBER COMMAND but it isn't a branch of the military, it's just a unit inside the DoD.
- freitzkriesler2 2y ago
  I roasted some Booz Allen booth people when they asked me about cyber with a," the 90s called and wants its buzz word back." The look they gave was priceless.
bluedino 2y ago
I still think of how 'cyber' was used in AOL Chatrooms back in the 90's...
- whycome 2y ago
  I forgot about that until this mention. Definitely not relegated to AOL chats.
jijijijij 2y ago
"Cyber sector" is reeeeally pushing it. Full body cringe.

PreInternet01 2y ago

The "cyber sector" is... awful? Nah... irresponsible? Nah... immature? Yeah, probably! Right now, pretty much everyone is looking to outsource their "security" to a single vendor, disregarding the fact that security is not a product, but a process. That... won't change! And incumbents will get less-awful about their impact on "protected" systems. And yet, there's an opportunity here! Do you truly understand Windows? And whatever happens on that platform? And how to monitor that activity for adverse actions? Without taking down your customers on a regular/observable basis? Step right up! There are a lot of incumbents facing imminent replacement...

silverquiet 2y ago
There can't be a person alive who "truly understands" Windows; though made by humans (allegedly), any modern OS is going to be beyond the understanding of any individual. This is the fundamental problem of managing modern systems.
- hypeatei 2y ago
  Yes, we're in a complexity crisis.
- nick__m 2y ago
  I can name two: Raymond Chen and Mark Russinovich ! I don't know if Mark is still up-to-date into the latest Windows internals now that he is the CTO of Azure but Mr Chen sure is.
  - silverquiet 2y ago
    I was thinking Linus for Linux, but in spite of how talented these people are, it's still hard to imagine them having a detailed grasp of the entire codebase at this point.
    - pxc 2y ago
      Linus would definitely defer to lieutenants when it comes to understanding the specifics of code in specialized parts of the kernel. Even these experts would probably want to consult with others instead of trying to just grow what's going on on their own.

hcfman 2y ago

Probably get a lot more of this when the full force of the cyber resilience act kicks in.

ramesh31 2y ago

I remember being proud of the fact that I had an intimate knowledge and understanding of every single process running on my dev machine. Things felt sane. I could fully comprehend what was happening on my system at all times. Then the button pusher configurator class got called a new name, "DevOps" and started pushing all this crap on us. I'm ready to just start doing work on a private machine at this point.

blibble 2y ago

they're correct, all the others are similarly shit sentinelone, tanium, guardicore, defender endpoint, delina all running as root (or worse), sucking up absurd amounts of resources, often more than the software running on the machine (but advertised as "LOW IMPACT") they also cause reliable software to break due to bugs in e.g. their EBPF also often serialises all network and disk on the machine through to one single thread (so much for multi-queue NVMe/NICs) the risk and compliance attitude that results in this corporate mandated malware being required needs to go this software creates more risk than it prevents

altdataseller 2y ago
So whats the alternative? Have no endpoint protection? Have nothing in place to warn you when malware ends up in your system? (Just playing devils advocate. I hate Crowdstrike as much as anyone here :)
- iwwr 2y ago
  Or maybe switch to an operating system that isn't a security dumpster fire?
  - altdataseller 2y ago
    And what if this bug happened to affect Linux somehow too? What then?
  - esafak 2y ago
    How do you objectively assess an operating system's security? I wanted to convince friends that Windows is insecure but I couldn't find unassailable evidence. Got some? There are confounding variables like the age of the operating system and size of the userbase (distorting the event volume), its attractiveness to attackers, and the tendency of organizations of different levels of technical ability to prefer different operating systems...
    - freedomben 2y ago

UweSchmidt 2y ago

So how do you actually cybersecure a company in a compliant and practical way?

rawgabbit 2y ago
Today, we comply by ticking all the boxes in a checklist; it takes care of the most obvious hacks. Is it good enough? Today, we got our answer. Practically speaking, that is all the end-user can do with Windows machines. My point is Windows is fundamentally unsecure. It is a dike with thousands of holes some of which are not even visible to Microsoft themselves. The reason of that is security has been after-thought. It is band aids/plasters put on top of other plasters.
- UweSchmidt 2y ago
  Because security experts seem to have this slight dismissive attitude about companies' and individuals' attempts to do security, while not usually having answers or providing secure systems.
  - bostik 2y ago
    Sadly most so-called security experts are not hands-on professionals but hands-off "cybersecurity persons". They do not do any real work themselves, they only generate useless busywork for others. There are people in that category who are not hands-on themselves but still have sufficiently deep understanding of technical details. But as one might guess, they are about as common as four-leaf clovers.
  - type0 2y ago
    windows security experts are as useful as astrologists, but astrologists are cheaper
jabroni_salad 2y ago
I firmly believe that most routine security issues are really just operations issues and vulns are just bugs and security largely doesnt need to be its own category at all. I know everybody hates the C-word but if I look at 27001 requirements or the CIS benchmarks, there is nothing in there that I do not want for myself. If you can keep a list of the products and services you are running, have actually put the time into implementing it correctly, and have an ongoing maintenance plan then you are probably in the top 1% of networks.

ehPReth 2y ago

I've been trialling application allowlisting, but wow is it ever frustrating. So much stuff isn't signed, and when it is the accompanying DLLs aren't. or the signature is invalid. or some of Windows' own executables/dlls aren't signed (why?? you make applocker??) or the installer is, but none of the actual resultant end files Is it just me?

Avamander 2y ago
It's not just you. Windows software management sucks and people just find excuses for it. WDAC is really difficult to use directly because of it. OEM software is usually the worst offender here, all these installers and support utilities should be fully abolished. Drivers should stop loading if they aren't coming from Windows Update and haven't passed some quality control.

bluedino 2y ago

Security: You must install Microsoft Defender on all Linux VM's Devs: Ugh...why? Security: For safety! Devs: Fine, we won't argue. Deploy it if you may. A few moments later... Devs: All of our VM's are slow as crap! Defender is using 100% of the CPU! Security: Add another core to your VM's. ticket closed Management: Why are our developers up 30% on their cloud spend!?

lanstin 2y ago
Even worse, at least aesthetically, than the AV is the pacct stuff to log 1 line for every single syscall. Talk about sublinear scaling. And for what, no one can say. Actually, even worse than that was we had to install AV on all the images that the ephemeral map-reduce/Hadoop clusters we spun up, but the way the AV stuff worked, the computes were gone by the time the registration for the new compute had gone thru whole process. And, in AWS accounts where there were maybe 63 IPs and say 600 EC2s/day they used IP as the primary key for the "list of compute in the VPC." So they stitched together totally unrelated stuff as if it was the same continuous compute. I guess it would be eventually fixed, but the bad security data was not a real concern of the devops team that was building out stuff as rapidly as possible, nor were the EC2 made in a sealed off VPC and only lived for a few tens of minutes or hours at best a serious security concern to the actual security people. Just a check list solution hitting a novel environment.
blibble 2y ago
and all microsoft have to do to increase cloud revenue is make defender chew up 5% more CPU every few months directly incentivised to make shit software
- 01HNNWZ0MV43FF 2y ago
  To a point. After a while they're wasting hardware and another vendor will undercut them. After some decades people will start leaving
  - sebazzz 2y ago
    That is very long term. Not interesting to shareholders.

guru4consulting 2y ago

For a business that relies on SaaS applications over cloud and uses dumb machines (windows, iPad, whatever) as client terminals, can someone please explain what are the actual threat factors that these EDR tools like Crowdstrike Falcon address? And if SaaS applications can restrict access, detect anomalies with user behavior, have MFA for auth, etc.. will that mitigate these risks? I guess common issues like key loggers, malwares, virus attacks have much simpler solutions than a complex EDR which seems to need root access!! Someone, please educate.

mikewarot 2y ago

We build our "cyber fortress" out of the Turing Complete analog of Crates of C4... and wonder why things go wrong all the time. As I say every time this happens (and it will keep happening for the next decade or so)... Ambient Authority systems can't be secured, we need to switch to Operating Systems designed around Capability Based Security. We need at least 2 of them, from competing projects.

deleted 2y ago
[deleted]

1vuio0pswjnm7 2y ago

Works where archive.ph is blocked: https://webcache.googleusercontent.com/search?q=cache:https://www.ft.com/content/d2f529fe-3050-4ef5-a8c7-6487d533e57a https://webcache.googleusercontent.com/search?q=cache:https:...

exabrial 2y ago

Honestly until we can get rid of the perception that SOC2/Sables-Oxly/HiTrust provides any meaningful security we’re stuck.

deleted 2y ago

[deleted]

1vuio0pswjnm7 2y ago

Works where archive.ph is blocked: https://cc.bingj.com/cache.aspx?d=464600016483&w=i7yXBm7GwofOVaoKk5qaOqy633_CL224 https://cc.bingj.com/cache.aspx?d=464600016483&w=i7yXBm7Gwof...

sebazzz 2y ago

Centralisation is really the core of the problem here. Take ZScaler, which is a service that proxies all network connections of a computer to a central cloud proxy server, mitms it (decrypt, inspect, log, and encrypt), and then forwards it to the target server. Imagine that this is hacked, and this isn't immediately discovered. Hackers listening in and being able to tap off cookies, bearer tokens and other confidential information for weeks. That would affect so many companies. And if they would want to cause a DoS, many computers and servers would be left without an operational internet connection.

sconi 2y ago
Yes, Zscaler or any other Zscaler clone (e.g., Netskope, Cato, etc) -- they're all just sitting ducks, and once they are compromised, what happens to all the customers? It doesn't make any sense and shows how much we're willing to give up for convenience.

craxdevil7 2y ago

thats true, it is disheartening when security controls are only seen as a checklist to comply with some framework, and not actually implemented. https://medium.com/@confusedcyberwarrior/what-is-soc2-how-to-do-it-wrong-c72d7374fc31 https://medium.com/@confusedcyberwarrior/what-is-soc2-how-to... This gives a false sense of security, which is further bad for cyber space. Crowdstrike incident on other hand shows that how we still have single points of failure on our supposedly secure and safe systems. https://medium.com/@confusedcyberwarrior/when-security-becomes-the-threat-the-crowdstrike-incident-9bbaeab9db9d https://medium.com/@confusedcyberwarrior/when-security-becom...