20 ms·

CrowdStrike debacle provides road map of American vulnerabilities to adversaries

279 points by jmsflknr 2y ago

bell-cot 2y ago
On the one hand - you can read this as a PSA for the apathetic and/or clueless 99.9%. On the other hand - it's d*mn hard to imagine that any of America's "A List" or "B List" adversaries didn't have a far-more-detailed road map, years ago.
- pdimitar 2y ago
  Does the last part of your comment imply that USA should just give up and accept all its adversaries already have backdoors and nothing can be done about it?
  - nonrandomstring 2y ago
    > accept all its adversaries already have backdoors This is actually a really useful hypothetical standpoint to work out security from. Designing systems that start from the assumption of insecurity helps us build more robust protocols and management. Qubes OS starts from the position that all VMs are or soon will be compromised. Zero-trust in network design assumes the bad guys already have the whole network. Plenty out there would like to shrug and say "the endpoints are all rotten too" (especially with phones which are a veritable hell to secure) and move trust into the application via trusted execution methods. > and nothing can be done about it? No, That doesn't follow. It's prudent to be realistic about threats. but there's always a way out, at a cost. The cost, in a complexity crisis, is throwing away a lot of what we've done.
    - pdimitar 2y ago
      Yeah I am completely with you and I agree. But it seems that reducing costs is more important even compared to preventing people on life support in hospitals from dying. What a world.
    - warkdarrior 2y ago
      > Qubes OS starts from the position that all VMs are or soon will be compromised. Zero-trust in network design assumes the bad guys already have the whole network. So what does Qubes OS do to protect against a hypervisor bug? Those must exist. How do you ensure that your systems are still working and retrieving data from databases, etc, when bad guys have the whole network and can block all communications?
      - nonrandomstring 2y ago
        The answer to both those questions is you can't. So you you either need to make other provisions at different levels of the stack or design your architecture to make them irrelevant to your security model.
  - bell-cot 2y ago
    No, and I've no idea where you got that from. Here's the HN Title: "CrowdStrike debacle provides road map of American vulnerabilities to adversaries" My assertion: America's serious-threat adversaries already had far more detailed road maps, years ago. The intel value of whatever "road map" data they got from the CrowdStrike debacle was pretty marginal. Neither the HN Title nor I said anything about backdoors. And within 2 para's, the NYT story makes it clear that CloudStrike's Big Oopsied had nothing to do with bad guys hacking anything.
  - marcosdumay 2y ago
    Did any large company clean their datacenters after they stopped using Solar Winds?
- red_admiral 2y ago
  I'm sure there's a few adversaries who could pull something like this off, and have 0-days ready. But if they use them, the US could see that as a hostile action and get very upset about it.
  - LtWorf 2y ago
    they can just bribe a company to do it :D
ndesaulniers 2y ago
Yeah, didn't the US just ban Kaspersky, over fears that Kaspersky could cause such an outage (among other fears)? Turns out our homegrown CrowdStrike was just as bad as our fears over Kaspersky were. Perhaps worse.
- supriyo-biswas 2y ago
  The Kaspersky issue could have been better handled by simply requiring divestment or by having requiring an US-appointed auditor to investigate produce reports to assuage such concerns; as was proposed in the case of Tiktok.
- dataflow 2y ago
  > over fears that Kaspersky could cause such an outage (among other fears)? Citation? I thought it was the "other fears"; this is the first time I'm hearing accidental outages were one of the concerns.
  - mathgeek 2y ago
    Not GP, but the decision and reasoning is at https://public-inspection.federalregister.gov/2024-13532.pdf https://public-inspection.federalregister.gov/2024-13532.pdf (I am not claiming any specific “other fears”, just linking to the source)
    - dataflow 2y ago
      Thanks. Yeah, i, ii, and iii all talk about malicious events, not accidental.
  - mardifoufs 2y ago
    Yes but any accidental outages from an entity like Kaspersky would have been considered non accidental regardless of the actual root cause. If crowdstrike was Russian, the headlines would be a bit more suspicious about yesterday's event. or if they had brought down Russian infrastructure Russia would have probably been suspicious about American involvement, even if it's just accidental.
pdimitar 2y ago
Wouldn't any memory-safe language help prevent this NULL pointer access? Why are all these crucial pieces still written in C/C++, when it's obvious to anybody keeping even remote track of CVEs that these languages are just not up to the task with today's climate of a 24/7 shadow internet war? (The one that's likely been going on for at least 25 years at this point?) When will we learn? You hate Rust -- fine (not fine but OK, I guess people get super triggered over it and it's a reality I can't change but I am still baffled by it because they throw away reason for emotions and these people should really know better). Fine. Just use Golang or any other GC language really (Java or C# as well, if you must). When will we abandon convenient routine and start adapting to modern realities? ("Modern" being at least 25-year old here but hey, I am willing to give you some leeway and not roast you too much. Let's assume these are "modern" realities, f.ex. just the last 5 years.)
- imranhou 2y ago
  Windows does not support drivers based on rust language, so perhaps may be another 5 years
  - pdimitar 2y ago
    Rust can't compile DLLs? AFAIK it can?
    - cmrdporcupine 2y ago
      This is a kernel driver. Runs in kernel space. Intercepts syscalls. You'd definitely be fighting uphill to write it in Rust. And your code would be riddled with `unsafe` by necessity anyways.
      - pdimitar 2y ago
        Fair enough, still Rust's unsafe is not dropping all of its guarantees. Quite a lot of them remain in place. Not saying you can't write bugs in Rust, of course -- that would be crazily delusional. I am saying they needed a better process. And I am saying that a stricter language could have improved the process a bit as well.
hilbert42 2y ago
"What Happened to Digital Resilience?" Was there ever such a time? If so then tell me when it was. "The latest chaos wasn’t caused by an adversary, but it provided a road map of American vulnerabilities at a critical moment." I've no doubt that road maps of American vulnerabilities are currently being planned, roadmaped and stockpiled for future use by those who aren't on the best terms with the US. In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that. Remember, this happened big-time when Microsoft introduced Windows especially Windows 95. To capture the market Microsoft made everything as easy as possible for nontechnical users—just click on something and it'd happen, things would happen with ease. And all this happened without due consideration to security. When viruses, vulnerabilities, breaches got out of hand restrictions were introduced which meant users had less freedom to do what they'd gotten used to doing. What Microsoft did was to get the world used to slack operating procedures and efforts reign this in has met with user resistance ever since. We're now stuck with a major problem that was easily foreseeable even before Microsoft launched Windows 95. Fixing it will be extremely difficult.
- dralley 2y ago
  This is an area where studying Ukraine's experience will be very useful (and probably has already been useful) There were years of cyberattacks against pretty much every peice of critical infrastructure they have. Things went down, there were disruptions, but they adapted. Sometimes by falling back to low-tech solutions, sometimes by developing new systems with robustness into new systems and purging the old (much easier to politically justify when the problem is tangible and immediate). I seem to recall that one of the first things we did when tensions started ramping up was sending teams of cyber security experts from the NSA to help them lock down and root out infiltrations.
  - emseetech 2y ago
    | Sometimes by falling back to low-tech solutions My first thought in all this was wondering if there's a business opportunity for a consulting firm or startup that designs and manages offline paper backup systems that can quickly and seamlessly integrate back with digital systems once they come back online.
encoderer 2y ago
Just told my family yesterday that if we are ever in a real war expect everything to stop working within 8 hours. We will go back to cash and paperwork but it will be painful and slow.
- newzisforsukas 2y ago
  Just storm EDR company offices slap guns to devs' heads, push geofenced destruction.
- makeitdouble 2y ago
  Looking at two countries in an actual long running war, both kept using cashless means, with actually increases in usage: https://cbr.ru/eng/press/event/?id=18776 https://cbr.ru/eng/press/event/?id=18776 https://bank.gov.ua/en/news/all/drugiy-rik-povnomasshtabnoyi-viyni-obsyagi-bezgotivkovih-rozrahunkiv-zrostayut https://bank.gov.ua/en/news/all/drugiy-rik-povnomasshtabnoyi...
  - newzisforsukas 2y ago
    https://www.csmonitor.com/World/Middle-East/2024/0703/gaza-war-palestinians-cash-atms https://www.csmonitor.com/World/Middle-East/2024/0703/gaza-w... https://www.bloomberg.com/news/articles/2024-04-10/gaza-faces-major-cash-shortage-after-war-destroys-banking-system https://www.bloomberg.com/news/articles/2024-04-10/gaza-face...
    - makeitdouble 2y ago
      It's not a war, it's a genocide. At this point it's not cash that is missing, it's absolutely everything that sustain human life, and there's probably not enough working things to even barter.
  - kjkjadksj 2y ago
    This isn’t really all hell breaking loose actual war. If it were Kyiv would have been a ruin years ago.
cmrdporcupine 2y ago
How long before our evident incompetence as a profession comes back to bite us in the form of more draconian regulation about who and what is allowed to run in kernel space, or other privileged contexts, on critical infrastructure?
- remram 2y ago
  Is that a "bite"? I have wished for this for a long time.
- pdimitar 2y ago
  Whenever that time comes it would be at least 50 years too late. But I hope to see it in my lifetime.
  - cmrdporcupine 2y ago
    The problem I think is that it would just take the form of regulatory capture. A few companies would be blessed, and the rest of us locked out. And we'd still have screwups like yesterday, but this time with Government Approval. Already it's amazing how the media is presenting this like it's a natural disaster, instead of an entirely preventable display of incompetence... A business entity whose shares only dropped 10% after causing untold billions of damage to the economy. Gives us all a bad name.
    - pdimitar 2y ago
      > Already it's amazing how the media is presenting this like it's a natural disaster, instead of an entirely preventable display of incompetence Amazing? Super predictable I'd say. > Gives us all a bad name. That is sadly true.
    - pdimitar 2y ago
      > The problem I think is that it would just take the form of regulatory capture. A few companies would be blessed, and the rest of us locked out. And we'd still have screwups like yesterday, but this time with Government Approval. Yeah agreed. It would require no corruption... which is the true fantasy trope of our times.
adfm 2y ago
This piece was written by someone covering national security and the Biden administration for the NYT. It’s a global issue exposing vulnerabilities across the board. It’s journalism like this that’s the real vuln. Word.
- tomrod 2y ago
  Huh?
  - adfm 2y ago
    Expect more depth and less bias from the media you consume.
    - tomrod 2y ago
      That doesn't seem related to your first comment?
      - adfm 2y ago
        Besides ticking off a few boxes, there’s not much substance to the piece. It’s framed as a domestic issue, not a roadmap of vulnerabilities on a global scale. If I’m reading the NYT, I expect more effort.
- sigseg1v 2y ago
  Agreed. They would have never written this if they remembered to don their tinfoil headgear first.
GeoAtreides 2y ago
Here's an interesting exercise: what's the minimum quantity of explosives that would lead to 1% drop in western GDP? would doubling it lead to 2% or 4%? is the relationship linear? I don't have an answer, but thinking about it makes one understand how incredible fragile our complex logistic chains (and indeed our economy) are. One day all this complexity will collapse upon itself and we'll wonder what happened.
- verzali 2y ago
  Probably not a lot. Blowing up a ship in the middle of the Panama or Suez canal might do it, especially if you wreck it badly enough to block the canal for months. Even easier if you target a big oil tanker. I don't think this is linear though. It's easy to target a weak point to inflict a small amount of damage, but hitting say 10% of GDP would mean targeting multiple sectors of the economy and putting millions of people out of productive work.
- pjc50 2y ago
  Bomb in TSMC clean room. Almost any size. That takes out the AI market. 1% of GDP gone. However, it's less than linear; not many targets of such critical importance. The ability to get the bomb in the right place is far more important than the quantity of explosives, as was demonstrated by the recent suicide sniper missing. The IRA https://en.wikipedia.org/wiki/1993_Bishopsgate_bombing https://en.wikipedia.org/wiki/1993_Bishopsgate_bombing was estimated to cause more economic damage than all other IRA bombing put together. It's interesting that (apart from the first WTC bombing) American terrorists have stuck strictly to guns and not attempted car bombs.
  - krisoft 2y ago
    > It's interesting that (apart from the first WTC bombing) American terrorists have stuck strictly to guns and not attempted car bombs. That is not true. Oklahoma City bombing is the first which comes to mind where the explosives was planted on a truck. But there are many others, there is a whole wikipedia list about them: https://en.m.wikipedia.org/wiki/Category:Car_and_truck_bombings_in_the_United_States https://en.m.wikipedia.org/wiki/Category:Car_and_truck_bombi...
AlbertCory 2y ago
"Diversity" (but not in the sense of marginalized people) If more of the critical machines were running different OS's, the damage would be contained. When we talk about the dangers of "monoculture" it's usually about plants. The same danger applies to computing infrastructure.
- mr90210 2y ago
  On top of that, I am still struggling to understand how the people in charge of running orgs that run highly critical systems were OK with the idea that a 3rd party software provider could push at anytime patches to the software they provide. Sorry for being harsh with my following statement, but I believe that the companies affected by Crowdstrike share some responsibility on what happened yesterday.
  - chrisjj 2y ago
    The are OK with "push at anytime patches to the software" because that's a big part of what they are paying for. Rapid response to threats.
    - pixl97 2y ago
      >Ping reply from 127.0.0.1 The threat is inside the building!
  - lambdaone 2y ago
    You're making the mistake of assuming that the people running those companies care about anything other than their job security, and buying in solutions is the best way to have a ready-made scapegoat when things go wrong. The mantra "no-one ever got sacked for buying IBM" still holds, you can just substitute "Oracle", or "Microsoft", or now - apparently - "Crowdstrike".
  - meiraleal 2y ago
    They share the whole of the responsibility of it. "my antivirus was updating" is not an acceptable excuse for a service to be down.
neonate 2y ago
https://archive.md/CJdHl https://archive.md/CJdHl
lambdaone 2y ago
This has been an open secret for decades. Just a handful of major OS and browser vendors, constantly shipping patches to their systems and most software having such vast software supply chains that it's effectively impossible to audit anything, let alone truly certify anything as safe, and "security" software just expands the attack surface. Everyone in the industry knows this. Interesting to see the NYT just catching up.
- newzisforsukas 2y ago
  > Interesting to see the NYT just catching up. Maybe it has to do with some major incident that happened yesterday, and the fact they are a news company?
  - lambdaone 2y ago
    It's the equivalent of not writing about Boeing until the day a 737 MAX crashes right in front of your newpaper offices.
    - newzisforsukas 2y ago
      [flagged]
    - lastofthemojito 2y ago
      It feels more like writing about Boeing and then writing about Boeing again after the crash, considering the Times has been writing about cyber security and American vulnerability for a while: https://www.nytimes.com/2021/02/06/technology/cyber-hackers-usa.html https://www.nytimes.com/2021/02/06/technology/cyber-hackers-... https://www.nytimes.com/2018/01/03/business/computer-flaws.html https://www.nytimes.com/2018/01/03/business/computer-flaws.h... https://www.nytimes.com/2013/07/14/world/europe/nations-buying-as-hackers-sell-computer-flaws.html https://www.nytimes.com/2013/07/14/world/europe/nations-buyi... etc
    - SoftTalker 2y ago
      Readers wouldn't have cared nearly as much. NYT: Boeing is run by bean-counters and isn't taking engineering seriously anymore. Boeing: That's not true. Our aircraft fly thousands of times a day, every day, and are very safe. Who would find that very interesting, absent any relevant, dramatic current events?
gmuslera 2y ago
There are some point where you should redefine what it mean to be an adversary. To be practically forced into a position that lead to this level of harm, by actors that you don't want to perceive, is something that you may want to analyze. The purpose of a system is what it actually do, not what it claims to do but fails every time at that. Turning everything to vulnerable as fragile with some big strategic and global plan ahead makes you into a disposable asset, a sacrificial victim in some higher level chess game. And you can agree with that with your decisions.
mistrial9 2y ago
externalizing a threat, from a national news source.. Thought experiment -- a healthy society has plural viewpoints, and plural economic strengths. What if a core and entitled group of groups imposed their "security" on a plural society, for their own profit at the expense of the majority? What if their security is monoculture and internally inconsistent, without the ability to admit error ? What if there is a reflex to blame external groups specifically to divert attention from an internal and unbalanced chain of actions, controls and monetary flows? What is the response of a Free Press to news stories exercising reflexive blame-game from allied core groups with major monetary interests in the outcomes?
- feedforward 2y ago
  > externalizing a threat Yes, it's illustrative of the USA. Due to monopolies, lack of local control of infrastructure etc., a feature is rolled out that grinds hospitals, airports etc. to a halt. Surely due to forces we're surely familiar with - a rush to get profit-making features out, a neglect of correctness and stability, cost cutting etc. Then we have the New York Times, considered the sober voice of the establishment. What is discussed? Reflection on how entirely US-internal corporate processes led to this? No. A thought experiment about what if some external actor, perhaps one tired of US imperialism or something, had performed this. I read this after seeing Hulk Hogan rip his shirt off at the RNC in an Idiocracy prophecy manifested, while the other presidential candidate immersed in the dementia of the gerentocracy clings to power amidst his cohorts pleading he step aside. As I watch the US arming the Ukraine to fight Russia, I think back to 1986 and Gore Vidal's plea for an alliance with Russia lest Americans become either farmers or just entertainment for the more efficient Asians. Another prophecy which seems due to come to pass.
FerretFred 2y ago
Kent Walker's betting the farm on AI spotting future f*ckups? One born every minute!
- saagarjha 2y ago
  Kent Walker of…Alphabet?
  - FerretFred 2y ago
    The same! “We are optimistic that A.I. is actually allowing us to make significant — not transformative yet, but significant — progress in being able to identify vulnerabilities, patch holes, improve the quality of coding,” Kent Walker, the president for global affairs at Google, said at the Aspen forum.
enceladus06 2y ago
Why does IT even pay $$$ for crowdstrike? Time to uninstall it and figure something else out. Just use linux or chromeOS.
- betaby 2y ago
  People downvote you, but in the context you are mostly right. In case of airlines there is no reason to use Windows there, checking software is web-based and ChromeOS is a perfect fit there. Same goes for banks, bank tellers mostly use web browsers to access banking applications.
  - enceladus06 2y ago
    Probably has to do of IT wanting to keep using windows to justify its own existence ;). Web + chrome is so much better. Then just use qnx or something for embedded. Why is the actual reason that our $600k confocal microscope has to run windows? Qnx or Unix is much better for scientific and healthcare equipment.
  - adrr 2y ago
    How do you detect if your ChromeOS gets breached? Linux apps runs in sandboxes so even user level HIDs won't function.
- hylaride 2y ago
  Mission critical use of Linux still needs malware and breach detection software. It's not as simple as switching OS's.
- anvuong 2y ago
  As if Linux doesn't have malware or security breach
  - mango7283 2y ago
    It's like these people didn't notice the raft of cves on hardened VPN and firewall devices lately. Cisco iOS regularly has cves. Android and iOS both have critical rces. And remember Mirai?
SkyPuncher 2y ago
For the non-tech folks, this probably felt like one step away from an attack from an adversary. I have a different take. This was still far from being an adversarial attack. There was no security breach. The failed configuration came from an SDLC that remained secure and fully in control of CrowdStrike. It was a terrible bug, but it was not an attack
- umvi 2y ago
  I would not call it a bug. I would call it a severe process or systemic failure. Their SDLC clearly did not include any sort of phased rollout or canary deployments. Bugs are inevitable, what matters is being able to catch them before you push them to every end user on the planet.
- cdchn 2y ago
  If CrowdStrike's system wasn't able to prevent a kernel driver thats all zeros from getting by, you can be sure a malicious payload would have breezed right through. There was no validation, phased roll-outs, almost certain no multi-person verification. I'd bet dollars to donuts this was pushed out by a low/mid-level functionary that could be carried out by dozens if not hundreds of employees. There may have not been a security breach, but it was still one minor security breach, distracted open laptop in a cafe, or disgruntled/paid-off inside actor away from absolute armageddon. It wasn't an attack, but it was a raccoon who came in through an unlocked screen door in the back of Fort Knox. If someone had used this to deliver a ransomeware package, they'd be buying a mega-yacht right now.
  - saagarjha 2y ago
    It’s not a driver, but a configuration file.
    - cdchn 2y ago
      Sources I've seen was that there was a .SYS file with all zeros that caused the BSOD. A configuration file shouldn't cause a bluescreen. EDIT: It is in the 'drivers' directory, has a .SYS extension, but was something called a "channel file" but I couldn't get much info on what a channel file does other than "something something named pipes"
ScottBurson 2y ago
Really interesting to me that none of the commentators I've seen in the press have even hinted that maybe an OS that requires frequent security patches shouldn't be used for infrastructure in the first place. For just one example, I've seen photos of BSODs on airport monitors that show flight lists -- why aren't those built on Linux or even OpenBSD? Security is not a feature that can be layered on. It has to be built in. We now have an entire industry dedicated to trying to layer security onto Windows -- but it still doesn't work.
- dopylitty 2y ago
  Or don't use an OS at all. We need to think about minimizing the use of software in critical infrastructure. If that means less efficiency because you have to be near something to maintain it then so be it. That would be good for jobs anyway.
  - antihero 2y ago
    How exactly would a lot of end user systems function without one?
    - advael 2y ago
      You can run single-purpose software on bare metal, and many OS-agnostic toolkits for things like user interfaces exist
      - sabas123 2y ago
        I'd like to pose two questions: 1. How does the software obtain new data at run time? 2. How do you make sure that thing doesn't pose a security hole when a vulnerability gets discovered? (assuming this never happens is unrealistic)
        advael 2y ago
        The answer to both questions is robust organizational infrastructure. To be frank, I think a minimal linux system as a baseline OS serves most use cases better than a bare metal application, but many applications have self-contained update systems and can connect to networks. Self-repairable infrastructure is a necessity, both in terms of tooling and staffing, for any organization for which an outage or a breach could be catastrophic, and the rise of centralized, cloud-reliant infrastructure in these contexts should be seen as a massive and unacceptable risk for those organizations to take on. Organizations being subject to unpatched vulnerabilities and inability to manage their systems competently are direct results of replacing internal competency and purpose-built systems with general-purpose systems maintained and controlled by unaccountable distant tech monopolies
siliconc0w 2y ago
Really, the problem is that all this critical infrastructure runs on Windows. Critical systems should effectively be appliances that run with a very minimal footprint. If you absolutely need to monitor them you can export disk snapshots or something out of band that can't impact operations.
cdchn 2y ago
If CrowdStrike's system wasn't able to prevent a kernel driver thats all zeros from getting by, you can be sure a malicious payload would have breezed right through.
- chrisoverzero 2y ago
  It wasn’t a driver.
- therein 2y ago
  Oh yeah, at a quick glance looks like that file could have had any payload and it would have been loaded right into the kernel.
  - saagarjha 2y ago
    What makes you think so?
    - therein 2y ago
      Earlier there were some screenshots showing an entirely zero-filled .sys file but now we know that's not what the payload was.
  - echoangle 2y ago
    To send a malicious payload into the kernel, you would have to take over crowdstrikes deploy infrastructure first, right? I hope the program doesn’t just accept updates from anywhere
bitexploder 2y ago
Umm, they (adversaries) already knew? Been in cybersecurity for 18 yrs. We told customers about issues like this all the time
dgoldstein0 2y ago
Crowdstrike has really redefined malicious compliance
simpaticoder 2y ago
Massive computer outage, worldwide affecting enterprises with Windows machines running CrowdStrike, a very popular software that is sold as hacking protection but which is, in reality, used by C-suite execs to spy on employee behavior. It is installed with extraordinary permissions and is difficult to fix or remove by design. I wonder if this will teach absolutely anyone a lesson about anything.
- BodyCulture 2y ago
  Can we please get more information about the spying features? Some screenshots would be great! Thanks!
  - spydum 2y ago
    Not as evil as they make it sound. Process Execution, detailed timestamps, and network metadata capture are core features of every EDR tool (CrowdStrike, MDE, SentinelOne, etc) that exists. They can just be abused to monitor user behavior, in addition to threat hunting or malicious activity. Telemetry isn't inherently evil, but organizations need to establish privacy and usage governance around security tools to prohibit abuse.
- skynr 2y ago
  I think it will. MS has published a number that it was 8.5 million machines, which I don't believe, bur seeing the effort that's gone into the response even at my own relatively mid sized org, there are super simple questions like how the heck do we even get to these devices when hald the crew work remote. The response is and always will be - how much will this cost. We now have the opposing figure, how much will this cost if we don't do it.
  - echoangle 2y ago
    Do you believe it was more or less than 8.5 million machines?
    - skynr 2y ago
johanneskanybal 2y ago
So yea let’s not use a company like this as best practice. Everything about this reeks of worst practices rising the wave of regulatory capture.
oneplane 2y ago
There is no "Digital Resilience" because that is perceived as too expensive, a cost center with hard to quantify value. So it's easier to try and carve out everything that doesn't fit into a spreadsheet, everything that isn't core business, and everything that is not able to present what value it generated. If general IT had the abilities of sales, marketing, or insurance, there might be a chance that the business would take the responsibility to have the internal knowledge and capabilities to assert control over their systems. But they don't, and as such they won't and instead shove that responsibility over to a third party generalist elsewhere with enough paperwork to have both parties feel their asses are covered. As long as everything seems to be working, the signals that are still getting through is project failures, be it complete failures or just time and/or money being consumed more than planned and maybe some requirements getting cut. But as soon as enough stuff breaks at the same time, we get news outlets writing articles about resilience and the greater public suddenly no longer agreeing with that is effectively just the result of the status quo because it impacts them directly.
sschueller 2y ago
If you are a non-US company you have to be insane to use this CrowdStrike service. The FBI can legally use a secret warrant[1] and force CrowdStrike to inject a DLL into your infrastructure! [1] https://en.wikipedia.org/wiki/United_States_Foreign_Intelligence_Surveillance_Court https://en.wikipedia.org/wiki/United_States_Foreign_Intellig...
- autoexec 2y ago
  You think they can't/don't do that to force Microsoft to push an "update" that does the same thing?
  - jenscow 2y ago
    I doubt they'd even need to go through Microsoft
- jml7c5 2y ago
  Are you sure that is correct? I was under the impression that US government could order companies to turn over data, but that they could not compel them to actually do work. This was the center of the dispute between the government and Apple after the San Bernadino shooting: Apple was within their legal rights to refuse to provide assistance. https://en.wikipedia.org/wiki/Apple%E2%80%93FBI_encryption_dispute https://en.wikipedia.org/wiki/Apple%E2%80%93FBI_encryption_d... The lengths that the NSA and CIA would go to to implant backdoors (interdicting shipments of laptops/phones and doing the work themselves) further suggests that they cannot compel this sort of action.
  - Fluorescence 2y ago
    That case was theatre / kayfabe. The FBI was using an emotive case for turning public opinion against encryption and set some legal precedent. The goal wasn't really to unlock the phone which could and was be done by other means. If they have a path to covertly compel action as a state secret under National Security / anti-terror laws we will only hear about it from whistleblowers. It won't be something the target can disclose let alone test in court. FWIW I also don't believe in Apple's nobility as resisting on user's behalf. They happily bow to the state and remove apps for e.g. organising protests, monitoring deaths in US wars, csam scanning etc. IMHO their interest in encryption is to prevent jail-breaking and protect their app-store cash cow. > interdicting shipments of laptops/phones and doing the work themselves I don't think that proves anything about their powers. Given the option, I'm sure they would prefer to install things themselves without third-party knowledge or consent. We have evidence of complicit action e.g. black rooms like Room 641A. I think the nature of "consent" and "obligation" gets pretty grey when it comes to the security agencies. They don't get results using court orders. I'm sure they have assets employed as staff in security sensitive positions.
notepad0x90 2y ago
In a twisted way, Crowdstrike just gave western civilization a disaster recovery and resilience forced test. an actual attack won't be rolled back within an hour. In case you don't know, Crowdstrike is hardly the only company with large scale access to this many companies,governments and resources. It takes one rogue employee to deploy a disk wiper that destroys every computer (including linux and macos) and affected systems won't recover at all. it would be months before critical systems are back online, the global economy would come to a halt worse than how it did with COVID in such a scenario. It isn't "why didn't Crowdstrike do better" (although they should have), it is more, why isn't technology in critical systems more resilient to one vendor screwing up or getting hacked? For example, let's say it wasn't just a boot loop but a disk wiper erased every boot disk, is there any reason pxe booting a recovery image or a backup image configured already on servers, atms, kiosks, point of sale systems,etc...? even if UEFI and bios were erased, it is technically not impossible to have an auto-recovery mechanism implemented right? If you have never been in an incident response (IT and security incidents) root cause analysis, I don't blame you for not thinking deeper about the root cause, but that is the type of root cause analysis that has been missing despite over a decade of rampant ransomware, disk wipers, and supply chain risks. Finding someone to blame and be angry at is easy and doesn't solve the root cause. Making hard technical decisions and not wasting this opportunity (never waste a good crisis) to push for resilient technology investments actually solves the root cause behind this and other repeating problems.
- yusyusyus 2y ago
  if the firmware is totally nuked, you'd need backup firmware. at some point, all of this crap can be made non-recoverable, but that isn't the real problem to solve. imma take your comment one step further and say that the emphasis on security is coming at the expense of discussions on resilience. and security matters a lot less, especially financially, than resilience.
  - notepad0x90 2y ago
    availability is one of the core tenets of security. Security = a measurement of confidentiality, integrity and availability. Backup firmware and boot images can be configured as read only.
thedataexchange 2y ago
1. It’s a good time to reread the article that got Dan Geer famous on “monocultures” => https://ccianet.org/wp-content/uploads/2003/09/cyberinsecurity%20the%20cost%20of%20monopoly.pdf https://ccianet.org/wp-content/uploads/2003/09/cyberinsecuri... 2. Also a great time to start prepping for AI Incidents => https://thedataexchange.media/ai-incident-response/ https://thedataexchange.media/ai-incident-response/
m3kw9 2y ago
the problem is that for a security scanner to scan threats properly, they need to sit on the kernel, there should be a mode where they allow scanners to read but is not able to crash the system. Some sort of sand box for all these kernel access
- kchr 2y ago
  > Some sort of sand box for all these kernel access Yeah, like eBPF in Linux or System Extensions in macOS.
Timber-6539 2y ago
The only vulnerability here was CrowdStrike's EDR product that runs exclusively in ring 0 and the entire corporate & technical class that lazily relied on this flawed security model and centalized this incompetence. As much as some people want to believe that Microsoft is blameless here, I hold them partly responsible. They need to create a stable API in their kernel and force third party security vendors to use it.
linearrust 2y ago
> It was, by all appearances, purely human error — a few bad keystrokes that demonstrated the fragility of a vast set of interconnected networks in which one mistake can cause a cascade of unintended consequences. Cute. It's always those bad keystrokes. If only these crowdstrike employees worked on their good keystrokes that morning. I blame management. > Russian hackers working on behalf of Vladimir V. Putin bring down hospital systems across the United States. In others, China’s military hackers trigger chaos, shutting down water systems and electric grids to distract Americans from an invasion of Taiwan. ... Among Washington’s cyberwarriors, the first reaction on Friday morning was relief that this wasn’t a nation-state attack. For two years now, the White House, the Pentagon and the nation’s cyberdefenders have been trying to come to terms with “Volt Typhoon,” a particularly elusive form of malware that China has put into American critical infrastructure. So we have cyberwarriors and cyberdefenders? And the russians, china, etc have 'hackers'. If ever there was a doubt what the nytimes really is. > The fear is, in an election year, that the next digital meltdown may have a deeper political purpose. Oh dear. More bad keystrokes on the way? Did anyone glean anything of value from the article? There was a lot of words but no substance.
pm90 2y ago
> It is not hopeless. > “We are optimistic that A.I. is actually allowing us to make significant — not transformative yet, but significant — progress in being able to identify vulnerabilities, patch holes, improve the quality of coding,” Kent Walker, the president for global affairs at Google, said at the Aspen forum. I disagree. If the only hope is some vague promise of bs AI, there is no hope indeed.
- pards 2y ago
  Yep. This is like arguing that the best way to prevent school shootings is to give teachers guns. I don't think AI would have done a better job convincing CrowdStrike's leadership team that a staged rollout strategy is worth the cost. These issues are caused by people, not technology, so more technology isn't going to fix it.
someonehere 2y ago
Can someone from one of the major services comment on why they don’t run the N -1 policy on Falcon? My onboarding sales engineer recommended this to me years ago to avoid this situation. Why do critical infrastructure companies run bleeding edge updates like this?
naveen99 2y ago
Or it’s just a front for the nsa and cia.
shirro 2y ago
I haven't worked in a Windows environment for a long time so was a little surprised how much of the online commentary suggests people in that environment are comfortable or at least resigned to the necessity of unattended live third party updates on critical infrastructure. I can't see any justification for that on the *nix side of things and hope that culture never transfers over.
- Osiris 2y ago
  I have worked at a few Windows places and they did not allow live updates. All updates were managed by IT and tested before being pushed out. This situation was caused basically by an anti virus definition update. Without any information, my guess is that CrowdStrike specifically doesn't provide a means for enterprises to manage the CrowdStrike updates because that would cause potentially weeks or months of delays for critical anti virus updates to be released. I'm more curious why they need CrowdStrike on every system in the first place. I can understand employee computers but servers and critical systems should have other security measures in place to make them less available to attack in the first place.
  - smileybarry 2y ago
    > I'm more curious why they need CrowdStrike on every system in the first place. I can understand employee computers but servers and critical systems should have other security measures in place to make them less available to attack in the first place. The boring answer(s) is compliance and necessity. You need a security solution installed & running for a bunch of certifications, and maybe you also need it to secure some unmaintained software you have to run. Plus, even if the machine isn't front-facing, a lot of these EDR solutions are installed to prevent or monitor for lateral movement.
mrjin 2y ago
The fire more deadly than enemy fire is friendly fire. For adversaries, they cannot do any harm unless they get in, even if they get in, the damage is limited to the access of the account they run on. But for AVs, they are invited in, which renders the 1st line of defense useless. Making it worse, they are running with SYSTEM privileges, which is higher than Admin privileges. And we just witnessed what could happen if AVs went rogue.
whjkh 2y ago
[flagged]

279 points by jmsflknr 2y ago

On the one hand - you can read this as a PSA for the apathetic and/or clueless 99.9%. On the other hand - it's d*mn hard to imagine that any of America's "A List" or "B List" adversaries didn't have a far-more-detailed road map, years ago.

pdimitar 2y ago
Does the last part of your comment imply that USA should just give up and accept all its adversaries already have backdoors and nothing can be done about it?
- nonrandomstring 2y ago
  > accept all its adversaries already have backdoors This is actually a really useful hypothetical standpoint to work out security from. Designing systems that start from the assumption of insecurity helps us build more robust protocols and management. Qubes OS starts from the position that all VMs are or soon will be compromised. Zero-trust in network design assumes the bad guys already have the whole network. Plenty out there would like to shrug and say "the endpoints are all rotten too" (especially with phones which are a veritable hell to secure) and move trust into the application via trusted execution methods. > and nothing can be done about it? No, That doesn't follow. It's prudent to be realistic about threats. but there's always a way out, at a cost. The cost, in a complexity crisis, is throwing away a lot of what we've done.
  - pdimitar 2y ago
    Yeah I am completely with you and I agree. But it seems that reducing costs is more important even compared to preventing people on life support in hospitals from dying. What a world.
  - warkdarrior 2y ago
    > Qubes OS starts from the position that all VMs are or soon will be compromised. Zero-trust in network design assumes the bad guys already have the whole network. So what does Qubes OS do to protect against a hypervisor bug? Those must exist. How do you ensure that your systems are still working and retrieving data from databases, etc, when bad guys have the whole network and can block all communications?
    - nonrandomstring 2y ago
      The answer to both those questions is you can't. So you you either need to make other provisions at different levels of the stack or design your architecture to make them irrelevant to your security model.
- bell-cot 2y ago
  No, and I've no idea where you got that from. Here's the HN Title: "CrowdStrike debacle provides road map of American vulnerabilities to adversaries" My assertion: America's serious-threat adversaries already had far more detailed road maps, years ago. The intel value of whatever "road map" data they got from the CrowdStrike debacle was pretty marginal. Neither the HN Title nor I said anything about backdoors. And within 2 para's, the NYT story makes it clear that CloudStrike's Big Oopsied had nothing to do with bad guys hacking anything.
- marcosdumay 2y ago
  Did any large company clean their datacenters after they stopped using Solar Winds?
red_admiral 2y ago
I'm sure there's a few adversaries who could pull something like this off, and have 0-days ready. But if they use them, the US could see that as a hostile action and get very upset about it.
- LtWorf 2y ago
  they can just bribe a company to do it :D

ndesaulniers 2y ago

Yeah, didn't the US just ban Kaspersky, over fears that Kaspersky could cause such an outage (among other fears)? Turns out our homegrown CrowdStrike was just as bad as our fears over Kaspersky were. Perhaps worse.

supriyo-biswas 2y ago
The Kaspersky issue could have been better handled by simply requiring divestment or by having requiring an US-appointed auditor to investigate produce reports to assuage such concerns; as was proposed in the case of Tiktok.
dataflow 2y ago
> over fears that Kaspersky could cause such an outage (among other fears)? Citation? I thought it was the "other fears"; this is the first time I'm hearing accidental outages were one of the concerns.
- mathgeek 2y ago
  Not GP, but the decision and reasoning is at https://public-inspection.federalregister.gov/2024-13532.pdf https://public-inspection.federalregister.gov/2024-13532.pdf (I am not claiming any specific “other fears”, just linking to the source)
  - dataflow 2y ago
    Thanks. Yeah, i, ii, and iii all talk about malicious events, not accidental.
- mardifoufs 2y ago
  Yes but any accidental outages from an entity like Kaspersky would have been considered non accidental regardless of the actual root cause. If crowdstrike was Russian, the headlines would be a bit more suspicious about yesterday's event. or if they had brought down Russian infrastructure Russia would have probably been suspicious about American involvement, even if it's just accidental.

pdimitar 2y ago

Wouldn't any memory-safe language help prevent this NULL pointer access? Why are all these crucial pieces still written in C/C++, when it's obvious to anybody keeping even remote track of CVEs that these languages are just not up to the task with today's climate of a 24/7 shadow internet war? (The one that's likely been going on for at least 25 years at this point?) When will we learn? You hate Rust -- fine (not fine but OK, I guess people get super triggered over it and it's a reality I can't change but I am still baffled by it because they throw away reason for emotions and these people should really know better). Fine. Just use Golang or any other GC language really (Java or C# as well, if you must). When will we abandon convenient routine and start adapting to modern realities? ("Modern" being at least 25-year old here but hey, I am willing to give you some leeway and not roast you too much. Let's assume these are "modern" realities, f.ex. just the last 5 years.)

imranhou 2y ago
Windows does not support drivers based on rust language, so perhaps may be another 5 years
- pdimitar 2y ago
  Rust can't compile DLLs? AFAIK it can?
  - cmrdporcupine 2y ago
    This is a kernel driver. Runs in kernel space. Intercepts syscalls. You'd definitely be fighting uphill to write it in Rust. And your code would be riddled with `unsafe` by necessity anyways.
    - pdimitar 2y ago
      Fair enough, still Rust's unsafe is not dropping all of its guarantees. Quite a lot of them remain in place. Not saying you can't write bugs in Rust, of course -- that would be crazily delusional. I am saying they needed a better process. And I am saying that a stricter language could have improved the process a bit as well.

hilbert42 2y ago

"What Happened to Digital Resilience?" Was there ever such a time? If so then tell me when it was. "The latest chaos wasn’t caused by an adversary, but it provided a road map of American vulnerabilities at a critical moment." I've no doubt that road maps of American vulnerabilities are currently being planned, roadmaped and stockpiled for future use by those who aren't on the best terms with the US. In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that. Remember, this happened big-time when Microsoft introduced Windows especially Windows 95. To capture the market Microsoft made everything as easy as possible for nontechnical users—just click on something and it'd happen, things would happen with ease. And all this happened without due consideration to security. When viruses, vulnerabilities, breaches got out of hand restrictions were introduced which meant users had less freedom to do what they'd gotten used to doing. What Microsoft did was to get the world used to slack operating procedures and efforts reign this in has met with user resistance ever since. We're now stuck with a major problem that was easily foreseeable even before Microsoft launched Windows 95. Fixing it will be extremely difficult.

dralley 2y ago
This is an area where studying Ukraine's experience will be very useful (and probably has already been useful) There were years of cyberattacks against pretty much every peice of critical infrastructure they have. Things went down, there were disruptions, but they adapted. Sometimes by falling back to low-tech solutions, sometimes by developing new systems with robustness into new systems and purging the old (much easier to politically justify when the problem is tangible and immediate). I seem to recall that one of the first things we did when tensions started ramping up was sending teams of cyber security experts from the NSA to help them lock down and root out infiltrations.
- emseetech 2y ago
  | Sometimes by falling back to low-tech solutions My first thought in all this was wondering if there's a business opportunity for a consulting firm or startup that designs and manages offline paper backup systems that can quickly and seamlessly integrate back with digital systems once they come back online.

encoderer 2y ago

Just told my family yesterday that if we are ever in a real war expect everything to stop working within 8 hours. We will go back to cash and paperwork but it will be painful and slow.

newzisforsukas 2y ago
Just storm EDR company offices slap guns to devs' heads, push geofenced destruction.
makeitdouble 2y ago
Looking at two countries in an actual long running war, both kept using cashless means, with actually increases in usage: https://cbr.ru/eng/press/event/?id=18776 https://cbr.ru/eng/press/event/?id=18776 https://bank.gov.ua/en/news/all/drugiy-rik-povnomasshtabnoyi-viyni-obsyagi-bezgotivkovih-rozrahunkiv-zrostayut https://bank.gov.ua/en/news/all/drugiy-rik-povnomasshtabnoyi...
- newzisforsukas 2y ago
  https://www.csmonitor.com/World/Middle-East/2024/0703/gaza-war-palestinians-cash-atms https://www.csmonitor.com/World/Middle-East/2024/0703/gaza-w... https://www.bloomberg.com/news/articles/2024-04-10/gaza-faces-major-cash-shortage-after-war-destroys-banking-system https://www.bloomberg.com/news/articles/2024-04-10/gaza-face...
  - makeitdouble 2y ago
    It's not a war, it's a genocide. At this point it's not cash that is missing, it's absolutely everything that sustain human life, and there's probably not enough working things to even barter.
- kjkjadksj 2y ago
  This isn’t really all hell breaking loose actual war. If it were Kyiv would have been a ruin years ago.

cmrdporcupine 2y ago

How long before our evident incompetence as a profession comes back to bite us in the form of more draconian regulation about who and what is allowed to run in kernel space, or other privileged contexts, on critical infrastructure?

remram 2y ago
Is that a "bite"? I have wished for this for a long time.
pdimitar 2y ago
Whenever that time comes it would be at least 50 years too late. But I hope to see it in my lifetime.
- cmrdporcupine 2y ago
  The problem I think is that it would just take the form of regulatory capture. A few companies would be blessed, and the rest of us locked out. And we'd still have screwups like yesterday, but this time with Government Approval. Already it's amazing how the media is presenting this like it's a natural disaster, instead of an entirely preventable display of incompetence... A business entity whose shares only dropped 10% after causing untold billions of damage to the economy. Gives us all a bad name.
  - pdimitar 2y ago
    > Already it's amazing how the media is presenting this like it's a natural disaster, instead of an entirely preventable display of incompetence Amazing? Super predictable I'd say. > Gives us all a bad name. That is sadly true.
  - pdimitar 2y ago
    > The problem I think is that it would just take the form of regulatory capture. A few companies would be blessed, and the rest of us locked out. And we'd still have screwups like yesterday, but this time with Government Approval. Yeah agreed. It would require no corruption... which is the true fantasy trope of our times.

adfm 2y ago

This piece was written by someone covering national security and the Biden administration for the NYT. It’s a global issue exposing vulnerabilities across the board. It’s journalism like this that’s the real vuln. Word.

tomrod 2y ago
Huh?
- adfm 2y ago
  Expect more depth and less bias from the media you consume.
  - tomrod 2y ago
    That doesn't seem related to your first comment?
    - adfm 2y ago
      Besides ticking off a few boxes, there’s not much substance to the piece. It’s framed as a domestic issue, not a roadmap of vulnerabilities on a global scale. If I’m reading the NYT, I expect more effort.
sigseg1v 2y ago
Agreed. They would have never written this if they remembered to don their tinfoil headgear first.

GeoAtreides 2y ago

Here's an interesting exercise: what's the minimum quantity of explosives that would lead to 1% drop in western GDP? would doubling it lead to 2% or 4%? is the relationship linear? I don't have an answer, but thinking about it makes one understand how incredible fragile our complex logistic chains (and indeed our economy) are. One day all this complexity will collapse upon itself and we'll wonder what happened.

verzali 2y ago
Probably not a lot. Blowing up a ship in the middle of the Panama or Suez canal might do it, especially if you wreck it badly enough to block the canal for months. Even easier if you target a big oil tanker. I don't think this is linear though. It's easy to target a weak point to inflict a small amount of damage, but hitting say 10% of GDP would mean targeting multiple sectors of the economy and putting millions of people out of productive work.
pjc50 2y ago
Bomb in TSMC clean room. Almost any size. That takes out the AI market. 1% of GDP gone. However, it's less than linear; not many targets of such critical importance. The ability to get the bomb in the right place is far more important than the quantity of explosives, as was demonstrated by the recent suicide sniper missing. The IRA https://en.wikipedia.org/wiki/1993_Bishopsgate_bombing https://en.wikipedia.org/wiki/1993_Bishopsgate_bombing was estimated to cause more economic damage than all other IRA bombing put together. It's interesting that (apart from the first WTC bombing) American terrorists have stuck strictly to guns and not attempted car bombs.
- krisoft 2y ago
  > It's interesting that (apart from the first WTC bombing) American terrorists have stuck strictly to guns and not attempted car bombs. That is not true. Oklahoma City bombing is the first which comes to mind where the explosives was planted on a truck. But there are many others, there is a whole wikipedia list about them: https://en.m.wikipedia.org/wiki/Category:Car_and_truck_bombings_in_the_United_States https://en.m.wikipedia.org/wiki/Category:Car_and_truck_bombi...

AlbertCory 2y ago

"Diversity" (but not in the sense of marginalized people) If more of the critical machines were running different OS's, the damage would be contained. When we talk about the dangers of "monoculture" it's usually about plants. The same danger applies to computing infrastructure.

mr90210 2y ago
On top of that, I am still struggling to understand how the people in charge of running orgs that run highly critical systems were OK with the idea that a 3rd party software provider could push at anytime patches to the software they provide. Sorry for being harsh with my following statement, but I believe that the companies affected by Crowdstrike share some responsibility on what happened yesterday.
- chrisjj 2y ago
  The are OK with "push at anytime patches to the software" because that's a big part of what they are paying for. Rapid response to threats.
  - pixl97 2y ago
    >Ping reply from 127.0.0.1 The threat is inside the building!
- lambdaone 2y ago
  You're making the mistake of assuming that the people running those companies care about anything other than their job security, and buying in solutions is the best way to have a ready-made scapegoat when things go wrong. The mantra "no-one ever got sacked for buying IBM" still holds, you can just substitute "Oracle", or "Microsoft", or now - apparently - "Crowdstrike".
- meiraleal 2y ago
  They share the whole of the responsibility of it. "my antivirus was updating" is not an acceptable excuse for a service to be down.

neonate 2y ago

https://archive.md/CJdHl https://archive.md/CJdHl

lambdaone 2y ago

This has been an open secret for decades. Just a handful of major OS and browser vendors, constantly shipping patches to their systems and most software having such vast software supply chains that it's effectively impossible to audit anything, let alone truly certify anything as safe, and "security" software just expands the attack surface. Everyone in the industry knows this. Interesting to see the NYT just catching up.

newzisforsukas 2y ago
> Interesting to see the NYT just catching up. Maybe it has to do with some major incident that happened yesterday, and the fact they are a news company?
- lambdaone 2y ago
  It's the equivalent of not writing about Boeing until the day a 737 MAX crashes right in front of your newpaper offices.
  - newzisforsukas 2y ago
    [flagged]
  - lastofthemojito 2y ago
    It feels more like writing about Boeing and then writing about Boeing again after the crash, considering the Times has been writing about cyber security and American vulnerability for a while: https://www.nytimes.com/2021/02/06/technology/cyber-hackers-usa.html https://www.nytimes.com/2021/02/06/technology/cyber-hackers-... https://www.nytimes.com/2018/01/03/business/computer-flaws.html https://www.nytimes.com/2018/01/03/business/computer-flaws.h... https://www.nytimes.com/2013/07/14/world/europe/nations-buying-as-hackers-sell-computer-flaws.html https://www.nytimes.com/2013/07/14/world/europe/nations-buyi... etc
  - SoftTalker 2y ago
    Readers wouldn't have cared nearly as much. NYT: Boeing is run by bean-counters and isn't taking engineering seriously anymore. Boeing: That's not true. Our aircraft fly thousands of times a day, every day, and are very safe. Who would find that very interesting, absent any relevant, dramatic current events?

gmuslera 2y ago

There are some point where you should redefine what it mean to be an adversary. To be practically forced into a position that lead to this level of harm, by actors that you don't want to perceive, is something that you may want to analyze. The purpose of a system is what it actually do, not what it claims to do but fails every time at that. Turning everything to vulnerable as fragile with some big strategic and global plan ahead makes you into a disposable asset, a sacrificial victim in some higher level chess game. And you can agree with that with your decisions.

mistrial9 2y ago

externalizing a threat, from a national news source.. Thought experiment -- a healthy society has plural viewpoints, and plural economic strengths. What if a core and entitled group of groups imposed their "security" on a plural society, for their own profit at the expense of the majority? What if their security is monoculture and internally inconsistent, without the ability to admit error ? What if there is a reflex to blame external groups specifically to divert attention from an internal and unbalanced chain of actions, controls and monetary flows? What is the response of a Free Press to news stories exercising reflexive blame-game from allied core groups with major monetary interests in the outcomes?

feedforward 2y ago
> externalizing a threat Yes, it's illustrative of the USA. Due to monopolies, lack of local control of infrastructure etc., a feature is rolled out that grinds hospitals, airports etc. to a halt. Surely due to forces we're surely familiar with - a rush to get profit-making features out, a neglect of correctness and stability, cost cutting etc. Then we have the New York Times, considered the sober voice of the establishment. What is discussed? Reflection on how entirely US-internal corporate processes led to this? No. A thought experiment about what if some external actor, perhaps one tired of US imperialism or something, had performed this. I read this after seeing Hulk Hogan rip his shirt off at the RNC in an Idiocracy prophecy manifested, while the other presidential candidate immersed in the dementia of the gerentocracy clings to power amidst his cohorts pleading he step aside. As I watch the US arming the Ukraine to fight Russia, I think back to 1986 and Gore Vidal's plea for an alliance with Russia lest Americans become either farmers or just entertainment for the more efficient Asians. Another prophecy which seems due to come to pass.

FerretFred 2y ago

Kent Walker's betting the farm on AI spotting future f*ckups? One born every minute!

saagarjha 2y ago
Kent Walker of…Alphabet?
- FerretFred 2y ago
  The same! “We are optimistic that A.I. is actually allowing us to make significant — not transformative yet, but significant — progress in being able to identify vulnerabilities, patch holes, improve the quality of coding,” Kent Walker, the president for global affairs at Google, said at the Aspen forum.

enceladus06 2y ago

Why does IT even pay $$$ for crowdstrike? Time to uninstall it and figure something else out. Just use linux or chromeOS.

betaby 2y ago
People downvote you, but in the context you are mostly right. In case of airlines there is no reason to use Windows there, checking software is web-based and ChromeOS is a perfect fit there. Same goes for banks, bank tellers mostly use web browsers to access banking applications.
- enceladus06 2y ago
  Probably has to do of IT wanting to keep using windows to justify its own existence ;). Web + chrome is so much better. Then just use qnx or something for embedded. Why is the actual reason that our $600k confocal microscope has to run windows? Qnx or Unix is much better for scientific and healthcare equipment.
- adrr 2y ago
  How do you detect if your ChromeOS gets breached? Linux apps runs in sandboxes so even user level HIDs won't function.
hylaride 2y ago
Mission critical use of Linux still needs malware and breach detection software. It's not as simple as switching OS's.
anvuong 2y ago
As if Linux doesn't have malware or security breach
- mango7283 2y ago
  It's like these people didn't notice the raft of cves on hardened VPN and firewall devices lately. Cisco iOS regularly has cves. Android and iOS both have critical rces. And remember Mirai?

SkyPuncher 2y ago

For the non-tech folks, this probably felt like one step away from an attack from an adversary. I have a different take. This was still far from being an adversarial attack. There was no security breach. The failed configuration came from an SDLC that remained secure and fully in control of CrowdStrike. It was a terrible bug, but it was not an attack

umvi 2y ago
I would not call it a bug. I would call it a severe process or systemic failure. Their SDLC clearly did not include any sort of phased rollout or canary deployments. Bugs are inevitable, what matters is being able to catch them before you push them to every end user on the planet.
cdchn 2y ago
If CrowdStrike's system wasn't able to prevent a kernel driver thats all zeros from getting by, you can be sure a malicious payload would have breezed right through. There was no validation, phased roll-outs, almost certain no multi-person verification. I'd bet dollars to donuts this was pushed out by a low/mid-level functionary that could be carried out by dozens if not hundreds of employees. There may have not been a security breach, but it was still one minor security breach, distracted open laptop in a cafe, or disgruntled/paid-off inside actor away from absolute armageddon. It wasn't an attack, but it was a raccoon who came in through an unlocked screen door in the back of Fort Knox. If someone had used this to deliver a ransomeware package, they'd be buying a mega-yacht right now.
- saagarjha 2y ago
  It’s not a driver, but a configuration file.
  - cdchn 2y ago
    Sources I've seen was that there was a .SYS file with all zeros that caused the BSOD. A configuration file shouldn't cause a bluescreen. EDIT: It is in the 'drivers' directory, has a .SYS extension, but was something called a "channel file" but I couldn't get much info on what a channel file does other than "something something named pipes"

ScottBurson 2y ago

Really interesting to me that none of the commentators I've seen in the press have even hinted that maybe an OS that requires frequent security patches shouldn't be used for infrastructure in the first place. For just one example, I've seen photos of BSODs on airport monitors that show flight lists -- why aren't those built on Linux or even OpenBSD? Security is not a feature that can be layered on. It has to be built in. We now have an entire industry dedicated to trying to layer security onto Windows -- but it still doesn't work.

dopylitty 2y ago
Or don't use an OS at all. We need to think about minimizing the use of software in critical infrastructure. If that means less efficiency because you have to be near something to maintain it then so be it. That would be good for jobs anyway.
- antihero 2y ago
  How exactly would a lot of end user systems function without one?
  - advael 2y ago
    You can run single-purpose software on bare metal, and many OS-agnostic toolkits for things like user interfaces exist
    - sabas123 2y ago
      I'd like to pose two questions: 1. How does the software obtain new data at run time? 2. How do you make sure that thing doesn't pose a security hole when a vulnerability gets discovered? (assuming this never happens is unrealistic)
      - advael 2y ago
        The answer to both questions is robust organizational infrastructure. To be frank, I think a minimal linux system as a baseline OS serves most use cases better than a bare metal application, but many applications have self-contained update systems and can connect to networks. Self-repairable infrastructure is a necessity, both in terms of tooling and staffing, for any organization for which an outage or a breach could be catastrophic, and the rise of centralized, cloud-reliant infrastructure in these contexts should be seen as a massive and unacceptable risk for those organizations to take on. Organizations being subject to unpatched vulnerabilities and inability to manage their systems competently are direct results of replacing internal competency and purpose-built systems with general-purpose systems maintained and controlled by unaccountable distant tech monopolies

siliconc0w 2y ago

Really, the problem is that all this critical infrastructure runs on Windows. Critical systems should effectively be appliances that run with a very minimal footprint. If you absolutely need to monitor them you can export disk snapshots or something out of band that can't impact operations.

cdchn 2y ago

If CrowdStrike's system wasn't able to prevent a kernel driver thats all zeros from getting by, you can be sure a malicious payload would have breezed right through.

chrisoverzero 2y ago
It wasn’t a driver.
therein 2y ago
Oh yeah, at a quick glance looks like that file could have had any payload and it would have been loaded right into the kernel.
- saagarjha 2y ago
  What makes you think so?
  - therein 2y ago
    Earlier there were some screenshots showing an entirely zero-filled .sys file but now we know that's not what the payload was.
- echoangle 2y ago
  To send a malicious payload into the kernel, you would have to take over crowdstrikes deploy infrastructure first, right? I hope the program doesn’t just accept updates from anywhere

bitexploder 2y ago

Umm, they (adversaries) already knew? Been in cybersecurity for 18 yrs. We told customers about issues like this all the time

dgoldstein0 2y ago

Crowdstrike has really redefined malicious compliance

simpaticoder 2y ago

Massive computer outage, worldwide affecting enterprises with Windows machines running CrowdStrike, a very popular software that is sold as hacking protection but which is, in reality, used by C-suite execs to spy on employee behavior. It is installed with extraordinary permissions and is difficult to fix or remove by design. I wonder if this will teach absolutely anyone a lesson about anything.

BodyCulture 2y ago
Can we please get more information about the spying features? Some screenshots would be great! Thanks!
- spydum 2y ago
  Not as evil as they make it sound. Process Execution, detailed timestamps, and network metadata capture are core features of every EDR tool (CrowdStrike, MDE, SentinelOne, etc) that exists. They can just be abused to monitor user behavior, in addition to threat hunting or malicious activity. Telemetry isn't inherently evil, but organizations need to establish privacy and usage governance around security tools to prohibit abuse.
skynr 2y ago
I think it will. MS has published a number that it was 8.5 million machines, which I don't believe, bur seeing the effort that's gone into the response even at my own relatively mid sized org, there are super simple questions like how the heck do we even get to these devices when hald the crew work remote. The response is and always will be - how much will this cost. We now have the opposing figure, how much will this cost if we don't do it.
- echoangle 2y ago
  Do you believe it was more or less than 8.5 million machines?
  - skynr 2y ago

johanneskanybal 2y ago

So yea let’s not use a company like this as best practice. Everything about this reeks of worst practices rising the wave of regulatory capture.

oneplane 2y ago

There is no "Digital Resilience" because that is perceived as too expensive, a cost center with hard to quantify value. So it's easier to try and carve out everything that doesn't fit into a spreadsheet, everything that isn't core business, and everything that is not able to present what value it generated. If general IT had the abilities of sales, marketing, or insurance, there might be a chance that the business would take the responsibility to have the internal knowledge and capabilities to assert control over their systems. But they don't, and as such they won't and instead shove that responsibility over to a third party generalist elsewhere with enough paperwork to have both parties feel their asses are covered. As long as everything seems to be working, the signals that are still getting through is project failures, be it complete failures or just time and/or money being consumed more than planned and maybe some requirements getting cut. But as soon as enough stuff breaks at the same time, we get news outlets writing articles about resilience and the greater public suddenly no longer agreeing with that is effectively just the result of the status quo because it impacts them directly.

sschueller 2y ago

If you are a non-US company you have to be insane to use this CrowdStrike service. The FBI can legally use a secret warrant[1] and force CrowdStrike to inject a DLL into your infrastructure! [1] https://en.wikipedia.org/wiki/United_States_Foreign_Intelligence_Surveillance_Court https://en.wikipedia.org/wiki/United_States_Foreign_Intellig...

autoexec 2y ago
You think they can't/don't do that to force Microsoft to push an "update" that does the same thing?
- jenscow 2y ago
  I doubt they'd even need to go through Microsoft
jml7c5 2y ago
Are you sure that is correct? I was under the impression that US government could order companies to turn over data, but that they could not compel them to actually do work. This was the center of the dispute between the government and Apple after the San Bernadino shooting: Apple was within their legal rights to refuse to provide assistance. https://en.wikipedia.org/wiki/Apple%E2%80%93FBI_encryption_dispute https://en.wikipedia.org/wiki/Apple%E2%80%93FBI_encryption_d... The lengths that the NSA and CIA would go to to implant backdoors (interdicting shipments of laptops/phones and doing the work themselves) further suggests that they cannot compel this sort of action.
- Fluorescence 2y ago
  That case was theatre / kayfabe. The FBI was using an emotive case for turning public opinion against encryption and set some legal precedent. The goal wasn't really to unlock the phone which could and was be done by other means. If they have a path to covertly compel action as a state secret under National Security / anti-terror laws we will only hear about it from whistleblowers. It won't be something the target can disclose let alone test in court. FWIW I also don't believe in Apple's nobility as resisting on user's behalf. They happily bow to the state and remove apps for e.g. organising protests, monitoring deaths in US wars, csam scanning etc. IMHO their interest in encryption is to prevent jail-breaking and protect their app-store cash cow. > interdicting shipments of laptops/phones and doing the work themselves I don't think that proves anything about their powers. Given the option, I'm sure they would prefer to install things themselves without third-party knowledge or consent. We have evidence of complicit action e.g. black rooms like Room 641A. I think the nature of "consent" and "obligation" gets pretty grey when it comes to the security agencies. They don't get results using court orders. I'm sure they have assets employed as staff in security sensitive positions.

notepad0x90 2y ago

In a twisted way, Crowdstrike just gave western civilization a disaster recovery and resilience forced test. an actual attack won't be rolled back within an hour. In case you don't know, Crowdstrike is hardly the only company with large scale access to this many companies,governments and resources. It takes one rogue employee to deploy a disk wiper that destroys every computer (including linux and macos) and affected systems won't recover at all. it would be months before critical systems are back online, the global economy would come to a halt worse than how it did with COVID in such a scenario. It isn't "why didn't Crowdstrike do better" (although they should have), it is more, why isn't technology in critical systems more resilient to one vendor screwing up or getting hacked? For example, let's say it wasn't just a boot loop but a disk wiper erased every boot disk, is there any reason pxe booting a recovery image or a backup image configured already on servers, atms, kiosks, point of sale systems,etc...? even if UEFI and bios were erased, it is technically not impossible to have an auto-recovery mechanism implemented right? If you have never been in an incident response (IT and security incidents) root cause analysis, I don't blame you for not thinking deeper about the root cause, but that is the type of root cause analysis that has been missing despite over a decade of rampant ransomware, disk wipers, and supply chain risks. Finding someone to blame and be angry at is easy and doesn't solve the root cause. Making hard technical decisions and not wasting this opportunity (never waste a good crisis) to push for resilient technology investments actually solves the root cause behind this and other repeating problems.

yusyusyus 2y ago
if the firmware is totally nuked, you'd need backup firmware. at some point, all of this crap can be made non-recoverable, but that isn't the real problem to solve. imma take your comment one step further and say that the emphasis on security is coming at the expense of discussions on resilience. and security matters a lot less, especially financially, than resilience.
- notepad0x90 2y ago
  availability is one of the core tenets of security. Security = a measurement of confidentiality, integrity and availability. Backup firmware and boot images can be configured as read only.

thedataexchange 2y ago

1. It’s a good time to reread the article that got Dan Geer famous on “monocultures” => https://ccianet.org/wp-content/uploads/2003/09/cyberinsecurity%20the%20cost%20of%20monopoly.pdf https://ccianet.org/wp-content/uploads/2003/09/cyberinsecuri... 2. Also a great time to start prepping for AI Incidents => https://thedataexchange.media/ai-incident-response/ https://thedataexchange.media/ai-incident-response/

m3kw9 2y ago

the problem is that for a security scanner to scan threats properly, they need to sit on the kernel, there should be a mode where they allow scanners to read but is not able to crash the system. Some sort of sand box for all these kernel access

kchr 2y ago
> Some sort of sand box for all these kernel access Yeah, like eBPF in Linux or System Extensions in macOS.

Timber-6539 2y ago

The only vulnerability here was CrowdStrike's EDR product that runs exclusively in ring 0 and the entire corporate & technical class that lazily relied on this flawed security model and centalized this incompetence. As much as some people want to believe that Microsoft is blameless here, I hold them partly responsible. They need to create a stable API in their kernel and force third party security vendors to use it.

linearrust 2y ago

> It was, by all appearances, purely human error — a few bad keystrokes that demonstrated the fragility of a vast set of interconnected networks in which one mistake can cause a cascade of unintended consequences. Cute. It's always those bad keystrokes. If only these crowdstrike employees worked on their good keystrokes that morning. I blame management. > Russian hackers working on behalf of Vladimir V. Putin bring down hospital systems across the United States. In others, China’s military hackers trigger chaos, shutting down water systems and electric grids to distract Americans from an invasion of Taiwan. ... Among Washington’s cyberwarriors, the first reaction on Friday morning was relief that this wasn’t a nation-state attack. For two years now, the White House, the Pentagon and the nation’s cyberdefenders have been trying to come to terms with “Volt Typhoon,” a particularly elusive form of malware that China has put into American critical infrastructure. So we have cyberwarriors and cyberdefenders? And the russians, china, etc have 'hackers'. If ever there was a doubt what the nytimes really is. > The fear is, in an election year, that the next digital meltdown may have a deeper political purpose. Oh dear. More bad keystrokes on the way? Did anyone glean anything of value from the article? There was a lot of words but no substance.

pm90 2y ago

> It is not hopeless. > “We are optimistic that A.I. is actually allowing us to make significant — not transformative yet, but significant — progress in being able to identify vulnerabilities, patch holes, improve the quality of coding,” Kent Walker, the president for global affairs at Google, said at the Aspen forum. I disagree. If the only hope is some vague promise of bs AI, there is no hope indeed.

pards 2y ago
Yep. This is like arguing that the best way to prevent school shootings is to give teachers guns. I don't think AI would have done a better job convincing CrowdStrike's leadership team that a staged rollout strategy is worth the cost. These issues are caused by people, not technology, so more technology isn't going to fix it.

someonehere 2y ago

Can someone from one of the major services comment on why they don’t run the N -1 policy on Falcon? My onboarding sales engineer recommended this to me years ago to avoid this situation. Why do critical infrastructure companies run bleeding edge updates like this?

naveen99 2y ago

Or it’s just a front for the nsa and cia.

shirro 2y ago

I haven't worked in a Windows environment for a long time so was a little surprised how much of the online commentary suggests people in that environment are comfortable or at least resigned to the necessity of unattended live third party updates on critical infrastructure. I can't see any justification for that on the *nix side of things and hope that culture never transfers over.

Osiris 2y ago
I have worked at a few Windows places and they did not allow live updates. All updates were managed by IT and tested before being pushed out. This situation was caused basically by an anti virus definition update. Without any information, my guess is that CrowdStrike specifically doesn't provide a means for enterprises to manage the CrowdStrike updates because that would cause potentially weeks or months of delays for critical anti virus updates to be released. I'm more curious why they need CrowdStrike on every system in the first place. I can understand employee computers but servers and critical systems should have other security measures in place to make them less available to attack in the first place.
- smileybarry 2y ago
  > I'm more curious why they need CrowdStrike on every system in the first place. I can understand employee computers but servers and critical systems should have other security measures in place to make them less available to attack in the first place. The boring answer(s) is compliance and necessity. You need a security solution installed & running for a bunch of certifications, and maybe you also need it to secure some unmaintained software you have to run. Plus, even if the machine isn't front-facing, a lot of these EDR solutions are installed to prevent or monitor for lateral movement.

mrjin 2y ago

The fire more deadly than enemy fire is friendly fire. For adversaries, they cannot do any harm unless they get in, even if they get in, the damage is limited to the access of the account they run on. But for AVs, they are invited in, which renders the 1st line of defense useless. Making it worse, they are running with SYSTEM privileges, which is higher than Admin privileges. And we just witnessed what could happen if AVs went rogue.

whjkh 2y ago

[flagged]