13 ms·

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdSt

by zmmmmm 2y ago

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer starting up and then in the critical path of every network connection the computer makes. The risk of ZScaler being a central point of failure is not considered. But - the risk of failing the compliance checkbox it satisfies is paramount. All over the place I'm seeing checkbox compliance being prioritised above actual real risks from how the compliance is implemented. Orgs are doing this because they are more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting. So we need to hold regulatory bodies accountable as well - when they frame regulation such that organisations are cornered into this they get to be part of the culpability here too.

wparad 2y ago
That for sure is a great point, it's actually something Dorota brought up in her talk this year https://vimeo.com/940487124 https://vimeo.com/940487124
SuperSandro2000 2y ago
Sounds familiar
RedShift1 2y ago
Don't put your eggs in one basket, I use multiple anti-virus products so that if one blows up at least not all computers are affected. Looks like my old wisdom is still new wisdom. Clarification: I mean that every computer has one anti-virus product, but not every computer has the same anti-virus product. I'm not installing multiple anti-virus products on the same computer.
- cqqxo4zV46cp 2y ago
  This is much easier applied personally than it is to a 30k person organisation. No need to be condescending.
  - Twisell 2y ago
    In both case it's costly. But cost of maintenance aside it wouldn't be that bad to deploy each half the fleet with two distincts EDR. This is actually implicitly in place for big companies that support BYOD. If half your fleet is on Windows another 40% on MacOs and 10% on Linux you need distinct EDR solutions and a single issue can't affect all your fleet at once.
- NDizzle 2y ago
  You use multiple anti-virus products. Let's assume you use 3. Do you have multiple clusters of machines, each running their own AV product, so in case one has this problem the other two are unaffected? How much overhead are we talking about here? Because if you're just using multiple AV software installed on one machine, 1) holy shit, the performance penalty, 2) you'd still be impacted by this, as CS would have taken it down.
  - mschuster91 2y ago
    Not the person you're replying to, but in any reasonable organization with automated software deployment it should be easy to pool machines into groups, so you can make sure that each department has at least one machine that uses a different anti-virus software. Bonus, in case you do catch a malware, chances are higher that one of the three products you use will flag it.
    - NDizzle 2y ago
      Again, "should be" academic stuff. So you have multiple AV products and you target those groups. You have those groups isolated on their own networks, right? With all the overhead that comes with strict firewall rules and transmission policies between various services on each one. With redundant services on each network... you've doubled or tripled your network device costs solely to isolate for anti virus software. So if only one thing finds the zero day network based virus, it won't propagate to the other networks that haven't been patched against this zero day thing. How far down the rabbit hole do we want to go? If you assume many companies are doing this kind of thing, or even a double digit percentage of companies, I have bad news for you.
  - ta1243 2y ago
    They surely mean that all odd number assets are running crowdstrike and even are running sential-one (or similar, %3, %4, etc etc). At least then you only lose half your estate.
    - RedShift1 2y ago
- fire_lake 2y ago
  If you have all of them on a critical path then your risk of blow up increases!
- deleted 2y ago
  [deleted]
- xorcist 2y ago
  Not sure that's a great idea. This stuff tends to have very high privilege access. It's enough that one of your anti virus vendors get hacked for your whole organization to get owned...
  - interludead 2y ago
    I see one main challenge - it can increase the incidence of false positives.
- Lovesong 2y ago
  Was there any situation where having 3 anti-virus was more beneficial than having only 2?
  - ben_w 2y ago
    I'm reading this as first third of computers have AV brand A, second third have brand B, remainder have brand C. Thus, if brand A does something actively harmful all by itself, only 1/3rd of machines are impacted. This is an improvement on having only 2 brands, as having 1/3rd of your machines go down is better than having 1/2 of your machines go down.
- interludead 2y ago
  Indeed a wise strategy
baxtr 2y ago
hey, but at least (a) we have a process (b) we documented it and (c) we review it regularly! what can go wrong?!
- rubatuga 2y ago
  There should be a new term called compliance hell.
  - c0l0 2y ago
    The term already exists, but the "hell" is mostly silent (even in writing).
    - baxtr 2y ago
      Is there a process for that??
theallan 2y ago
The flip side is, if you don't do auto updates and an exploit is published and used against you and you haven't yet tested / pushed the patch, that you would have been protected against if it had auto updated, you are up the creak without a paddle in that situation as well. To some degree you have to trust the software you are using not to mess things up.
- cqqxo4zV46cp 2y ago
  Yes. And the vast majority of the time, it doesn’t mess things up. The notion that you may take on risk to net alleviate risk is somehow lost on a lot of people in these conversations.
- zmmmmm 2y ago
  So since I do mission critical healthcare I do run into this concept. But it's not as unresolvable as you portray. Consider for example HIPAA "break the glass" requirement. It says that whatever else you implement in terms of security you must implement a bypass that can be activated by routinely non-authorised staff to access health information if someone's life is in danger. Similarly, when I questioned, "why can't users turn off ZScaler in an emergency" we were told that it wouldn't be compliant. But it's completely implementable at a technical level (Zscaler even supports this). You give users a code to use in an emergency and they can activate it and it will be logged and reviewed after use. But the org is too scared of compliance failure to let users do it.
  - jenscow 2y ago
    While I agree with the requirement, but it sounds like a vault would need to have an unlocked door with a sign.
    - ivan_gammel 2y ago
      A vault with firearms in the police station to which every staff member has a key. Sounds reasonable to me. Users are not prisoners left in the burning building without a fire escape.
tempaway4575144 2y ago
You are completely right. BTW It wasn't a software update, it was a content update, a 'channel file'. Someone didn't do enough testing. edit: or any testing at all? https://x.com/George_Kurtz/status/1814235001745027317 https://x.com/George_Kurtz/status/1814235001745027317 https://x.com/brody_n77/status/1814185935476863321 https://x.com/brody_n77/status/1814185935476863321
- glimshe 2y ago
  It passed all unit tests!
  - livrem 2y ago
    It passed the type checker!
- Shank 2y ago
  It's an automatic update of the product. Semantic "channel vs. binary" doesn't indicate anything. If your software's definition files can cause a kernel mode driver to crash in a bootloop you have bigger problems, but the outcome is the same as if the driver itself was updated.
  - tempaway4575144 2y ago
    Indeed. Its worse really, it means there was a bug lurking in their product that was waiting for a badly formatted file to surface it. Given how widespread the problem is it also means they are pushing these files out without basic testing. edit: It will be very interesting to see how CrowdStrike wriggle out of the obvious conclusion that their company no longer deserves to exist after a f*k up like this.
    - account42 2y ago
      Simple: They are obviously too big to fail now.
bonoboTP 2y ago
> more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting. Duh, else there would be no need to audit them to force compliance, they'd just do it by themselves. The only reason it needs forcing is that they otherwise aren't motivated enough.
- lucianbr 2y ago
  Good point. But the audit seems useless now. It's supposed to prevent the carelessness from causing... this thing that happened anyway. Sure, maybe it prevented even more events like this from happening. But still.
  - felurx 2y ago
    To be fair, I'd claim that it's pretty rare for anything anyone ever does to not be a trade-off.
  - sobkas 2y ago
    > Good point. But the audit seems useless now. It's supposed to prevent the carelessness from causing... this thing that happened anyway. > Sure, maybe it prevented even more events like this from happening. But still. Because the point of audit is not to prevent hacks, it's to prove that you did your due diligence to not get hacked, so fact that hack happened is not your fault. You can hide under umbrella of "sometimes hacks happen no matter what you do".
    - lucianbr 2y ago
      CYA is the reason you do the audit. But the reason for the audit's existence and requirement is definitely so that hacks don't happen. Don't tell me regulatory agencies require things so that companies can hide behind them.
      - bonoboTP 2y ago
        The reason for the audit's existence is CYA one level above. The chain ends with a politician's CYA in front of the electorate.
stefan_ 2y ago
When we think "security" on HN we think about the people who escalate wiggling voltages at just the right time into a hypervisor shell on XBox, but I've had to recognize that my learned bias is not correct in the real world. In the real world, "computer security" is a profession full of hucksters that can't tell post-quantum from heap and whose daily work of telling people repeatedly to not click links in Outlook and filling out checklists made by people exactly like them has essentially no bearing on actual security of any sort.
- Shank 2y ago
  It's driven by a lot of things. Part of it is driven by rising cyber liability insurance rates, for one. A lot of organizations would rather not pay for CrowdStrike, but the premiums for not having an "EDR/XDR/NGAV" solution can be astoundingly high at-scale. Fundamentally there's a lot of factors in this ecosystem. It's really wild how incentives that seem unrelated end up with crazy "security" products or practices deployed.
  - teeray 2y ago
    > A lot of organizations would rather not pay for CrowdStrike, but the premiums for not having an "EDR/XDR/NGAV" solution can be astoundingly high at-scale. Just like a lot of homeowners would rather not pay for ADT, but insurance requires a box-ticking “professionally-monitored fire alarm system.” Nevermind that I can dial 911 as well as the “professional” when I get the same notification as they do.
- l33t7332273 2y ago
  I’ve met highly paid “security engineers” that talked about not really being into programming or being okay with python but everything else is too complicated. It shocks me that such a low level of technical competence is required.
- tristor 2y ago
  > In the real world, "computer security" is a profession full of hucksters Always has been. The information security model is about analogizing digital systems as physical systems, and employing the analogues of those physical controls that date back hundreds of years on those digital systems. At no point, in my relatively long career, have I ever met anyone in Information Security who actually understands at depth anything about how to secure digital systems. I say this as someone who has spent a lot of my career trying to do information security correctly, but from the perspective of operations and software engineering, which is where it must start. The entire information security model the world works with is tacking on security after the fact, thinking you need to builds walls and a vault door to protect the room after the house has already been built, when in fact you need to build the house to be secure from the start because attacks don't go through doors, attacks are airborne (I recognize the irony of my analogizing digital concepts to physical concepts surrounding security, but I do it because of any infosec people that may read my comment so they can understand my point). Because of this model, we have gone from buying "boxes" to buying "services", but it has never matured away from the box-checking exercise it's been since day one. In fact, many information security people have /no training or education/ in security, it's entirely in regulatory compliance.
2-3-7-43-1807 2y ago
I don't even want to know how many mission critical systems automatically deploy open source software downloaded from github or (effectively random) public repositories.
- rini17 2y ago
  Unlike Windows, there is at least the option to use curated software distributions such as Debian or RH that won't apply random stuff from upstream repositories.
  - 2-3-7-43-1807 2y ago
    I'm talking about all sorts of software projects implemented using especially Python, Ruby, node.js etc.
  - lmz 2y ago
    I like Debian but it's not like they need random upstream repositories when they can make random patches themselves e.g. the OpenSSL Purify issue.
- interludead 2y ago
  These risks must be carefully managed that's it I think
higeorge13 2y ago
So true! It seems like all of these were invented to create another market for b2b saas security, audit, monitoring, etc. companies. Nobody cares about actual security or infrastructure anymore. Everything is just buying some subscription for random saas companies, not checking their permissions and grant policies and ticking boxes because... compliance.
kmlx 2y ago
> the risk of failing the compliance checkbox it satisfies is paramount. i'm curios as to what compliance is there to be satisfied to necessitate such a hardcore measure?
- rcxdude 2y ago
  There's a reasonable number of circumstances where there are cybersecurity standards that get imposed on organisations: insurance, from a customer, or from the government (especially if they are a customer). These standards are usually fairly reasonably written, but they are also necessarily vague and say stuff like "have a risk assessment", and "take industry-standard precautions". This vagueness can create a kind of escalation ratchet: when people tasked with (or responsible for) compliance are risk-averse and/or lazy, they will essentially just try to find as many precautions as they can find and throw them all in blanket-style, because it's the easiest and safest way to say that you're in compliance. This is especiallly true when you can more or less just buy one or two products which promise to basically tick every possible box. And if something else pops up as a suggestion, they'll throw that in too. Which then becomes the new 'industry standard', and it becomes harder to justify not doing it, and so on.
- victorbjorklund 2y ago
  I worked in orgs where customers put a certain security standard in the contract. So if you fail that you are kind of in breach of contract
izacus 2y ago
Of course, disabling those auto updates will have you fail the external security audit and now your security team needs to fight with the rest of the leadership in the company explaining why you're generating needless delays, costs against the "state of the art in security industry" and why your security guys are smarter than the people who have the power to approve or deny your security certification.
- chii 2y ago
  > you fail the external security audit aka, you fail the cover-your-ass security, rather than actual security.
  - izacus 2y ago
    yep, but "trust us, we're secure, pinky promise by our internal employees" doesn't really work either.
    - ryandrake 2y ago
      Don't forget the press releases all saying "We take security very seriously!"
  - StefanBatory 2y ago
    Employees would rather care for their employment and keeping boss happy rather than going against their orders.
- lysp 2y ago
  Or lose IT/security insurance for not installing or disabling it.
  - nlitened 2y ago
    Well, let’s see how much insurance companies will pay now
zh3 2y ago
"The metric becomes the measure", i.e. Goodhart's Law. https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
belter 2y ago
> So we need to hold regulatory bodies accountable as well - when they frame regulation such that organisations are cornered into this they get to be part of the culpability here too. No, we need to hold Architects accountable, and this is the core of the issue. Creating systems with single, outsourced responsibility, in the critical path.
- ta1243 2y ago
  As a CTO, when your company goes down you get fired. When every company goes down you get let off. The sensible thing is follow the herd and centralise. You're outsourcing the risk to your own job.
  - zer00eyz 2y ago
    This is the point of much of the security efforts we see now. Outsourcing of security functions, and things like login push a lot of liability and legal issues off into someone else's house. It's hard to be the source of a password leak, or be compromised when you don't control the passwords. But like any chain your only as secure as your weakest link... Snowflake is a great current example of this. Mean while the USPS just told us "oops" we had tracking pixels for a bunch of vendors all over our delivery preview tool. Candidly, most people stacks look a lot less like software and more like a toolbar riddled IE5 install circa 2000. I don't think our industry is in a good place.
  - belter 2y ago
    > When every company goes down you get let off. And when only companies who use a certain OS and/or Cloud vendor go down? ;-) Do you also get let off?
    - ta1243 2y ago
      If it's large enough. Nobody loses their job when office 365 goes down, even if that happens once a year. However if you decided to choose a small company which goes down once every 5 years, you're screwed.
  - 2y ago
lelanthran 2y ago
> The risk of ZScaler being a central point of failure is not considered. But - the risk of failing the compliance checkbox it satisfies is paramount. You're conflating Risk and Impact, and you're not considering the target of that Risk and that Impact. Failing an audit: 1. Risk: high (audits happen all the time) 2. Impact to business: minimal (audits are failed all the time and then rectified) 3. Impact to manager: high (manager gets dinged for a failing audit). Compare with failing an actual threat/intrusion: 1. Risk: low (so few companies get hacked) 2. Impact to business: extremely high 3. Impact to manager: minimal, if audits were all passed. Now, with that perspective, how do you expect a rational person to behave? [EDIT: as some replies pointed out, I stupidly wrote "Risk" instead of "Odds" (or "Chance"). Risk is, of course, the expected value, which is probability X impact. My post would make a lot more sense if you mentally replace "Risk" with "probability".]
- Freak_NL 2y ago
  Almost correct, but you mean 'chance' where you write 'risk': Risk = Chance × Impact The chance of failing an audit initially are high (or medium, present at least). The impact is usually low-ish. It means a bunch of people need to fix policy and set out improvement plans in a rush. It won't cost you your certification if the rectification is handled properly. It's actually possible that both of your examples are awarded the same level of risk, but in practice the latter example will have its chance minimized to make the risk look acceptable.
  - codetrotter 2y ago
    Chance has more positive connotations than it has negative connotations IMO. Probability is a more neutral word, and fits better.
- noahmasur 2y ago
  I think the parent was correct in the use of the word "Risk"; it's different than your definition, which appears to be closer to "likelihood". Risk is a combination of likelihood and impact. If "risk" were just equivalent to "likelihood" then leaving without an umbrella on a cloudy day would be a "high-risk situation". A rational person needs to weigh both the likelihood and impact of a threat in order to properly evaluate its risk. In many cases, the impact is high enough that even a low likelihood needs to be addressed.
cm2012 2y ago
[flagged]
p0seidon 2y ago
This is 100% the reality
monkeydust 2y ago
"All over the place I'm seeing checkbox compliance being prioritised above actual real risks from how the compliance is implemented." Great statement and one that needs to be seriously considered - would DORA regulation in the EU address this I wonder? Its a monster piece of tech legislation that SHOULD target this but WILL it - someone should use todays disaster and apply it to the regs to see if its fit for purpose.
- piker 2y ago
  Broken regulations? Fix by adding more!
  - idle_zealot 2y ago
    Well, yeah. If a regulation is broken and not achieving its goal it should be changed. What's the alternative? "Regulation? We tried that once and it didn't work perfectly, so now we let The Market™ sort out safety standards."
    - InsideOutSanta 2y ago
      Who needs regulation when you can have free Fentanyl with your CrowdStrike subscription! All of your systems will go down, but you won't care, and the chance of accidental overdose is probably less than 10%!
      - jddj 2y ago
        The child slave labour is what really gets the deal across the line
    - piker 2y ago
      Yes, in many contexts that may well be the correct conclusion. Your comment presumes that regulation here has proven itself useful and not resulted in a single point of failure which potentially reduces overall safety. It’s of course the correct comment from a regulator’s perspective.
dijit 2y ago
It's possible that CrowdStrike heavily incentivises being left to update itself. Removing the features that would allow sysadmins to actually do it automatically, even via the installer itself- would definitely be one way, but another one could be aggressive focus-stealing nags (similar to Windows' own nags) which in a server environment can actually cause some major issues, especially when automating processes in Windows (as you need to close the program when updating). I think it's easy to blame the sysadmins, but I would also be remiss if I didn't point out that in the Windows world we have been slowly accepting these automatic dark patterns and alternative (more controlled) mechanisms have been removed over time. I almost don't recognise the deployment environment today as to what it was in 2004; and yes, 20 years is a long time, but the total loss of control over what a computer is doing is only going to make issues like this significantly more common.
- mr_mitm 2y ago
  They say it was caused by a faulty channel file. I don't know what a channel file is, and they claim to not rely on virus signatures, but typically anti virus product need the latest signatures all the time and poll them probably once an hour or so. So I'm not surprised that an anti virus product wants to stay hyper updated and updates are rolled out immediately to everyone globally.
  - HereBeBeasties 2y ago
    No, I'm not surprised either. But if you're operating at this kind of scale and with this level of immediate roll-out, what I would expect are: * A staggered process for the roll-out, so that machines that are updated check-in with some metrics that say "this new version is OK" (aka "canary deployment") and that the update is paused/rolled back if not. * Basic smoke testing of the files before they're pushed to any customers * Validation that the file is OK before accepting an update (via a checksum or whatever, matched against the "this update works" automated test checksums) * Fuzz tests that broken files don't brick the machine Literally any of the above would have saved millions and millions of dollars today.
- bandrami 2y ago
lucasRW 2y ago
Electricity too is a SPOF.
- RobotToaster 2y ago
  Most mission critical medical systems have a backup generator.
  - XorNot 2y ago
    And IT has dual power supply servers, and endpoint UPSs. Electricity is aggressively made redundant for mission-critical systems.
jrockway 2y ago
Zscaler is truly amazing. It can't do HTTP/2. Our product is HTTP/2-only. So we can't use our own product at work.
- Avamander 2y ago
  Their visibility and process in general for handling abuse of their services is also abysmal.
- mrweasel 2y ago
  I know a few people who have Zscaler deployed at work. It will routinely kick them of the internet, like multiple times a day. It has gotten to the point where they can sort of tell in advance that it's about to happen. The theory so far it that it's related to their activities, working in DevOps they will sometimes generate "suspicious" traffic patterns which will then trigger someone policy in Zscaler, but they're not actually sure.
- supertrope 2y ago
  ZScaler itself uses port 443 UDP, but blocks QUIC. The last time I checked it didn't support IPv6 so they told customers to disable IPv6. Security software is legacy software out of the box and cuts the performance of computers in half.
- pwarner 2y ago
  What Zscaler can and will do though is break your network randomly and in strange ways. They don't even seem to charge for that feature!
grishka 2y ago
If I were running an organization that needs these audits, I'd always have fallback procedures in place that would keep everything running even if all computers suddenly stop working, like they did today. General-purpose software is too fragile to be fully relied upon, IMO. If a general-purpose computer must be used for something mission-critical, it should not have an internet connection and it should definitely not allow an outside organization to remotely push arbitrary kernel-mode code to it. It should probably also boot from a read-only OS image so that it could always be restored to a known-good state by just rebooting.
- sys_64738 2y ago
  Organizations don't want to increase risk by listening to an employee with their personal opinion. Orgs want an outside vendor who they can point at and say "it's their fault", and await a solution. Employees going rogue and not following the vendor defined SW updates is a much higher risk than this particular crisis.
sobkas 2y ago
It depends on what your position is. Are you there to actually provide security to your org or to tick a in an audit. If both which is more important. Because failing an audit have real consequences, while having breaches in security have almost none. Just look at credit score companies.
the8472 2y ago
Regulation or auditors rarely require specific solutions. It's the companies themselves that choose to achieve the goals by applying security like tinctures: "security solutions". The issue is that the tinctures are an approved remedy.
lqet 2y ago
Can anyone explain how CrowdStrike could possibly fix this now? If affected machines are stuck in an endless BSOD cycle, is it even possible to remotely roll out a fix? My understanding is that the machines will never come to the point where a CS update would be automatically installed. Is the only feasible option the official workaround of manually deleting system files after booting into the recovery environment? How could this possibly be done on scale in organizations with tens of thousands of machines?
- ExoticPearTree 2y ago
  CrowdStrike itself will not fix anything. They published a guide on how to workaround the problem and that's it. Most likely a lot of sales reps and VPs will be fielding calls all over the weekend explaining large customers how did they manage to screw up and how much discount will they offer on the next renewal cycle. Legally, I think somewhere in their license it says is that they're not responsible in any way or form if their software malfunctions in any way.
  - delfinom 2y ago
    Small companies get the shitty generic license. Big companies negotiate liability terms.
    - betaby 2y ago
      > Big companies negotiate liability terms. Never heard that in the context of the software licenses.
      - rayrey 2y ago
        Pretty standard in enterprise b2b, most of the sales cycle is in contracts
        betaby 2y ago
        Well, probably no? I've never seen liabilities in dollar value, or rather any significant value. Also I saw our company Ceowdstrike contract for 10k+ seats, no liabilities there.
jojobas 2y ago
The question is, how did they manage to not crash everything for so long without a staged/rolling update deployment strategy?
- chrisjj 2y ago
  Perhaps it took this long for the offending file to pass 64K Excel rows... :)
fireflash38 2y ago
That's some very twisted logic. If I expect someone to clean the kitchen as part of restaurant closeup checklist, and they fuck it all up, would I blame the checklist, or the person doing the work? You blame the person fucking it up. In this case, it's someone who only cares about checking a box. Or someone who pushes broken shit.
- lqet 2y ago
  If this person simultaneously fucks up millions of kitchens around the world, you do not blame that person. You blame the checklist which encouraged giving a single person global interlocked control over millions of kitchens, without any compartmentalization.
  - fireflash38 2y ago
    > If this person simultaneously fucks up millions of kitchens around the world, you do not blame that person. No, you definitely do, even more than before. Let's say for example that the requirement is to disinfect anything that touches food. And the bleach supplier fucks it all up. You blame the bleach supplier. You don't throw out the disinfectant requirement.
ExoticPearTree 2y ago
> All over the place I'm seeing checkbox compliance being prioritized above actual real risks from how the compliance is implemented. Because if everyone is doing their job and checks their box, they're not gonna get fired. Might be out of a job because the company goes under, but hey, it was no one's fault, they just did their job.
dagaci 2y ago
Most enterprises will have teams of risk and security people. They will be asking who authorized deployment of an untested update into production. If CrowdStrike deployments cannot be managed, then they will switch to a product which can be managed.
basisword 2y ago
>> It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. While orgs using auto update should reconsider, the fact that CrowdStrike don't test these updates on a small amount of live traffic (e.g. 1%) is a huge failure on their part. If they released to 1% of customers and waited even 24 hours before rolling out further this seems like it would have been caught and had minimal impact. You have to be pretty arrogant to just roll out updates to millions of customers devices in one fell swoop.
- lqet 2y ago
  Why even test the updates on a small amount of live customers first? Wouldn't this issue already have surfaced if they tested the update on a handful of their own machines?
  - basisword 2y ago
    I would hope they've done that and it passed internal QA. But maybe not a good idea to assume they're doing any sort of testing at all.
    - greggsy 2y ago
      They’d have rigorous test harnesses, but you can’t really account for the complexities of a highly configurable platform like Windows.
      - kristjansson 2y ago
        The prevalence of the issue makes it seem unlikely to have been caused by site-specific configurations
shadowgovt 2y ago
Consolidation / optimization of labor. When Crowdstrike messes up and BSODs thousands of machines, they have a dedicated team of engineers working the problem and can deliver a solution. When your company gets owned because you didn't check a compliance checkbox, it's on you to fix it (and you may not even currently have the talent to do so). We see similar risk tradeoffs in cloud computing in general; yes, hosting your stuff on AWS leaves you vulnerable to AWS outages, but it's not like outages don't happen if you run your own iron. You're just going to have to dispatch someone a three hour drive away to the datacenter to fix it when they do.
- deleted 2y ago
  [deleted]
- chrisjj 2y ago
  > they have a dedicated team of engineers working the problem and can deliver a solution. No. Can merely facilitate the customer's on-site admin to deliver a solution.
afh1 2y ago
Isn't there a way to schedule the updates? With Windows updates, when I used to work at a firm with a critical system running on Windows, we had main and DR servers and the updates were scheduled to first rollout on the main server and a day after I think at the DR, which has saved us at least once in the past from a bad Windows update...
- vladvasiliu 2y ago
  More or less. You can set up some update policies which and apply those to subsets of your machines. You can disable updates during time blocks, or block them altogether. There's also the option of automatically installing the "n-1" update. We run auto n-1 at work, but this also happened at the same time on my test machine with runs "auto n". It never happened before, so this looks like something different than the actual installed sensor version, especially since the latest version was released something like a week ago.
jimnotgym 2y ago
> So we need to hold regulatory bodies accountable as well... My bank, my insurer, my payment card processor, my accounting auditor and probably others may all insist I have anti-virus and insist that it is up to date. That is why we have to have these systems. However, I used to prefer systems that allowed me to control the update cycle and push it to smaller groups.
bhbh 2y ago
CrowdStrike has various auto update policies, including not to automatically update to the latest version, but to the latest version -1 or even -2. Customers with those two policies are also impacted.
lenkite 2y ago
It's a big stretch to call this the regulator's fault when its basic lack of testing by Microsoft and/or Crowdstrike. If a car manufacturer made safety belts that broke, you don't blame the regulators. The root cause is automatic, mindless software update without proper testing - nothing to do with regulators.
beeboobaa3 2y ago
It's what you get when you let luddites, also known as managers, make the rules.
number65259 2y ago
> My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer starting up Hi fellow CVS employee. Are you enjoying your zscaler induced SSO outages every week that torpedo access to email and every internal application? Well now your VMs can bluescreen too. A few more vendor parasites and we'll be completely nonfunctional. Sit tight!
Puts 2y ago
This is an oversimplification. IF we are talking about compliance to ISO 27001 you are supposed to do your own risk assessment and implement necessary controls. The auditor will basically just check that you done the risk assessment, and that you have done the controls you said yourself you need to do. I'd say this has nothing with regulatory compliance to do at all. The real truth is that modern organizations are way too attached to cloud solutions. And this runs across all parts of the organization with Saas and PaaS whether it's email (imagine Google Workspace having a major issue), AWS, Azure, Okta… I've had the discussions so many times and the answer is always – the risks doesn't matter because the future is cloud and even talking about self hosting anything is naive and honestly we need to evaluate your competence for even suggesting it. (Also the cloud would maybe not be this fragile if it wasn't for lock-in with different vendors. If you read the TOS it says basically on all cloud services that you are responsible for the backup – but getting your data out of the service is still pain in the ass – if possible at all)
- crazygringo 2y ago
  > The real truth is that modern organizations are way too attached to cloud solutions. I'm confused. This is a security product for your local machine. Not the cloud. Unless you call software auto-update "the cloud", but that's not what people usually mean. The cloud isn't about downloading files, it's about running programs and storage remotely. I mean, if CloudStrike were running entirely on the cloud, it seems like the problem would be vastly easier to catch immediately and fix. Cloud engineers can roll back software versions a lot easier than millions of end users can figure out how to safe boot and follow a bunch of instructions.
  - Puts 2y ago
    Well, in all times usually there has been the option to run a local proxy/cache for your updates so that you can properly test them inside your own organization before rolling them out to all your clients (precisely to avoid this kind of shit show). But doing that requires an internal team running it and actually testing all updates. But modern organizations don't want an IT-department, they want to be "cloud first". So they rely on services that promise they can solve everything for them (until they don't). Cloud is not just about where things are – it's also about the idea that you can outsource every single piece of responsibility to a intangible vendor somewhere on the other side of the globe – or "in the cloud".
coliveira 2y ago
This is all well deserved for executives who trust MS to run their businesses. If you have the resources, like a bank, it is crime to put your company in the hands of MS.
- alecco 2y ago
  IME, Boomer managers refuse to use anything but Windows. We have a few more years of this.
metalliqaz 2y ago
this is the world that lawyers gave us
pookha 2y ago
It's easy to blame CrodStrike because they're the ones to blame here. They lit a billion system32 folders on fire with an untested product and push out fear mongering corny marketing material. Turns out you should be afraid.
MetaWhirledPeas 2y ago
> Orgs are doing this because they are more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting. If a failed audit is the big scary monster in their closet, then it sounds like the senior leadership is not intimately familiar with the technology and software in general, and is unable to properly weigh the risks of their decisions. More and more companies are becoming software companies whether they like it or not. The software is essential to the product. And just like you would never want a non-lawyer running your law firm, you don't want a non-software person running your software company.
temporarely 2y ago
> So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. TIL that US government has pressured foreign nations to install a mystery blob in the kernel of machines that run critical software "for compliance". If this wasn't a providential goof on the part of Crowdstrike -- the entire planet is now aware of this little known fact -- then some helpful soul in Crowdstrike has given us a heads-up.
narrator 2y ago
No doubt there will be another piece of software mandated to check up on the compliance software. When that causes a global IT outage, software that checks up on the software that checks up on the compliance software will be mandated.
radiator 2y ago
Very sharp and to the point, this comment. I would like to add that in large companies the audit will, in my experience, very often examine documents only -- not actual configuration or code.
phone8675309 2y ago
> Orgs are doing this because they are more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting. I've been someone in one of those audit meetings defending decisions made and defending things based on the records we keep and I understand this because it is both a deeply unpleasant and expensive affair to pull people from current projects and place them before auditors for several hours to debate what compliance actually means.
maxrecursion 2y ago
I work for government organization that is constantly audited and I've seen this play out over and over. An important aspect I never see mentioned is most Cyber Security personnel don't have the technical experience to truly understand the systems they are assessing, they are, like you said, just pushing to check those compliance boxes. I say this as someone who is currently in a Cyber Security role, unfortunately, as I'm coming to learn cyber roles suck. But this isn't a jab at those Cyber Security personnel's intelligence. It's literally impossible to understand multiple systems at a deep level, it takes employees working on those systems weeks to months to understand this stuff, and that's with them being in the loop. Cyber is always on the outside looking in, trying like hell to piece it all together. Sorry for the rant. I just wanted to add on with my personal opinions on the cyber security framework being severely broken because I deal with it on a daily basis.
- lokar 2y ago
  Our compliance and security people turned up with an urgent request to patch out Linux kernels in AWS. The pcmcia driver had a vuln I don’t listen to them much anymore
- Hizonner 2y ago
  > It's literally impossible to understand multiple systems at a deep level,\ No, it's not. It takes above average intelligence, and major investment in actual education (not just "training"), and actual depth of experience, but it's not impossible.
- unethical_ban 2y ago
  I know of an important company currently pushing to implement a redundant network data loss prevention solution, while they don't have persistent VPN enabled and multiple known misconfigurations of things that prevent web decryption working properly. Because someone needs a checkbox.
- zmmmmm 2y ago
  Do you think it comes from a fundamental misconception of how these roles should be structured? My take is that you just can't fundamentally assess technical elements from the outside unless they have been designed that way in the first place (for assessability). For example I educate my team that they have structure their git commits in a way that demonstrates their safety for audit / compliance purposes (never ever combine a high risk change with a low risk one, for example). That should go all the way up the chain. Failure to produce an auditable output is failure to produce an output that can be deployed.
smsm42 2y ago
Well, if you fail at compliance, you can be fired and sometimes even sued. If your compliance efforts cause system wide outage - nobody's to blame, shit happens. I predict this screwup will end up with zero consequences for anyone who took the decisions that led to it too. So how else do you expect this system to evolve, given this incentive structure?
bargle0 2y ago
CrowdStrike, ZScaler, and the rest of these people surely have lobbyists that ensure their software is compelled by regulators.
23orj2o3 2y ago
When a metric becomes a target ...
xphilter 2y ago
It’s even worse. The consultants who run the audits (usually business school recent grads) work with other consultants who shill the third party software and implementation work.
wyager 2y ago
Zscaler is such insane garbage. Legitimately one of the worst pieces of software I have ever used. If your organization is structurally competent, it will never use Zscaler and will just use wireguard or something.
ShuffleBoard 2y ago
Picking up pennies in front of a steamroller...
Gormo 2y ago
> So we need to hold regulatory bodies accountable as well - when they frame regulation such that organisations are cornered into this they get to be part of the culpability here too. Replacing common-law liability with prescriptive regulation is one of the main drivers of this problem today. Instead of holding people accountable for the actual consequences of their decisions, we increasingly attempt to preempt their decisions, which is the very thing that incentivizes cargo-cult "checkbox compliance". It motivates people who otherwise have skin in the game and immediate situational awareness to outsource their responsibility to systems of generalized rules, which by definition are incapable of dealing effectively with outliers.
casey2 2y ago
It's VERY easy to blame CrowdStrike and companies like them as they are the one LOBBYING for those checkboxes. Both zscaler and Crowdstrike spent 500K last year lobbying.

by zmmmmm 2y ago

wparad 2y ago

That for sure is a great point, it's actually something Dorota brought up in her talk this year https://vimeo.com/940487124 https://vimeo.com/940487124

SuperSandro2000 2y ago

Sounds familiar

RedShift1 2y ago

Don't put your eggs in one basket, I use multiple anti-virus products so that if one blows up at least not all computers are affected. Looks like my old wisdom is still new wisdom. Clarification: I mean that every computer has one anti-virus product, but not every computer has the same anti-virus product. I'm not installing multiple anti-virus products on the same computer.

cqqxo4zV46cp 2y ago
This is much easier applied personally than it is to a 30k person organisation. No need to be condescending.
- Twisell 2y ago
  In both case it's costly. But cost of maintenance aside it wouldn't be that bad to deploy each half the fleet with two distincts EDR. This is actually implicitly in place for big companies that support BYOD. If half your fleet is on Windows another 40% on MacOs and 10% on Linux you need distinct EDR solutions and a single issue can't affect all your fleet at once.
NDizzle 2y ago
You use multiple anti-virus products. Let's assume you use 3. Do you have multiple clusters of machines, each running their own AV product, so in case one has this problem the other two are unaffected? How much overhead are we talking about here? Because if you're just using multiple AV software installed on one machine, 1) holy shit, the performance penalty, 2) you'd still be impacted by this, as CS would have taken it down.
- mschuster91 2y ago
  Not the person you're replying to, but in any reasonable organization with automated software deployment it should be easy to pool machines into groups, so you can make sure that each department has at least one machine that uses a different anti-virus software. Bonus, in case you do catch a malware, chances are higher that one of the three products you use will flag it.
  - NDizzle 2y ago
    Again, "should be" academic stuff. So you have multiple AV products and you target those groups. You have those groups isolated on their own networks, right? With all the overhead that comes with strict firewall rules and transmission policies between various services on each one. With redundant services on each network... you've doubled or tripled your network device costs solely to isolate for anti virus software. So if only one thing finds the zero day network based virus, it won't propagate to the other networks that haven't been patched against this zero day thing. How far down the rabbit hole do we want to go? If you assume many companies are doing this kind of thing, or even a double digit percentage of companies, I have bad news for you.
- ta1243 2y ago
  They surely mean that all odd number assets are running crowdstrike and even are running sential-one (or similar, %3, %4, etc etc). At least then you only lose half your estate.
  - RedShift1 2y ago
fire_lake 2y ago
If you have all of them on a critical path then your risk of blow up increases!
deleted 2y ago
[deleted]
xorcist 2y ago
Not sure that's a great idea. This stuff tends to have very high privilege access. It's enough that one of your anti virus vendors get hacked for your whole organization to get owned...
- interludead 2y ago
  I see one main challenge - it can increase the incidence of false positives.
Lovesong 2y ago
Was there any situation where having 3 anti-virus was more beneficial than having only 2?
- ben_w 2y ago
  I'm reading this as first third of computers have AV brand A, second third have brand B, remainder have brand C. Thus, if brand A does something actively harmful all by itself, only 1/3rd of machines are impacted. This is an improvement on having only 2 brands, as having 1/3rd of your machines go down is better than having 1/2 of your machines go down.
interludead 2y ago
Indeed a wise strategy

baxtr 2y ago

hey, but at least (a) we have a process (b) we documented it and (c) we review it regularly! what can go wrong?!

rubatuga 2y ago
There should be a new term called compliance hell.
- c0l0 2y ago
  The term already exists, but the "hell" is mostly silent (even in writing).
  - baxtr 2y ago
    Is there a process for that??

theallan 2y ago

The flip side is, if you don't do auto updates and an exploit is published and used against you and you haven't yet tested / pushed the patch, that you would have been protected against if it had auto updated, you are up the creak without a paddle in that situation as well. To some degree you have to trust the software you are using not to mess things up.

cqqxo4zV46cp 2y ago
Yes. And the vast majority of the time, it doesn’t mess things up. The notion that you may take on risk to net alleviate risk is somehow lost on a lot of people in these conversations.
zmmmmm 2y ago
So since I do mission critical healthcare I do run into this concept. But it's not as unresolvable as you portray. Consider for example HIPAA "break the glass" requirement. It says that whatever else you implement in terms of security you must implement a bypass that can be activated by routinely non-authorised staff to access health information if someone's life is in danger. Similarly, when I questioned, "why can't users turn off ZScaler in an emergency" we were told that it wouldn't be compliant. But it's completely implementable at a technical level (Zscaler even supports this). You give users a code to use in an emergency and they can activate it and it will be logged and reviewed after use. But the org is too scared of compliance failure to let users do it.
- jenscow 2y ago
  While I agree with the requirement, but it sounds like a vault would need to have an unlocked door with a sign.
  - ivan_gammel 2y ago
    A vault with firearms in the police station to which every staff member has a key. Sounds reasonable to me. Users are not prisoners left in the burning building without a fire escape.

tempaway4575144 2y ago

You are completely right. BTW It wasn't a software update, it was a content update, a 'channel file'. Someone didn't do enough testing. edit: or any testing at all? https://x.com/George_Kurtz/status/1814235001745027317 https://x.com/George_Kurtz/status/1814235001745027317 https://x.com/brody_n77/status/1814185935476863321 https://x.com/brody_n77/status/1814185935476863321

glimshe 2y ago
It passed all unit tests!
- livrem 2y ago
  It passed the type checker!
Shank 2y ago
It's an automatic update of the product. Semantic "channel vs. binary" doesn't indicate anything. If your software's definition files can cause a kernel mode driver to crash in a bootloop you have bigger problems, but the outcome is the same as if the driver itself was updated.
- tempaway4575144 2y ago
  Indeed. Its worse really, it means there was a bug lurking in their product that was waiting for a badly formatted file to surface it. Given how widespread the problem is it also means they are pushing these files out without basic testing. edit: It will be very interesting to see how CrowdStrike wriggle out of the obvious conclusion that their company no longer deserves to exist after a f*k up like this.
  - account42 2y ago
    Simple: They are obviously too big to fail now.

bonoboTP 2y ago

> more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting. Duh, else there would be no need to audit them to force compliance, they'd just do it by themselves. The only reason it needs forcing is that they otherwise aren't motivated enough.

lucianbr 2y ago
Good point. But the audit seems useless now. It's supposed to prevent the carelessness from causing... this thing that happened anyway. Sure, maybe it prevented even more events like this from happening. But still.
- felurx 2y ago
  To be fair, I'd claim that it's pretty rare for anything anyone ever does to not be a trade-off.
- sobkas 2y ago
  > Good point. But the audit seems useless now. It's supposed to prevent the carelessness from causing... this thing that happened anyway. > Sure, maybe it prevented even more events like this from happening. But still. Because the point of audit is not to prevent hacks, it's to prove that you did your due diligence to not get hacked, so fact that hack happened is not your fault. You can hide under umbrella of "sometimes hacks happen no matter what you do".
  - lucianbr 2y ago
    CYA is the reason you do the audit. But the reason for the audit's existence and requirement is definitely so that hacks don't happen. Don't tell me regulatory agencies require things so that companies can hide behind them.
    - bonoboTP 2y ago
      The reason for the audit's existence is CYA one level above. The chain ends with a politician's CYA in front of the electorate.

stefan_ 2y ago

When we think "security" on HN we think about the people who escalate wiggling voltages at just the right time into a hypervisor shell on XBox, but I've had to recognize that my learned bias is not correct in the real world. In the real world, "computer security" is a profession full of hucksters that can't tell post-quantum from heap and whose daily work of telling people repeatedly to not click links in Outlook and filling out checklists made by people exactly like them has essentially no bearing on actual security of any sort.

Shank 2y ago
It's driven by a lot of things. Part of it is driven by rising cyber liability insurance rates, for one. A lot of organizations would rather not pay for CrowdStrike, but the premiums for not having an "EDR/XDR/NGAV" solution can be astoundingly high at-scale. Fundamentally there's a lot of factors in this ecosystem. It's really wild how incentives that seem unrelated end up with crazy "security" products or practices deployed.
- teeray 2y ago
  > A lot of organizations would rather not pay for CrowdStrike, but the premiums for not having an "EDR/XDR/NGAV" solution can be astoundingly high at-scale. Just like a lot of homeowners would rather not pay for ADT, but insurance requires a box-ticking “professionally-monitored fire alarm system.” Nevermind that I can dial 911 as well as the “professional” when I get the same notification as they do.
l33t7332273 2y ago
I’ve met highly paid “security engineers” that talked about not really being into programming or being okay with python but everything else is too complicated. It shocks me that such a low level of technical competence is required.
tristor 2y ago
> In the real world, "computer security" is a profession full of hucksters Always has been. The information security model is about analogizing digital systems as physical systems, and employing the analogues of those physical controls that date back hundreds of years on those digital systems. At no point, in my relatively long career, have I ever met anyone in Information Security who actually understands at depth anything about how to secure digital systems. I say this as someone who has spent a lot of my career trying to do information security correctly, but from the perspective of operations and software engineering, which is where it must start. The entire information security model the world works with is tacking on security after the fact, thinking you need to builds walls and a vault door to protect the room after the house has already been built, when in fact you need to build the house to be secure from the start because attacks don't go through doors, attacks are airborne (I recognize the irony of my analogizing digital concepts to physical concepts surrounding security, but I do it because of any infosec people that may read my comment so they can understand my point). Because of this model, we have gone from buying "boxes" to buying "services", but it has never matured away from the box-checking exercise it's been since day one. In fact, many information security people have /no training or education/ in security, it's entirely in regulatory compliance.

2-3-7-43-1807 2y ago

I don't even want to know how many mission critical systems automatically deploy open source software downloaded from github or (effectively random) public repositories.

rini17 2y ago
Unlike Windows, there is at least the option to use curated software distributions such as Debian or RH that won't apply random stuff from upstream repositories.
- 2-3-7-43-1807 2y ago
  I'm talking about all sorts of software projects implemented using especially Python, Ruby, node.js etc.
- lmz 2y ago
  I like Debian but it's not like they need random upstream repositories when they can make random patches themselves e.g. the OpenSSL Purify issue.
interludead 2y ago
These risks must be carefully managed that's it I think

higeorge13 2y ago

So true! It seems like all of these were invented to create another market for b2b saas security, audit, monitoring, etc. companies. Nobody cares about actual security or infrastructure anymore. Everything is just buying some subscription for random saas companies, not checking their permissions and grant policies and ticking boxes because... compliance.

kmlx 2y ago

> the risk of failing the compliance checkbox it satisfies is paramount. i'm curios as to what compliance is there to be satisfied to necessitate such a hardcore measure?

rcxdude 2y ago
There's a reasonable number of circumstances where there are cybersecurity standards that get imposed on organisations: insurance, from a customer, or from the government (especially if they are a customer). These standards are usually fairly reasonably written, but they are also necessarily vague and say stuff like "have a risk assessment", and "take industry-standard precautions". This vagueness can create a kind of escalation ratchet: when people tasked with (or responsible for) compliance are risk-averse and/or lazy, they will essentially just try to find as many precautions as they can find and throw them all in blanket-style, because it's the easiest and safest way to say that you're in compliance. This is especiallly true when you can more or less just buy one or two products which promise to basically tick every possible box. And if something else pops up as a suggestion, they'll throw that in too. Which then becomes the new 'industry standard', and it becomes harder to justify not doing it, and so on.
victorbjorklund 2y ago
I worked in orgs where customers put a certain security standard in the contract. So if you fail that you are kind of in breach of contract

izacus 2y ago

Of course, disabling those auto updates will have you fail the external security audit and now your security team needs to fight with the rest of the leadership in the company explaining why you're generating needless delays, costs against the "state of the art in security industry" and why your security guys are smarter than the people who have the power to approve or deny your security certification.

chii 2y ago
> you fail the external security audit aka, you fail the cover-your-ass security, rather than actual security.
- izacus 2y ago
  yep, but "trust us, we're secure, pinky promise by our internal employees" doesn't really work either.
  - ryandrake 2y ago
    Don't forget the press releases all saying "We take security very seriously!"
- StefanBatory 2y ago
  Employees would rather care for their employment and keeping boss happy rather than going against their orders.
lysp 2y ago
Or lose IT/security insurance for not installing or disabling it.
- nlitened 2y ago
  Well, let’s see how much insurance companies will pay now

zh3 2y ago

"The metric becomes the measure", i.e. Goodhart's Law. https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law

belter 2y ago

> So we need to hold regulatory bodies accountable as well - when they frame regulation such that organisations are cornered into this they get to be part of the culpability here too. No, we need to hold Architects accountable, and this is the core of the issue. Creating systems with single, outsourced responsibility, in the critical path.

ta1243 2y ago
As a CTO, when your company goes down you get fired. When every company goes down you get let off. The sensible thing is follow the herd and centralise. You're outsourcing the risk to your own job.
- zer00eyz 2y ago
  This is the point of much of the security efforts we see now. Outsourcing of security functions, and things like login push a lot of liability and legal issues off into someone else's house. It's hard to be the source of a password leak, or be compromised when you don't control the passwords. But like any chain your only as secure as your weakest link... Snowflake is a great current example of this. Mean while the USPS just told us "oops" we had tracking pixels for a bunch of vendors all over our delivery preview tool. Candidly, most people stacks look a lot less like software and more like a toolbar riddled IE5 install circa 2000. I don't think our industry is in a good place.
- belter 2y ago
  > When every company goes down you get let off. And when only companies who use a certain OS and/or Cloud vendor go down? ;-) Do you also get let off?
  - ta1243 2y ago
    If it's large enough. Nobody loses their job when office 365 goes down, even if that happens once a year. However if you decided to choose a small company which goes down once every 5 years, you're screwed.
- 2y ago

lelanthran 2y ago

> The risk of ZScaler being a central point of failure is not considered. But - the risk of failing the compliance checkbox it satisfies is paramount. You're conflating Risk and Impact, and you're not considering the target of that Risk and that Impact. Failing an audit: 1. Risk: high (audits happen all the time) 2. Impact to business: minimal (audits are failed all the time and then rectified) 3. Impact to manager: high (manager gets dinged for a failing audit). Compare with failing an actual threat/intrusion: 1. Risk: low (so few companies get hacked) 2. Impact to business: extremely high 3. Impact to manager: minimal, if audits were all passed. Now, with that perspective, how do you expect a rational person to behave? [EDIT: as some replies pointed out, I stupidly wrote "Risk" instead of "Odds" (or "Chance"). Risk is, of course, the expected value, which is probability X impact. My post would make a lot more sense if you mentally replace "Risk" with "probability".]

Freak_NL 2y ago
Almost correct, but you mean 'chance' where you write 'risk': Risk = Chance × Impact The chance of failing an audit initially are high (or medium, present at least). The impact is usually low-ish. It means a bunch of people need to fix policy and set out improvement plans in a rush. It won't cost you your certification if the rectification is handled properly. It's actually possible that both of your examples are awarded the same level of risk, but in practice the latter example will have its chance minimized to make the risk look acceptable.
- codetrotter 2y ago
  Chance has more positive connotations than it has negative connotations IMO. Probability is a more neutral word, and fits better.
noahmasur 2y ago
I think the parent was correct in the use of the word "Risk"; it's different than your definition, which appears to be closer to "likelihood". Risk is a combination of likelihood and impact. If "risk" were just equivalent to "likelihood" then leaving without an umbrella on a cloudy day would be a "high-risk situation". A rational person needs to weigh both the likelihood and impact of a threat in order to properly evaluate its risk. In many cases, the impact is high enough that even a low likelihood needs to be addressed.

cm2012 2y ago

[flagged]

p0seidon 2y ago

This is 100% the reality

monkeydust 2y ago

"All over the place I'm seeing checkbox compliance being prioritised above actual real risks from how the compliance is implemented." Great statement and one that needs to be seriously considered - would DORA regulation in the EU address this I wonder? Its a monster piece of tech legislation that SHOULD target this but WILL it - someone should use todays disaster and apply it to the regs to see if its fit for purpose.

piker 2y ago
Broken regulations? Fix by adding more!
- idle_zealot 2y ago
  Well, yeah. If a regulation is broken and not achieving its goal it should be changed. What's the alternative? "Regulation? We tried that once and it didn't work perfectly, so now we let The Market™ sort out safety standards."
  - InsideOutSanta 2y ago
    Who needs regulation when you can have free Fentanyl with your CrowdStrike subscription! All of your systems will go down, but you won't care, and the chance of accidental overdose is probably less than 10%!
    - jddj 2y ago
      The child slave labour is what really gets the deal across the line
  - piker 2y ago
    Yes, in many contexts that may well be the correct conclusion. Your comment presumes that regulation here has proven itself useful and not resulted in a single point of failure which potentially reduces overall safety. It’s of course the correct comment from a regulator’s perspective.

dijit 2y ago

It's possible that CrowdStrike heavily incentivises being left to update itself. Removing the features that would allow sysadmins to actually do it automatically, even via the installer itself- would definitely be one way, but another one could be aggressive focus-stealing nags (similar to Windows' own nags) which in a server environment can actually cause some major issues, especially when automating processes in Windows (as you need to close the program when updating). I think it's easy to blame the sysadmins, but I would also be remiss if I didn't point out that in the Windows world we have been slowly accepting these automatic dark patterns and alternative (more controlled) mechanisms have been removed over time. I almost don't recognise the deployment environment today as to what it was in 2004; and yes, 20 years is a long time, but the total loss of control over what a computer is doing is only going to make issues like this significantly more common.

mr_mitm 2y ago
They say it was caused by a faulty channel file. I don't know what a channel file is, and they claim to not rely on virus signatures, but typically anti virus product need the latest signatures all the time and poll them probably once an hour or so. So I'm not surprised that an anti virus product wants to stay hyper updated and updates are rolled out immediately to everyone globally.
- HereBeBeasties 2y ago
  No, I'm not surprised either. But if you're operating at this kind of scale and with this level of immediate roll-out, what I would expect are: * A staggered process for the roll-out, so that machines that are updated check-in with some metrics that say "this new version is OK" (aka "canary deployment") and that the update is paused/rolled back if not. * Basic smoke testing of the files before they're pushed to any customers * Validation that the file is OK before accepting an update (via a checksum or whatever, matched against the "this update works" automated test checksums) * Fuzz tests that broken files don't brick the machine Literally any of the above would have saved millions and millions of dollars today.
bandrami 2y ago

lucasRW 2y ago

Electricity too is a SPOF.

RobotToaster 2y ago
Most mission critical medical systems have a backup generator.
- XorNot 2y ago
  And IT has dual power supply servers, and endpoint UPSs. Electricity is aggressively made redundant for mission-critical systems.

jrockway 2y ago

Zscaler is truly amazing. It can't do HTTP/2. Our product is HTTP/2-only. So we can't use our own product at work.

Avamander 2y ago
Their visibility and process in general for handling abuse of their services is also abysmal.
mrweasel 2y ago
I know a few people who have Zscaler deployed at work. It will routinely kick them of the internet, like multiple times a day. It has gotten to the point where they can sort of tell in advance that it's about to happen. The theory so far it that it's related to their activities, working in DevOps they will sometimes generate "suspicious" traffic patterns which will then trigger someone policy in Zscaler, but they're not actually sure.
supertrope 2y ago
ZScaler itself uses port 443 UDP, but blocks QUIC. The last time I checked it didn't support IPv6 so they told customers to disable IPv6. Security software is legacy software out of the box and cuts the performance of computers in half.
pwarner 2y ago
What Zscaler can and will do though is break your network randomly and in strange ways. They don't even seem to charge for that feature!

grishka 2y ago

If I were running an organization that needs these audits, I'd always have fallback procedures in place that would keep everything running even if all computers suddenly stop working, like they did today. General-purpose software is too fragile to be fully relied upon, IMO. If a general-purpose computer must be used for something mission-critical, it should not have an internet connection and it should definitely not allow an outside organization to remotely push arbitrary kernel-mode code to it. It should probably also boot from a read-only OS image so that it could always be restored to a known-good state by just rebooting.

sys_64738 2y ago
Organizations don't want to increase risk by listening to an employee with their personal opinion. Orgs want an outside vendor who they can point at and say "it's their fault", and await a solution. Employees going rogue and not following the vendor defined SW updates is a much higher risk than this particular crisis.

sobkas 2y ago

It depends on what your position is. Are you there to actually provide security to your org or to tick a in an audit. If both which is more important. Because failing an audit have real consequences, while having breaches in security have almost none. Just look at credit score companies.

the8472 2y ago

Regulation or auditors rarely require specific solutions. It's the companies themselves that choose to achieve the goals by applying security like tinctures: "security solutions". The issue is that the tinctures are an approved remedy.

lqet 2y ago

Can anyone explain how CrowdStrike could possibly fix this now? If affected machines are stuck in an endless BSOD cycle, is it even possible to remotely roll out a fix? My understanding is that the machines will never come to the point where a CS update would be automatically installed. Is the only feasible option the official workaround of manually deleting system files after booting into the recovery environment? How could this possibly be done on scale in organizations with tens of thousands of machines?

ExoticPearTree 2y ago
CrowdStrike itself will not fix anything. They published a guide on how to workaround the problem and that's it. Most likely a lot of sales reps and VPs will be fielding calls all over the weekend explaining large customers how did they manage to screw up and how much discount will they offer on the next renewal cycle. Legally, I think somewhere in their license it says is that they're not responsible in any way or form if their software malfunctions in any way.
- delfinom 2y ago
  Small companies get the shitty generic license. Big companies negotiate liability terms.
  - betaby 2y ago
    > Big companies negotiate liability terms. Never heard that in the context of the software licenses.
    - rayrey 2y ago
      Pretty standard in enterprise b2b, most of the sales cycle is in contracts
      - betaby 2y ago
        Well, probably no? I've never seen liabilities in dollar value, or rather any significant value. Also I saw our company Ceowdstrike contract for 10k+ seats, no liabilities there.

jojobas 2y ago

The question is, how did they manage to not crash everything for so long without a staged/rolling update deployment strategy?

chrisjj 2y ago
Perhaps it took this long for the offending file to pass 64K Excel rows... :)

fireflash38 2y ago

That's some very twisted logic. If I expect someone to clean the kitchen as part of restaurant closeup checklist, and they fuck it all up, would I blame the checklist, or the person doing the work? You blame the person fucking it up. In this case, it's someone who only cares about checking a box. Or someone who pushes broken shit.

lqet 2y ago
If this person simultaneously fucks up millions of kitchens around the world, you do not blame that person. You blame the checklist which encouraged giving a single person global interlocked control over millions of kitchens, without any compartmentalization.
- fireflash38 2y ago
  > If this person simultaneously fucks up millions of kitchens around the world, you do not blame that person. No, you definitely do, even more than before. Let's say for example that the requirement is to disinfect anything that touches food. And the bleach supplier fucks it all up. You blame the bleach supplier. You don't throw out the disinfectant requirement.

ExoticPearTree 2y ago

> All over the place I'm seeing checkbox compliance being prioritized above actual real risks from how the compliance is implemented. Because if everyone is doing their job and checks their box, they're not gonna get fired. Might be out of a job because the company goes under, but hey, it was no one's fault, they just did their job.

dagaci 2y ago

Most enterprises will have teams of risk and security people. They will be asking who authorized deployment of an untested update into production. If CrowdStrike deployments cannot be managed, then they will switch to a product which can be managed.

basisword 2y ago

>> It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. While orgs using auto update should reconsider, the fact that CrowdStrike don't test these updates on a small amount of live traffic (e.g. 1%) is a huge failure on their part. If they released to 1% of customers and waited even 24 hours before rolling out further this seems like it would have been caught and had minimal impact. You have to be pretty arrogant to just roll out updates to millions of customers devices in one fell swoop.

lqet 2y ago
Why even test the updates on a small amount of live customers first? Wouldn't this issue already have surfaced if they tested the update on a handful of their own machines?
- basisword 2y ago
  I would hope they've done that and it passed internal QA. But maybe not a good idea to assume they're doing any sort of testing at all.
  - greggsy 2y ago
    They’d have rigorous test harnesses, but you can’t really account for the complexities of a highly configurable platform like Windows.
    - kristjansson 2y ago
      The prevalence of the issue makes it seem unlikely to have been caused by site-specific configurations

shadowgovt 2y ago

Consolidation / optimization of labor. When Crowdstrike messes up and BSODs thousands of machines, they have a dedicated team of engineers working the problem and can deliver a solution. When your company gets owned because you didn't check a compliance checkbox, it's on you to fix it (and you may not even currently have the talent to do so). We see similar risk tradeoffs in cloud computing in general; yes, hosting your stuff on AWS leaves you vulnerable to AWS outages, but it's not like outages don't happen if you run your own iron. You're just going to have to dispatch someone a three hour drive away to the datacenter to fix it when they do.

deleted 2y ago
[deleted]
chrisjj 2y ago
> they have a dedicated team of engineers working the problem and can deliver a solution. No. Can merely facilitate the customer's on-site admin to deliver a solution.

afh1 2y ago

Isn't there a way to schedule the updates? With Windows updates, when I used to work at a firm with a critical system running on Windows, we had main and DR servers and the updates were scheduled to first rollout on the main server and a day after I think at the DR, which has saved us at least once in the past from a bad Windows update...

vladvasiliu 2y ago
More or less. You can set up some update policies which and apply those to subsets of your machines. You can disable updates during time blocks, or block them altogether. There's also the option of automatically installing the "n-1" update. We run auto n-1 at work, but this also happened at the same time on my test machine with runs "auto n". It never happened before, so this looks like something different than the actual installed sensor version, especially since the latest version was released something like a week ago.

jimnotgym 2y ago

> So we need to hold regulatory bodies accountable as well... My bank, my insurer, my payment card processor, my accounting auditor and probably others may all insist I have anti-virus and insist that it is up to date. That is why we have to have these systems. However, I used to prefer systems that allowed me to control the update cycle and push it to smaller groups.

bhbh 2y ago

CrowdStrike has various auto update policies, including not to automatically update to the latest version, but to the latest version -1 or even -2. Customers with those two policies are also impacted.

lenkite 2y ago

It's a big stretch to call this the regulator's fault when its basic lack of testing by Microsoft and/or Crowdstrike. If a car manufacturer made safety belts that broke, you don't blame the regulators. The root cause is automatic, mindless software update without proper testing - nothing to do with regulators.

beeboobaa3 2y ago

It's what you get when you let luddites, also known as managers, make the rules.

number65259 2y ago

> My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer starting up Hi fellow CVS employee. Are you enjoying your zscaler induced SSO outages every week that torpedo access to email and every internal application? Well now your VMs can bluescreen too. A few more vendor parasites and we'll be completely nonfunctional. Sit tight!

Puts 2y ago

This is an oversimplification. IF we are talking about compliance to ISO 27001 you are supposed to do your own risk assessment and implement necessary controls. The auditor will basically just check that you done the risk assessment, and that you have done the controls you said yourself you need to do. I'd say this has nothing with regulatory compliance to do at all. The real truth is that modern organizations are way too attached to cloud solutions. And this runs across all parts of the organization with Saas and PaaS whether it's email (imagine Google Workspace having a major issue), AWS, Azure, Okta… I've had the discussions so many times and the answer is always – the risks doesn't matter because the future is cloud and even talking about self hosting anything is naive and honestly we need to evaluate your competence for even suggesting it. (Also the cloud would maybe not be this fragile if it wasn't for lock-in with different vendors. If you read the TOS it says basically on all cloud services that you are responsible for the backup – but getting your data out of the service is still pain in the ass – if possible at all)

crazygringo 2y ago
> The real truth is that modern organizations are way too attached to cloud solutions. I'm confused. This is a security product for your local machine. Not the cloud. Unless you call software auto-update "the cloud", but that's not what people usually mean. The cloud isn't about downloading files, it's about running programs and storage remotely. I mean, if CloudStrike were running entirely on the cloud, it seems like the problem would be vastly easier to catch immediately and fix. Cloud engineers can roll back software versions a lot easier than millions of end users can figure out how to safe boot and follow a bunch of instructions.
- Puts 2y ago
  Well, in all times usually there has been the option to run a local proxy/cache for your updates so that you can properly test them inside your own organization before rolling them out to all your clients (precisely to avoid this kind of shit show). But doing that requires an internal team running it and actually testing all updates. But modern organizations don't want an IT-department, they want to be "cloud first". So they rely on services that promise they can solve everything for them (until they don't). Cloud is not just about where things are – it's also about the idea that you can outsource every single piece of responsibility to a intangible vendor somewhere on the other side of the globe – or "in the cloud".

coliveira 2y ago

This is all well deserved for executives who trust MS to run their businesses. If you have the resources, like a bank, it is crime to put your company in the hands of MS.

alecco 2y ago
IME, Boomer managers refuse to use anything but Windows. We have a few more years of this.

metalliqaz 2y ago

this is the world that lawyers gave us

pookha 2y ago

It's easy to blame CrodStrike because they're the ones to blame here. They lit a billion system32 folders on fire with an untested product and push out fear mongering corny marketing material. Turns out you should be afraid.

MetaWhirledPeas 2y ago

> Orgs are doing this because they are more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting. If a failed audit is the big scary monster in their closet, then it sounds like the senior leadership is not intimately familiar with the technology and software in general, and is unable to properly weigh the risks of their decisions. More and more companies are becoming software companies whether they like it or not. The software is essential to the product. And just like you would never want a non-lawyer running your law firm, you don't want a non-software person running your software company.

temporarely 2y ago

> So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. TIL that US government has pressured foreign nations to install a mystery blob in the kernel of machines that run critical software "for compliance". If this wasn't a providential goof on the part of Crowdstrike -- the entire planet is now aware of this little known fact -- then some helpful soul in Crowdstrike has given us a heads-up.

narrator 2y ago

No doubt there will be another piece of software mandated to check up on the compliance software. When that causes a global IT outage, software that checks up on the software that checks up on the compliance software will be mandated.

radiator 2y ago

Very sharp and to the point, this comment. I would like to add that in large companies the audit will, in my experience, very often examine documents only -- not actual configuration or code.

phone8675309 2y ago

> Orgs are doing this because they are more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting. I've been someone in one of those audit meetings defending decisions made and defending things based on the records we keep and I understand this because it is both a deeply unpleasant and expensive affair to pull people from current projects and place them before auditors for several hours to debate what compliance actually means.

maxrecursion 2y ago

I work for government organization that is constantly audited and I've seen this play out over and over. An important aspect I never see mentioned is most Cyber Security personnel don't have the technical experience to truly understand the systems they are assessing, they are, like you said, just pushing to check those compliance boxes. I say this as someone who is currently in a Cyber Security role, unfortunately, as I'm coming to learn cyber roles suck. But this isn't a jab at those Cyber Security personnel's intelligence. It's literally impossible to understand multiple systems at a deep level, it takes employees working on those systems weeks to months to understand this stuff, and that's with them being in the loop. Cyber is always on the outside looking in, trying like hell to piece it all together. Sorry for the rant. I just wanted to add on with my personal opinions on the cyber security framework being severely broken because I deal with it on a daily basis.

lokar 2y ago
Our compliance and security people turned up with an urgent request to patch out Linux kernels in AWS. The pcmcia driver had a vuln I don’t listen to them much anymore
Hizonner 2y ago
> It's literally impossible to understand multiple systems at a deep level,\ No, it's not. It takes above average intelligence, and major investment in actual education (not just "training"), and actual depth of experience, but it's not impossible.
unethical_ban 2y ago
I know of an important company currently pushing to implement a redundant network data loss prevention solution, while they don't have persistent VPN enabled and multiple known misconfigurations of things that prevent web decryption working properly. Because someone needs a checkbox.
zmmmmm 2y ago
Do you think it comes from a fundamental misconception of how these roles should be structured? My take is that you just can't fundamentally assess technical elements from the outside unless they have been designed that way in the first place (for assessability). For example I educate my team that they have structure their git commits in a way that demonstrates their safety for audit / compliance purposes (never ever combine a high risk change with a low risk one, for example). That should go all the way up the chain. Failure to produce an auditable output is failure to produce an output that can be deployed.

smsm42 2y ago

Well, if you fail at compliance, you can be fired and sometimes even sued. If your compliance efforts cause system wide outage - nobody's to blame, shit happens. I predict this screwup will end up with zero consequences for anyone who took the decisions that led to it too. So how else do you expect this system to evolve, given this incentive structure?

bargle0 2y ago

CrowdStrike, ZScaler, and the rest of these people surely have lobbyists that ensure their software is compelled by regulators.

23orj2o3 2y ago

When a metric becomes a target ...

xphilter 2y ago

It’s even worse. The consultants who run the audits (usually business school recent grads) work with other consultants who shill the third party software and implementation work.

wyager 2y ago

Zscaler is such insane garbage. Legitimately one of the worst pieces of software I have ever used. If your organization is structurally competent, it will never use Zscaler and will just use wireguard or something.

ShuffleBoard 2y ago

Picking up pennies in front of a steamroller...

Gormo 2y ago

> So we need to hold regulatory bodies accountable as well - when they frame regulation such that organisations are cornered into this they get to be part of the culpability here too. Replacing common-law liability with prescriptive regulation is one of the main drivers of this problem today. Instead of holding people accountable for the actual consequences of their decisions, we increasingly attempt to preempt their decisions, which is the very thing that incentivizes cargo-cult "checkbox compliance". It motivates people who otherwise have skin in the game and immediate situational awareness to outsource their responsibility to systems of generalized rules, which by definition are incapable of dealing effectively with outliers.

casey2 2y ago

It's VERY easy to blame CrowdStrike and companies like them as they are the one LOBBYING for those checkboxes. Both zscaler and Crowdstrike spent 500K last year lobbying.