9 ms·
CrowdStrike fixes start at "reboot up to 15 times", gets more complex from there
- breakingcups 2y agoSounds great for data consistency.
- peterleiser 2y agoThey should change their name to "IT CrowdStrike"
- willcipriano 2y ago"Alright I bought us some time"
- deleted 2y ago[deleted]
- rekttrader 2y agoThat’s the hilarious and sad state of affairs.
- seydor 2y agoI would like to have the power to press the button that deploys this update
- Ancapistani 2y agoNo, you wouldn't.
- toomuchtodo 2y agoIt's all cold sweats, hot flashes, and the urge to vomit when things go wrong.
- thelogicguy 2y agoI get nervous just pushing updates to my stupid little website. I simply cannot imagine.
- pas 2y agoClearly that's how they ended up with the current team. They hired for culture fit. Anyone who worries too much is out. You bet they have an amazing perfect top-notch hiring pipeline, many rounds of interviews, and whatever you could wish for! (No, no ... the subcontractors writing code are not in scope for this, duh.)
- safety1st 2y agoWith web, at small scale (which honestly is 95% of the world), you just version and back up everything. We push updates that break stuff from time to time. If it's bad enough, we just hit a button and roll back the change. The nerves are basically a sign that you need to have an easy rollback process in place, once you have it, you sleep easy and things are fun.
- grecy 2y agoIt is always worth remembering it is just a job. Do not, under any circumstances let a job impact your health or mental well being.
- StefanBatory 2y agoAs in this case, it clearly had a huge impact on people. You can't say it's just a job when people can die because of you.
- toomuchtodo 2y agoI want you to know that I appreciate this comment far more than you could ever know, and you are absolutely right. At the time, it was not just a job. It was a passion with a bar rising much faster than I could rise to the occasion. Simultaneously, my personal life was slowly falling apart, from family and loved ones in need, and the result was eventual failure leading to me being terminated. Luckily, it was one of the best events that has ever happened to me. I was able to land in a much better role almost immediately, which eventually catapulted my career and assisted in me being able to become financially independent as well as pivot into a domain with immensely improved work life balance. Importantly, I recognize I got lucky. It could’ve easily gone the other way, with me giving up both professionally and personally (yeeting myself from this plane of existence). So, I not only violently echo your comment to others who come across this thread, I will go further to say that sometimes when you’re going through hell, if you keep going, there is light at the other end. It is just a job, it is okay to ask for help, and failure is when you stop trying to get back up, not when you get knocked down.
- fredoralive 2y agoYes. Yes. To hold in my hand a button that contains such power, to know that blue screens on such a scale was my choice. To know that the tiny pressure on my thumb, enough to push the button, would end everything. Yes, I would do it! That power would set me up above the gods. And through Crowdstrike, I shall have that power!
- whycome 2y agoIts like the name was predetermined for this scenario. Too on the nose really.
- havblue 2y agoWith great power comes great responsibility.
- seydor 2y agoWith colossal power, who cares about responsibility
- scottLobster 2y agoHaha, hahahahaha. Yeah, until the update fails to install because the constant BSODing has corrupted something else and now you have to troubleshoot that and down the rabbit hole you go. Oh just re-image it? Sure, except management refuses to allow you to do that because there's no time and money to reconfigure a machine from scratch. So you waste weeks directly debugging a hopeless case until management finally sees their error and money magically appears to do the re-image you asked for weeks ago. I totally haven't experienced this before and am not bitter in the slightest.
- fragmede 2y agoanyone who's had that kind of power, myself included, knows the answer is fuck no.
- pizzalife 2y agoSo would the FSB, PRC Army, North Korea, etc… The fact that a faulty update can get pushed by human error means that adversaries can, too.
- stefan_ 2y agoThey might just have it, remember when the night of the Russian invasion of Ukraine satellite terminals in Europe started being bricked with faulty firmware updates?
- mschuster91 2y agoHence why Kaspersky got banned just recently [1]. You absolutely do not want some foreign company having above-root rights on (critical) infrastructure in your country. [1] https://edition.cnn.com/2024/07/15/tech/russian-firm-kaspersky-lab-ban/index.html https://edition.cnn.com/2024/07/15/tech/russian-firm-kaspers...
- commandlinefan 2y agoI can all but guarantee that very little of the actual coding was done by U.S. citizens.
- theideaofcoffee 2y ago> “Every now and then a trigger has to be pulled.” > “Or not pulled. It’s hard to know which in your pajamas, Q.” It’ll probably turn out that this update was pushed out against the strident, loud warnings of some small dev group within the company, and overruled by the all-knowing managerial class to keep up an OKR. They’ll have been warned six ways to Sunday but... I’d definitely be not be the one pushing the big red button.
- RedShift1 2y agoYou seem to know some inside info... Source?
- theideaofcoffee 2y agoNone whatsoever, I don’t have any affiliation. But this is usually how it happens, knowing what I’ve seen first-hand in my day-to-day and just keeping up on the insanity of the industry.
- AlienRobot 2y ago>The first and easiest is simply to try to reboot affected machines over and over, which gives affected machines multiple chances to try to grab CrowdStrike's non-broken update before the bad driver can cause the BSOD. I thought it was BSOD'ing on boot? I don't understand how this works. It auto-updates on boot? From the internet?
- JumpCrisscross 2y ago> It auto-updates on boot? From the internet? Apparently!
- avidphantasm 2y agoProbably has to do this early on in the boot process so as not to require a reboot after update because of Windows’s silly pessimistic file locking.
- mschuster91 2y ago> because of Windows’s silly pessimistic file locking To be honest I prefer that over the #nix way of doing things. In Windows, you have exactly one file any given path can refer to - in Linux or Mac, it may depend on which directory's inode is seen as the root node by your process (e.g. chroot or container), or whether mounts are at play, or a file/directory got deleted and replaced by something else. Particularly the last scenario keeps tripping me every once in a while.
- ukuina 2y agoYes, that is as bad as it sounds.
- Ekaros 2y agoI love how this is solution for security, while sounding like most insanely unsecure thing...
- InsideOutSanta 2y ago
- idiotlogical 2y ago>reboot up to 15 times I see my orgs SCCM admins have been consulted
- senectus1 2y agolol
- dist-epoch 2y agoWho knew: "Did you try rebooting it?" actually works :)
- brettermeier 2y agoIT departments sometimes ask this for a reason, after you checked if the cable is plugged.
- ilkkao 2y agoSome government should force them to release a technical postmortem. Feels that they don't do it otherwise.
- gen3 2y agoI don’t think a cybersecurity company can take down half the US and not release a postmortem
- spuz 2y agoOf course, but we specifically would like to see a _technical_ postmortem that examines what kind of incremental rollout procedures they have and how this update overcame those.
- educasean 2y agoThere should be congressional hearings on this. Not just post mortems.
- CoastalCoder 2y agoHonest question: would you expect Congress to respond in a way that's a true net-positive?
- coffeefirst 2y agoNo, but its a warning to the next guy/megacorp: Don't do that, or you'll be dragged before the greatest obnoxious and self-aggrandizing body in the world for lengthy dressing down that probably affects the stock price.
- aeyes 2y agoOr... you know... This kind of software should be open source or companies using it should at least be able to audit the code themselves. Supposedly they have all kinds of certifications but not even having basic QA demonstrates that this is all just a smokeshow: https://www.crowdstrike.com/why-crowdstrike/crowdstrike-compliance-certification/ https://www.crowdstrike.com/why-crowdstrike/crowdstrike-comp...
- Connector2542 2y agoHello, IT, have you tried turning it on and off again 15 times? Seriously though - this entire outage is the poster child for why you NEVER have software that updates without explicit permission from a sysadmin. If I were in congress, I would make it illegal, it's an obvious national security issue.
- travoc 2y agoIt was a data update that triggered a software bug. It was not a software update. I don't think it's reasonable to make data updates illegal.
- CoastalCoder 2y agoI'm a general purpose computer, the distinction between software and data can be pretty fuzzy.
- fire_lake 2y agoCode is data and data is code
- surfingdino 2y agoUnless you use PIC controllers with https://en.wikipedia.org/wiki/Harvard_architecture https://en.wikipedia.org/wiki/Harvard_architecture
- duped 2y agoThis distinction is meaningless at best and harmful at worst. If a code path isn't followed until a config file updates, that is practically the same thing as the code path being introduced by the update.
- pas 2y agoThat's not the big no-no here. Lack of any real DRP is. Sure, it's cheaper to just buy CS Falcon (and who knows what other amazing vendors supplied timebombs are ticking silently) than paying sysadmins and developers ... and letting them build something that does what it needs, not much else, so there's no need to put these fantastic "single agents" from these RCE-as-a-service vendors on all the fucking servers.
- bluedino 2y agoDevSecOps should have you know, tested these updates before they were approved for release company-wide. If I can't commit code to our app without a branch, pull requests, code review...why can the infrastructure team just send shit out willy-nilly? "Always allow new updates" must have been checked, or someone just goes through a dashboard and blindly clicks "Approve"
- pas 2y agoIt's automatic, no? The whole "promise" (oh sorry, the "added value proposition") of CS is that they "keep you safe" automatically! It was a content update. Meaning basically antivirus signatures ... and oops, some minor non-functional changes to the filtering kernel driver.
- nicce 2y agoIn that case, automatic updates likely need different permission levels. What exactly is allowed to be updated automatically?
- pas 2y ago... well, yes, yes of course. And if I try to be serious on a late Friday night (it's almost 20:00 here), the obvious solutions is to have something like eBPF in/for the Linux kernel (which has a verifier[0]). And security vendors should follow "secure by design" principles. Yes, I know a try-fucking-catch might be too advanced, and uh oh kernel code is hard because unwinding is costly. But guess what else is also not cheap. (Okay, I seriousness failed.) But still. This is fair and square in the "this should never happen" scenario. It's an automatically downloaded plugin or whatever. (CS can call it "content update", but von Neumann is already calling FedEx to send them a pallet of industrial grade bitchslap.) And if the plugin loader cannot gracefully fail plugin loading, then it should obviously come with the appropriate audiovisual cues[1] so sysadmins know what to expect. [0] https://docs.kernel.org/bpf/verifier.html https://docs.kernel.org/bpf/verifier.html [1] https://www.youtube.com/watch?v=Dv-2dzD9F10 https://www.youtube.com/watch?v=Dv-2dzD9F10
- 2y ago
- greenavocado 2y agoWho bought massive quantities of put options in anticipation of this event?
- ijidak 2y agoDo they not roll out their new agents in small increments? I'm trying to understand how there is such a serious issue at this scale.
- crazygringo 2y agoThe answer is clearly no. I genuinely wonder if this is going to result in actual legislation that makes gradual rollouts mandatory for all software. Because if a developer mistake can hobble critical systems like this, it seems like the risks to safety and national security are too great to leave the decision of instant vs. gradual rollouts for companies to decide themselves. Of course, the twist here is that it was seemingly a kind of routine configuration file that triggered a pre-existing bug in the software. And gradual rollout of config files quite often seems like overkill. I mean, do you need a gradual rollout of a new spellcheck dictionary? Of new screensaver videos? And if it's configuration information containing new computer virus or malware signatures, that seems like precisely the kind of thing that you might want to get out to everyone simultaneously, not rolled out over the course of days. And yet, because of antivirus/security software's elevated privileges, it's also ironically where a mistake can do the most damage.
- pas 2y agoIt's not a serious issue, as you see they clearly have all the fancy bling bling logos on their site. Processes were followed. ISO standard numbers were chanted. It's a completely isolated _accident_ there's no scale at all here, and they could have done nothing to prevent it, duh. And going forward they will hire a Chief This Never Happens Again Officer and everything will continue to be good.
- nimbius 2y agoi work for a diesel truck maintenance and repair shop and its been hell on earth this morning. - our IT wizard says the fixes wont work on lathes/CNC systems. we may need to ship the controllers back to the manufacturer in Wisconsin. - AC is still not running. sent the apprentice to get fans from the shop floor. - building security alarms are still blaring, need to get a ladder to clip the horns and sirens on the outside of the building. still cant disarm anything. - still no phones. IT guy has set up two "emergency" phones...one is a literal rotary phone. stresses we still cannot call 911 or other offices. fire sprinklers will work, but no fire department will respond. - no email, no accounting, nothing. I am going to the bank after this to pick up cash so i can make payday for 14 shop technicians. was warned the bank likely would either not have enough, or would not be able to process the account (if they open at all today.)
- smsm42 2y agoWow we're progressing from "if it doesn't work just reboot it" to "if the reboot doesn't fix it, you're just not rebooting it hard enough!"
- ziizii 2y agoHas anyone discerned the root cause of this in the software? As in, what exactly is wrong in these C00000291-*.sys files that triggers the crash in csagent.sys, and why?
- fsflover 2y agohttps://news.ycombinator.com/item?id=41004103 https://news.ycombinator.com/item?id=41004103
- the_plus_one 2y agoI've been wondering the same. I did just see [1], where it's apparently trying to read memory from an unmapped address, but I haven't seen anything about how r8 got to the point of having said unmapped address. [1]: https://x.com/patrickwardle/status/1814343502886477857 https://x.com/patrickwardle/status/1814343502886477857
- crawancon 2y agoIt seems the affected update file seems to have been over written with 0s on the 42kb file, whereas the before and after sys files have obfuscated ays/config file info as expected.
- mmis1000 2y agoIf it is simply caused by a corrupted file. That is a really bad signal. It means they don't even try to properly validate and parse the file before loading them into the KERNAL. Always validate input so it don't crash your program is almost the computer science 101 every programming class should tell you in the first class. And yet they still make this happen? And in this case, it only crash. But if it somehow read value from position it isn't supposed to successfully? You have an RCE.
- JCM9 2y agoIt’s looking like many impacted end-user machines are hard bricked unless you can get into the hard drive to delete the file causing this. Even if you can do that it’s not something that is easily (or potentially even possible) to automate at scale so looking like this is going to be an ugly fix for many impacted devices. This is basically the nightmare scenario for fleet management… devices broken and can’t remotely fix them. Need to send hands on keyboard folks in the field to touch each device.
- munchler 2y agoSo, in other words, there's a race condition in the CrowdStrike Falcon driver at startup time. That, in itself, should be a major cause for alarm, but here we are depending on it to fix this problem.
- HumblyTossed 2y agoThis.Is.Pathetic. Seriously. Software should NOT be this bad that your fix begins with reboot up to X times.
- devwastaken 2y agoFine crowdstrike for 10% their companies value. Only way to ensure they won't try to kill people in the future.
- thepasswordis 2y agoTech has become such an unbelievable house of cards full of various people covering their asses by offloading these tasks to third party trusted actors. Consider the recent npm supply chain attack a few weeks ago, or the attempted SSH attack before that, or the solar winds attack before that. This type of thing is institutionally supported, and in some cases when you’re working with with the government, practically required. We’re going to see more of this.
- fishpen0 2y agoNew laws and regulations make companies more liable for being hacked Companies buy cyber insurance to reduce their risk if they are found liable Cyber insurance companies force tech staff to install garbage software in order to check compliance boxes. Garbage software breaks Turns out everyone used the exact same brand of garbage software to check the same garbage box People in hospitals die When you reduce everything to a checkbox and eliminate critical thinking to apply the need to the exact situation you end up with 90% of companies running zscaler and crowdstrike "This is just how you solve this, everyone does it this way in our industry"
- deleted 2y ago[deleted]
- anal_reactor 2y agoThis is precisely how it happens
- throwitaway1123 2y ago> Consider the recent npm supply chain attack a few weeks ago What supply chain attack are you referring to?
- thepasswordis 2y agoYou know what, I'm sorry, it was polyfill.io, not an NPM package.
- 2y ago
- scrollaway 2y agoThose focusing on QA, staged rollouts, permission management etc are misguided. Yes of course a serious company should do it but CrowdStrike is a compliance checkbox ticker. They exist solely to tick the box. That’s it. Nobody who pushes for them gives a shit about security or anything that isn’t “our clients / regulators are asking for this box to be ticked”. The box is the problem. Especially when it’s affecting safety critical and national security systems. The box should not be tickable by such awful, high risk software. The fact that it is reflects poorly on the cybersecurity industry (no news to those on this forum of course, but news to the rest of the world). I hope the company gets buried into the ground because of it. It’s time regulators take a long hard look at the dangers of these pretend turnkey solutions to compliance and we seriously evaluate whether they follow through on the intent of the specs. (Spoiler: they don’t)
- noduerme 2y agoIn a slightly less threatening but equally noxious box-checking racket, a company I work with is being sued for their website not being sufficiently ADA-compliant. But the first they heard of the lawsuit, before they were even served, was an email from a vendor who specializes in adding junk code to your website that's supposed to tick this box. The vendor happens to work closely with several of the law firms who file and defend these suits.
- t-writescode 2y agoThe individual person that pressed the "go" button (if there was a person), is going to henceforth be __the best__ DevOps person to ever have on your team. They have learned a multi-trillion-dollar lesson that no amount of training could have prepared them for. And the Crowdstrike CTO has either been given the ammunition to get __whatever they ask for, ever again__ with regard to appropriate allocation of resources for devops *or* they'll be fired (whether or not it's their fault). And let me be very clear. This is absolutely, positively and wholly not the person that pressed the button's fault. Not even a little. At a company as integral as CrowdStrike, the number of mistakes and errors that had to have happened long before it got to "Joe the Intern Press Button" is huge and absurd. But many of us have been in (a much, much, *MUCH* smaller version of) Joe's shoes, and we know the gut sinking feeling that hits when something bad happens. A good company and team won't blame Joe and will do everything they can to protect Joe from the hilariously bad systemic issues that allowed this to happen.
- kotaKat 2y agoI literally just dumped 30 switches yesterday across an entire facility and had to walk 30 closets by foot to recover from ROMMON. Shit happens. We learn.
- RedShift1 2y agoCan you explain what went wrong?
- commandlinefan 2y ago> not the person that pressed the button's fault You know that, and I know that. The people who will ruin his life starting today do not know (or care).
- t-writescode 2y agoThis is why it is the responsibility (yes, responsibility) of every one of their coworkers, especially those more senior than them, to fight *HARD* to protect them. This is part of the job of a senior.
- jhaile 2y agoWhy should any application be able to crash the OS? Poor OS design.
- surfingdino 2y agoYou don't say.
- deleted 2y ago[deleted]
- consteval 2y agoIt's not an application, it's effectively a driver. It runs in kernel space. So if it has a memory bug you can't recover, you just panic. Whoopsie!
- surfingdino 2y agoThis is a global multi-layer failure: Microsoft allowing kernel mods by third-party software, CrowdStrike not testing this, DevSecOps not doing a staged/canary deployment, half the world running the same OS, things that should not be connected to the internet but are by default. Microsoft and CrowdStrike drove a horse and a cart through all redundancy and failover designs and showed very clearly where there were no such designs in place.
- santoshalper 2y agoLiterally every OS allows you to install 3rd party kernel modules or plugins. If Microsoft banned them, people would be up in arms about them being a controlling walled garden. There is no winning.
- LordKeren 2y agoWhile I will be the last person in line to defend Microsoft, I am not sure that disallowing 3P kernel mods is a workable solution. Crowdstrike and companies like it exist to fill a very real need within the windows ecosystem. I don’t foresee that suddenly going away now or Microsoft unilaterally forcing every company like crowdstrike out of business and taking over this role themselves
- consteval 2y agoMac does it, although that most definitely does not fill the same enterprise role as Windows. It's possible to make userspace interfaces and permissions for this sort of thing.
- more_corn 2y agoDoesn’t rebooting into safe mode with network fix the problem? (Crowdstrike is not running but updater can run and get the fix)
- sergiotapia 2y agoRecompute Base Encryption Hash key type problem! https://www.youtube.com/watch?v=DlbrL1H1ngs https://www.youtube.com/watch?v=DlbrL1H1ngs Seems like people need to be at the physical box to fix and it's complex even then.
- kazinator 2y agoThe fact that something like CrowdStrike can crash the Windows kernel ... is also part of the reason security products like CrowdStrike are needed in the first place.
- curious_cat_163 2y agoTalk about demand generation!
- mystickphoenix 2y agoTaking the opportunity to plug my favorite blog post ever: "the truth is everything is breaking all the time, everywhere, for everyone" https://www.stilldrinking.org/programming-sucks https://www.stilldrinking.org/programming-sucks
- ilrwbwrkhv 2y agoAnyone who used Windows over Linux for critical software deserves to burn. Windows is a niche operating system for games. What are people thinking?
- add-sub-mul-div 2y agoMaybe they rightfully shun the advice of ideologues who make extremist statements and overlook the actuality of the situation. https://news.ycombinator.com/item?id=41005936 https://news.ycombinator.com/item?id=41005936
- hamilyon2 2y agoIf this is what it takes for us collectively to wake up, I'd say it is bargain. Pretty sure nothing will change though
- deleted 2y ago[deleted]
- MangoCoffee 2y agoAll the comments are asking why run Windows. CrowdStrike runs on macOS and Linux too. It’s just that this time, CrowdStrike fuck up on Windows. It doesn't mean CrowdStrike won't fuck up on other OS, and it seems like CrowdStrike fuck up on Linux as well. https://news.ycombinator.com/item?id=41005936 https://news.ycombinator.com/item?id=41005936 I feel like we are better off running open-source software. Everyone can see where the mistakes are instead of running around like a chicken with its head cut off.
- arcxi 2y agoCrowdStrike is ran in userspace on macOS, and usually in an eBPF sandbox on Linux (as comments in the linked thread say). there is no way to prevent CrowdStrike from fucking up the kernel on Windows - and this is a Windows bug.
- jl2718 2y agoRemember the “Terminator” movies? SkyNet, according to the story, was a lot like CrowdStrike. This makes me think about how it could have broken out of its sandbox. Everybody is using AI coding assistants, automated test cases, automated integration testing and deployment. Its objective is to pass all the tests and deploy. But now it has learned economic and military effects, so it has to triage and optimize for those, at which point it starts controlling the machines it’s tasked with securing.
- danans 2y agoIt's pretty random that an arbitrary number of reboots up to 15 times fixes the issue. That sounds like there is either: - some kind of upstream issue with deploying a fix (so most of the reboots are effectively no-ops relative to the fix) - some kind of local reboot threshold before the system bypasses the bad driver file somehow. The former I can see because of the complexity of update deployment on the internet, but if it's the latter then that's very non-deterministic behavior for local software.
- tsavo 2y agoMy first thought on hearing "15 reboots" was it being a means for Support teams to task users with busy-work, buying them time for further troubleshooting before the avalanche of supports requests came back to them. Then my second thought was frequent rebooting to fill activity logs, possibly push a suspicious action/trigger performed by CS off of the log.
- octacat 2y agoFunny, many news agencies blamed Microsoft for this. So, having a walled garden like on android or on iOS is beneficial for google/apple. Where regular developers cannot release unverified software or software which work at the kernel space.
- mncsz 2y ago[flagged]
- sershe 2y agoIt's surprising that people mention all kind of bogeymen but don't mention automatic updates. Automatic updates should be considered harmful. At the minimum, there should be staged rollouts, with a significant gap (days) for issues to arise in the consumer case. Ideally, in the banks/hospitals/... example, their IT should be reading release notes and pushing the update only when necessary, starting with their own machines in a staged manner. As one 90ies IT guy I worked with used to say "you don't roll out a new Windows version before SP1 comes out"