56 ms·
CrowdStrike Update: Windows Bluescreen and Boot Loops
My workplace has a number of people reporting Windows blue-screening and going into a boot loop. The IT Department have a number of servers recently gone offline and have said there's a chance that the two issues are related, and potentially due to a Crowd Strike application update.
My laptop blue-screened and rebooted, but is working fine after the reboot.
A local radio station has also said they've got the same issues with their laptops and their phone system is down as a result.
Not seeing anything on news sites yet. Anyone else seeing similar?
Above is all based in Australia.
- hhjjjjjjjd 2y ago[flagged]
- nknld 2y ago[flagged]
- deleted 2y ago[deleted]
- sammy2255 2y agoYep happened to us too. Its global. And it just started happening.
- VargaLand 2y agoSame in Adelaide. Reports coming in from Gov agencies, utilities etc.
- scriptsmith 2y agoIt's crowdstrike: https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_error_in_latest_crowdstrike_update/ https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_e... > 7/18/24 10:20PT - Hello everyone - We have widespread reports of BSODs on windows hosts, occurring on multiple sensor versions. Investigating cause. TA will be published shortly. Pinned thread. > SCOPE: EU-1, US-1, US-2 and US-GOV-1 > Edit 10:36PT - TA posted: https://supportportal.crowdstrike.com/s/article/Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19 https://supportportal.crowdstrike.com/s/article/Tech-Alert-W... > Edit 11:27 PM PT: > Workaround Steps: > Boot Windows into Safe Mode or the Windows Recovery Environment > Navigate to the C:\Windows\System32\drivers\CrowdStrike directory > Locate the file matching “C-00000291*.sys”, and delete it. > Boot the host normally.
- ineptech 2y agoCan you post a summary? We're affected but I don't have access to that portal.
- scriptsmith 2y agoSomeone posted this in the thread, but I also can't log in to verify > Summary > CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. > Details > Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor. > Current Action > Our Engineering teams are actively working to resolve this issue and there is no need to open a support ticket. > Status updates will be posted below as we have more information to share, including when the issue is resolved. > Latest Updates > 2024-07-19 05:30 AM UTC | Tech Alert Published. > Support > Find answers and contact Support with our Support Portal
- nofinator 2y agoThey've bumped this support info to a blog post that's linked from their home page: https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/ https://www.crowdstrike.com/blog/statement-on-falcon-content... It includes PDFs of some relevant support pages that someone printed with their browser 5 hours ago. That's probably the right thing to do in such a situation to get this kind of info publicly available ASAP, but still, oof. Looks like lots of people in the Reddit thread had trouble accessing the support info behind the login screen.
- unixhero 2y ago"Start your free trial now." Hahahahah you have got to ne kidding me :)
- Neil44 2y agoRight after you enter the bit locker recovery key. You do have your bit locker recovery key, right? .....right?
- 2y ago
- jamesbfb 2y agoOur company is in panic mode. 15 machines blue screened for no apparent reason and stuck in boot loop. I’m a gloating Linux user :) Also in Australia
- VargaLand 2y agoAccording to Reddit It's hitting Croatia, Philippines, US, Germany, Mexico, India, Japan. SAP servers dropping like flies, that's Defence,Banks, Payroll all affected. Major Retail Chains like Big W down.
- deleted 2y ago[deleted]
- ajdlinux 2y agoStarting to get media coverage: https://www.abc.net.au/news/2024-07-19/technology-shutdown-abc-media-banks-institutions/104119960 https://www.abc.net.au/news/2024-07-19/technology-shutdown-a...
- yen223 2y agoNewsreaders on the ABC are reading off paper notepads lmao
- jaza 2y agoI was listening to Triple J (one of ABC's radio stations), they said: "welcome to our first and possibly last ever Triple J's USB Fridays, we can't play any of our usual music because the computers are all down, all we can play is the songs that happen to be on the USB stick that one of us had in our pocket". LOL!
- gamma032 2y agoAustralian news reporting this has hit hospitals, fire and rescue, banking, media, airlines and many other companies worldwide.
- Napsty 2y agoyep spreading quickly now. https://www.geekersdigest.com/windows-blue-screen-crash-caused-by-bad-crowdstrike-update/ https://www.geekersdigest.com/windows-blue-screen-crash-caus... https://www.timesnownews.com/technology-science/latest-crowdstrike-update-causes-blue-screen-of-death-on-microsoft-windows-multiple-users-affected-article-111854018 https://www.timesnownews.com/technology-science/latest-crowd... This is gonna be a tough Friday for IT departments...
- Hawxy 2y agoThe impact is Australia is immense. https://downdetector.com.au/ https://downdetector.com.au/, it's almost every major org in the country. NSW Government is completely offline.
- thrdbndndn 2y agoI assume you have to install "CrowdStrike" yourself (i.e. not bundled with Windows by default)? I have no idea what it is before.
- sammy2255 2y agoIts paid antivirus software, they cater to businesses
- alams 2y agoAnyone found any fixes, while Crowdstrike comes up with a fix?
- jimmy_lbts 2y agowe will found a troubleshooting , we start in revovery mode and rename the path C:\windows\system32\drivers\Crowdstrike
- h4kunamata 2y agoDo this at your own risk!! Apparently there are two NON-OFFICIAL solutions: 1. Rename csagent.sys ( the file causing the BSOD ) 2. Rename c:\windows\System32\Drivers\Crowdstrike Again, do this at your own risk. Both workaround have been reported as "working". I am Linux user so I cannot tell.
- michelevr 2y agoHusband is a deputy in California. His department and many others here are down as well (including PDs, jails, ambulance companies, etc.)
- mcast 2y agoThis seems like a pretty severe point of failure.
- aenis 2y agoWe have ~50 thousand laptops in reboot loop and ~1.5k servers as well. No resolution yet.
- beardedowl 2y ago[dead]
- deleted 2y ago[deleted]
- bahtiyarTKM 2y ago[dead]
- beardedowl 2y ago[dead]
- zcretu 2y agoQuick fix that worked for us, In safe mode: 1.enter in drive C: 2.system 32 folder 3. Drivers 4. Rename crowdstrike folder to something else doesent matter what.
- _kyran 2y agoWas just using the energy vic website and thought I'd been rate limited when their API stopped working. Seems like it could be this.
- BLKNSLVR 2y agoIn terms of analysing risk factors to minimise something like this happening again, what are the factors at play here? A Crowdstrike update being able to blue-screen Windows Desktops and Servers. Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn't be able to do this kind of damage to an operating system. This makes me think that, maybe, Crowdstrike were unlucky enough to have accidentally discovered a bug that affects multiple versions of Windows (ie. it's a Windows bug, maybe more-so than it is a Crowdstrike bug). There also seems to have been a ball-dropped in regards to auto-updating all the things. Yes, gotta keep your infrastructure up to date to prevent security incidents, but is this done in test environments before it's put into production? Un-audited dependence on an increasingly long chain of third-parties. All the answers are difficult, time consuming, and therefore expensive, and are only useful in times like now. And if everyone else is down, then there's safety in the crowd. Just point at "them too", and stay the path. This isn't a profitable differentiation. But it should be! (raised fists towards the sky).
- aloha2436 2y ago> an application shouldn't be able to do this kind of damage to an operating system Antivirus software by its nature probably needs the kind of access that would let it bluescreen your computer.
- BLKNSLVR 2y agoWading out my depth here, so forgive any stupidity following. And there's a certain amount of sense to that, it has to get "under" the layer that viruses can typically get to, but I still think there should be another layer at which the OS is protected from misbehaving anti-virus software (which has been known to happen).
- eddythompson80 2y agoIt's a kernel mode driver. There aren't layers in kernel drivers. Any kernel module/driver can crash your system if it wants to.
- kswap 2y agoFaced the same issue few minutes back after few loops of reboot my system is up
- aenis 2y agoWe had a few machines come out of the boot loop - only to re-enter it 20 mins later. I am sure CS pulled the patch from their CDNs but ...maybe some cached versions still linger?
- 7am00dee 2y agoA system restore helps. But obviously not when you’ve got an environment of ~500 or more clients
- tru3_power 2y agoCrazy- wasn’t Azure having an outage earlier today? Is this related?
- BLKNSLVR 2y agoI'd guess that it's related. Azure, OneDrive, and Office 365 were having issues this morning Australian time (about 6 hours ago).
- jdleel 2y agoDoesn't actually look like. Noone really figured out the solution yet
- tru3_power 2y agoIf that’s the case I wonder why we’re only now seeing everyone’s systems BSODing. Should be an interesting write up (if they release one)
- cpf_au 2y agoThis is effecting our company. A colleague visited her local supermarket (Woolworths) and all the self-service checkouts were effected.
- Scoundreller 2y agoSilver lining?
- deleted 2y ago[deleted]
- vinura 2y agoAnyone has a good news? or still it's on the BSOD loop?
- tgshaik 2y agouse windows recovery to boot into safe mode and update latest crowdstrike. That is the only option at this time.
- ali2724 2y agoHarvey Norman’s system is down
- l0g4n_me 2y agoc:\system32\drivers\csagent.sys renaming this file on server with safe mode boot fixes the issue but disables agent
- BLKNSLVR 2y agoFlippant commentary: Thank fuck Netflix runs on Linux. I just hope the full chain from my TV to Netflix is immune...
- jmprspret 2y agoIt actually runs on a lot of FreeBSD iirc
- BLKNSLVR 2y agoEven better!
- Klonoar 2y agoNetflix uses FreeBSD for certain parts, but IIRC it's Linux outside of the streaming delivery. (Some of the team members lurk around here, maybe we'll get lucky and one comments)
- kohukohunui 2y agoI'd say regardless of the OS, you might find a company like Netflix is less likely to impose security-theatre box ticking exercises. Which makes it less likely to take a 3rd party agent from a snakeoil company that sells to execs, then embed it at low levels into mission critical services with elevated privileges, then give it realtime external updates that can break your platform at any point.
- sagarmitkar 2y ago[flagged]
- deleted 2y ago[deleted]
- sakopov 2y agoIf the workstations are stuck in a boot loop, how will they be able to push a hotfix out?
- ernth_16 2y agoUninstall the Current Version: Open the Command Prompt as an administrator. Run the following command to uninstall the current version: shell sc delete csagent
- jmcgough 2y agoMy entire emergency department got knocked offline by this. Really scary when you have ambulances coming in and are trying to stabilize a heart attack. Update: 911 is down in Oregon too, no more ambulances at least.
- sexy_seedbox 2y agoBut at least the best instant messaging app in the world Microsoft Teams and the best web browser in the world Microsoft Edge are working fine, right?
- jmcgough 2y agoIt's a bsod loop, so not really.
- jeffrallen 2y ago"Always look on the bright side of life!" - M Python
- spike021 2y agoDo you have offline backup processes at least? Nasty situation.
- jmcgough 2y agoWe're really prepared for epic to go down and have an isolated cluster that we access in emergencies. I transitioned from software engineering so I've only been in the ED for a year, but from what I could see there didn't seem to be a plan for what to do if every computer in the department bluescreened at once.
- prmoustache 2y agoAren't they doing canary release? Seems weird this would not have been detected on a smaller scale before with a good release process.
- makmanalp 2y agoCanary releases aren't a magical bug free fix. They might be doing it, but the conditions that trigger a problem can be sneaky and can happen outside your canary period, rendering it useless. It's a best effort method.
- prmoustache 2y agoIt depends on how global the issue is. But yeah I've seen US companies for example only doing their initial releases in the USA only which has zero value for issues that might appear with different localization/language settings for example.
- ikekkdcjkfke 2y agoBuying 10 computers with different configurations is not hard either. They are just lazy
- rubi1945 2y agotemporary fix is released https://x.com/s0lst1c3/status/1814182835269337548 https://x.com/s0lst1c3/status/1814182835269337548
- dalmo3 2y agoLooks like this also took down half of New Zealand's economy. https://www.nzherald.co.nz/nz/bank-problems-reports-bnz-asb-kiwibank-anz-visa-paywave-services-down/R2EY42QKQBALXNF33G5PA6U3TQ/ https://www.nzherald.co.nz/nz/bank-problems-reports-bnz-asb-...
- starkindustries 2y ago[dead]
- Taniwha 2y agoSatellite TV channels too
- Reason077 2y agoCrowdstrike must have some slick salespeople in New Zealand. Seems like nearly everyone uses it. Single point of failure problem.
- 0x800838383 2y agoCIO here. They are known to be incredibly pushy. In my company we RFP'd for our endpoint & cyber security. Found the CS salesperson went over me to approach our CEO who is completely non-technical to try and seal a contract because I was on leave for 1 week out of service (and this was known to them). When I found out by our CEO informing me of the approach we were happy to sign with SentinelOne
- sam_bristow 2y agoOne thing I'm really happy about at my current company is that when a sales person from a vendor (not Crowdstrike) tried that our CEO absolutely ripped them a new one and basically banned that company from being a vendor for a decade.
- dataengineer56 2y agoI got confused and thought that you were the CIO of Crowdstrike until I read further into your comment.
- starkindustries 2y ago[dead]
- jafru 2y ago"F8 + last known good configuration" worked for us
- gamma032 2y agoThe biggest mistake here is running a global update on a Friday. Disrespect to every sysadmin worldwide.
- usr1106 2y agoDisrespect to every CIO to make their business depend on a single operating system, running automatic updates of system software without any canaries and phased deployments.
- dist-epoch 2y agoYou're saying I should diversify my 100% Linux operation to also use Windows?
- ykonstant 2y agoMaybe some OpenBSD would be a good hedge. It can also help spot over-reliance on some Linux quirks.
- prmoustache 2y agoWhat makes you think windows is the only alternative? Have you never heard about Gnu Hurd? More seriously I am not saying you should run some critical services on menuetos or riscos but the BSDs are still alive and kicking as well as illumos and its derivatives. And yes I think a bit of diversity allows some additional resilience. It may necessitate more workforce but imho it is worth the downsides.
- usr1106 2y agoWhile I believe Linux is a more reasonable operating system than Windows, shit can happen everywhere. So if you have truly mission critical systems you should probably have more have at least 2 significantly different systems, each of them being able to maintain some emergency operations independently. Doing this with 2 Linux distros is easier than doing it with Linux and Windows. For workstations Macs could considered, for servers BSD. Probably many companies will accept the risk that everything goes down. (Well, they probably don't say that. They say maintaining a healthy mix is too expensive.) In that case you need a clearly phased approach to all updates. First update some canaries used by IT. If that goes well update 10% of the production. If that goes well (well, you have to wait until affected employees have actually worked a reasonable time) you can roll out increasingly more. No testing in a lab (whether at the vendor or you own IT) will ever find all problems. If something slips through and affects 10% of your company it's significantly different from affecting (nearly) everyone.
- Cub3 2y agoWorkaround + update (within a authenticated portal) https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_error_in_latest_crowdstrike_update/ldvwkbn/ https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_e...
- poshlandpro 2y agoOfficial CrowdStrike workarround: 1. Boot Windows into Safe Mode or the Windows Recovery Environment 2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory 3. Locate the file matching “C-00000291*.sys”, and delete it. 4. Boot the host normally.
- thallaran 2y agoall Delta flights grounded
- thallaran 2y agoAll flights grounded. World wide airline systems outage.
- jjjjjjjjjjjjjjj 2y agoJust boarded a flight in Iceland, no mention of being grounded.
- kalleboo 2y agoIt only applies to a few US carriers https://abcnews.go.com/US/american-airlines-issues-global-ground-stop-flights/story?id=112092372&cid=social_twitter_abcn https://abcnews.go.com/US/american-airlines-issues-global-gr...
- Scoundreller 2y ago“ American Airlines, United and Delta have asked the FAA for global ground stop on all flights, according to an alert from the FAA on Friday morning. The FAA is telling air traffic controllers to tell airborne pilots that airlines are currently experiencing communication issues.”
- hasindh09 2y agoHP laptops are booting and its looping DELL laptops are observed, after blue dump . Server is up and running fine Temporary workaround leads compliance issue. From India
- 1221 2y ago[dead]
- starkindustries 2y ago[dead]
- jkells 2y agoI don't understand why this outage and the Azure outage earlier don't make it to the front page. I'm getting more up to date technical details from the regular media. This outage looks to be huge.
- wfme 2y agoAgreed, I assume people are flagging this post for some reason?
- speedgoose 2y agoIt’s most likely the flame war filtering algorithm of HN. Posts that create a lot of discussion quickly are down ranked until an admin fix the rank manually, or not.
- blinkersareon 2y agoThe impact is more than huge. Our whole workforce is impacted as the world's largest tech service provider.
- Scoundreller 2y agoY2K came 24+ years late!
- DaoVeles 2y agoAnd 12 years before the Unix epoch!
- Scoundreller 2y agoMost HNers are on the clock while browsing (for work purposes of course) and forcibly afk at the moment.
- d1sxeyes 2y agoI'd guess also a significant proportion of us are involved in our company's response to the incident.
- larrymcp 2y agohttps://archive.is/AWVuv https://archive.is/AWVuv
- nallerooth 2y agoThanks!
- youngun11 2y ago[dead]
- deleted 2y ago[deleted]
- protocolture 2y agoDont see how anyone is getting out of this without applying the workaround or reimaging their whole fleet.
- kcd83 2y agoFix: Boot Windows into Safe Mode or the Windows Recovery Environment Navigate to the C:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Boot the host normally Did it work for you?
- LeoPanthera 2y agoBBC live coverage: https://www.bbc.com/news/live/cnk4jdwp49et https://www.bbc.com/news/live/cnk4jdwp49et Looks like this is a big deal.
- deleted 2y ago[deleted]
- deutschlerner 2y agohttps://www.bbc.co.uk/news/live/cnk4jdwp49et https://www.bbc.co.uk/news/live/cnk4jdwp49et, seems to be quite a wide impact from this, e.g. Sky News in the UK is off air!
- rosege 2y agoSo its not all bad then!
- Melatonic 2y agoUS Based and got a NANOG alert email just in time. At least half our windows servers down. I went into our crowdstrike policies and disabled auto update of the sensor. Hopefully this means it doesnt hit everything. Double check your policies!!! Edit: Crowdstrike has an article out on the manual fix: https://supportportal.crowdstrike.com/s/article/Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19 https://supportportal.crowdstrike.com/s/article/Tech-Alert-W...
- markus92 2y agoAll major US airlines have put in a total ground stop. No flights can take off anymore.
- kurthr 2y agoElectronic payment at grocery stores in the US is affected.
- deleted 2y ago[deleted]
- Scoundreller 2y agoPOS down? Payment processor down? Lotta different grocery stores and processors in USA. WHO?
- chgs 2y agoI’m Amazed at that. I had no idea the US had caught up and had electronic payments.
- dclowd9901 2y agoI’m guessing for chip reading? It’s one-time transaction key so I don’t think it works async.
- anal_reactor 2y agoAll five payment terminals in Germany are down. "Unfortunately, all online governmental services are affected. All two of them" says the chancellor.
- weberer 2y agoWhat are you talking about? Credit cards have been around since the 1950's
- Kye 2y agoFor much of their history they were written down or copied on carbon paper and manually processed by phone later. Electronic processing came in the 70s and wasn't universally used until much later. I saw plenty of credit card imprinters in use well into the 90s when I was growing up. https://en.wikipedia.org/wiki/Credit_card_imprinter https://en.wikipedia.org/wiki/Credit_card_imprinter Card issuers only stopped embossing them recently.
- tloriato 2y agoGLOBAL OUTAGES - Major banks, media and airlines affected by major IT outage - Significant disruption to some Microsoft services - 911 services disrupted in several US states - Services at London Stock Exchange disrupted - Sky News is off air - Flights in Berlin grounded - Reports the issue relates to problem at global cybersecurity firm Crowdstrike
- bux93 2y agoGood news for crowdstrike! It shows how critical their services are. Stock to go up! (And down a bit when they get sued, and up a bit when they don't get sued too much, etc.)
- chucky_z 2y agoTake a look at their stock right now.
- layoric 2y agoTried, "Real time quote data is not available". Big brain move, can't crash the stock price if the nasdaq is offline..
- trollstrike 2y agoYes, this has had no (statistically significant) effect.
- 2y ago
- aenis 2y agoWe are a major CS client, with 50k windows-based endpoints or so. All down. There exists a workaround but CS does not make it clear whether this means running without protection or not. (The workaround does get the windows boxes unstuck from the boot loop, but they do appear offline in the CS host management console - which of course may have many reasons).
- kurthr 2y agoYeah, the simple renaming .sys files in safe mode does seem like it would inhibit protection.
- Scoundreller 2y agoRenaming it .old or whatever would be what a sane person does. They recommended deleting the thing: https://news.ycombinator.com/item?id=41002199 https://news.ycombinator.com/item?id=41002199
- saagarjha 2y agoWhile you're at it probably delete the rest of the software. Then cancel your contract.
- InsideOutSanta 2y agoThen sue the shit out of them.
- hello_moto 2y agoThen your hardware becomes unprotected, congratulations, you won Cybersecurity award of the year.
- saagarjha 2y agoThere are more options than "you are using Crowdstrike" and "you have no protection".
- martypitt 2y agoDiscussed more thoroughly here: https://news.ycombinator.com/item?id=41002195 https://news.ycombinator.com/item?id=41002195 (Not sure why that's not on the frontpage)
- mirekrusin 2y agoBecause ppl are locked out to say they are locked out.
- pvg 2y agoProbably shoved off by the flamewar detector.
- itsgrimetime 2y agoI just landed at SeaTac an hour ago and the rideshare/app pickup was absolutely nutso. Like thousands of people standing around waiting for taxis and Ubers. The one person I asked what was going on said that the computer systems at all the regional hotels are down (not sure how that makes more people need cabs). Wonder if it’s from this
- itsgrimetime 2y agoJust realized this is posted on the SeaTac website now: “ SEA is experiencing temporary issues with the system that populates flight and baggage information on in terminal screens and the flySEA app/website. Travelers are recommended to check with their airlines for current gate and baggage claim information. Check With Your Airlines”
- Scoundreller 2y agoI remember when my big regional train system had like 95% of its morning trains cancelled overnight due to a big snowstorm in 2014. Of course, no news on its websites or socials because those people didn’t start until 9am. I think they finally fixed that.
- Klonoar 2y ago> just landed at SeaTac an hour ago and the rideshare/app pickup was absolutely nutso. Like thousands of people standing around waiting for taxis and Ubers. So a normal day at SeaTac?
- ThePhysicist 2y agoIf it's true that a bad patch was the reason for this I assume someone, or multiple people, will have a really bad day today. Makes me wonder what kind of testing they have in place for patches like this, normally I wouldn't expect something to go out immediately to all clients but rather a gradual rollout. But who knows, Microsoft keeps their master keys on a USB stick while selling cloud HSM so maybe Crowdstrike just yolos their critical software updates as well while selling security software to the world.
- nilsb 2y agoWho needs testing when apologizing to your customers is cheaper?
- falcor84 2y agoI would assume that its enterprise customers have an uptime SLA as part of their contract, and that breaching it isn't very cheap for Crowdstrike.
- perbu 2y agoSoftware doesn't have uptime guarantees. They might have time-to-fix on critical issues, though. I assume this is gross negligence, which would leave them open to claims made through courts, though.
- jsiepkes 2y agoI highly doubt their SLA says something about compensating for damages. At most you won't have to pay for the time they were down. And even more ironically; A botched update doesn't mean they are down. It means you are down. So I don't even think their SLA applies to this.
- InsideOutSanta 2y agoYeah, they'll pay with "credits" for the downtime, if what is currently happening even technically qualifies as downtime.
- joeldo 2y agoThe impact of this will be profound! Obviously bugs are inevitable, but why this wasn't progressively rolled out is beyond me.
- raverbashing 2y agoAre people counting this on the Windows TCO?
- raggi 2y agothey have kernel modules for macos and linux too afaik, so i wouldn't be counting those chickens too fast
- crycombinator 2y agoI would expect this to be a kernel specific bug. I'm on a company laptop with falcon, and we have linux systems using the same, no signs of problems so far.
- Kwpolska 2y agoIt's not Microsoft's fault someone installed third-party spyware/malware on their systems.
- mrdeveloper16 2y agoGo to advance repair option then advanced open cmd. Go to windows/system32/drivers/crowdstrike. Then list all the file and delete file name having 291 at the end using cmd "del filenameendingwith291"
- RedShift1 2y agoThis is why you don't make changes on a Friday. Lots of weekends absolutely ruined now.
- mirekrusin 2y agoEarly weekend here :D
- YoboDev 2y agoAll US flights are grounded too. The people I was traveling with cant check into hotels
- deleted 2y ago[deleted]
- lpcvoid 2y agoMaybe the world can finally reconsider their use of software products that cater to security theater. And the politics in companies which lead to things like this being introduced ("nobody gets fired for buying IBM"). Edit: took out a bit of snark.
- westpfelia 2y agoI dont know how CS is considered snake oil. Or what IBM has to offer. CS and S1 are really just the best out there.
- kasabali 2y agowell, apparently they aren't
- lpcvoid 2y ago"Endpoint protection" is just the new, hip term for antivirus/intrusion prevention/incident logging of the past. Why not provide immutable Linux based machines (like Chromebooks, Fedora Silverblue) which are locked down outside of the browser? I am aware that this isn't possible in some areas of the industry that rely on large amounts of Windows-only desktop software, but in many cases it may be worth a thought. If I am being naive here, happy to hear other opinions. But I hate opening my company Windows laptop and having the fans turn to 11 just because some "security" software is parsing random files for malicious signatures or running an update that BSOD loops.
- hcs 2y agoThe IBM quip is an old saying: pick the dominant vedor because even if it's a mistake no one will blame you for it.
- crazytony 2y agoHave spent all my afternoon and all evening on a bridge trying to support flailing systems. Was supposed to be on a plane in 5 hours to start my vacation. Guaranteed it's not gonna happen. With hearing 911 and other safety critical systems going down, I hope that the worst that comes out of this is a couple delayed flights and a couple missed bank payments.
- baq 2y agoGood news (um, as in better than the bad news today) is the plane won't be taking off anyway, so you're golden.
- tamimio 2y agoOTA update went wrong? How can an update go live without proper testing for the millions of live connected endpoints?
- pageandrew 2y agoCan they recover this OTA considering the systems can’t even boot?
- DaoVeles 2y agoSo far requires going into recovery mode and removing/rename the cloud strike executable. Then you can boot into Windows from there, it will probably be a sys admin thing dependent on the organisation setup.
- tamimio 2y agoI’m sure the patch itself can be fixed, and there will be a workaround to boot up the machine to fix it. My only concern is the BitLocker keys. If the hard drive is encrypted by Windows and assuming no backup for that key has been done, the system admins will have to activate their disaster recovery plans for these devices, and I hope they have that too, but hope isn’t a strategy!
- pageandrew 2y agoWhy would the BitLocker keys not be recoverable?
- tamimio 2y agoIf the keys aren’t backed up, you will be locked out of the system, and as soon as you try to boot into the safe mode to perform that workaround, you will be asked to enter it manually (or if you have it back it up on a USB drive), if you don’t have, or don’t know the key, you will have an encrypted drive with all of your data locked there.
- deleted 2y ago[deleted]
- sneak 2y agoMaybe one day we will stop giving RCE to so many vendors via auto update.
- attentive 2y agoI am going to stop saying this but people don't realize CS has an official RCE as a feature. As in run remote commands as root/admin on windows or linux/mac through their web.
- Havoc 2y agoThis is why I subscribe to /r/sysadmin despite not being one ... like a canary in the coalmine for stuff like this
- kitd 2y agoA reminder why switching off auto-update is a thing.
- seanieb 2y agoIt looks like it wasn’t a software update, it was a AV definitions update, so internal to the CA application.
- kitd 2y agoTrue, though tbf it's still part of the running system. I read that many of those affected are global orgs. When I worked at an oil major, everything was tested to oblivion before going into production in the DCs, the reason being to avoid precisely this kind of situation where at all possible. There were clusters set aside for operational acceptance testing to ensure everything, from business application right down to kernel, ran successfully. The idea of leaving auto-update on in any production system was unthinkable. Yet here we are.
- balex 2y agoAdmin: We should turn off the AV auto update in prod and test it in staging first. Manager/CISO: That would increase our exposure time on zero day vulnerabilities. Overruled.
- bb123 2y agoThere appears to be a workaround but my question is how are they going to get all of these endpoints out of a BSOD loop?
- DaoVeles 2y agoMight need to manually do it. Depends on if there is any lower level admin access than windows for each system.
- chad1n 2y agoThe antivirus did its job, now you can't get viruses. Jokes aside, I've checked their website and it was full of AI buzzwords so I guess that happens when you focus on nonsense instead of what your customers actually need (I know that all antiviruses have a machine learning component, but usually you don't advertise it as some sort of AI to get better stocks).
- surfingdino 2y agoBack in the 1990s when Microsoft wanted to enter the embedded systems market there was a saying "You don't want Windows controlling your car's breaks". We now let them control a huge part of our lives. Should we let them add AI to the already unpalatable cocktail?
- novaRom 2y agoChances if Microsoft or Crowdstrike will be held liable for financial losses caused by this outage?
- rwmj 2y agoNone whatsoever, their contracts with customers will limit liability to the price paid for the software/subscription. If there was open-ended liability for software failures then very little software would get written.
- account42 2y agoThis is an insane take. Do you think other industries get away with limiting their liability to the product cost? No, because that doesn't provide adequate incentives for making a safe product. The amount of software that gets written depends mostly on the demand for that software. Even if Micrososft would not be willing to up their game to make the risk viable then someone else would.
- rwmj 2y agoThe thing is we know how to make (eg) food that is safe or to a lesser extent bridges that don't fall down. If you sell food that makes people sick you should have known how to avoid that and so you can be held liable. We don't have a good idea how to make software that is flawless, at least, not at scale for a cost that is acceptable. This is changing a little bit now with the drive by governments to use memory-safe languages, but that only covers a small part of the possible spectrum of bugs in software and hardware.
- iamkneel 2y agoCan't wait for the Kevin Fang video about this.
- chall84321 2y agoi work liquor distribution in the united states and our entire company is out across 44 states, allegedly due to this “crowd strike outage”
- nmcveity 2y agoThis gem from the ABC news coverage has my mind 100% boggled: "711 has been affected by the outage … went in to buy a sandwich and a coffee and they couldn’t even open the till. People who had filled up their cars were getting stuck in the shop because they couldn’t pay." Can't even take CASH payment without the computer, what a world!
- skerit 2y agoI'm seeing several reports of things like being unable to buy tickets for the train on-line in Belgium. They use Windows as a part of their server infrastructure?
- timthorn 2y agoAt least they'd take cash if the computer wasn't broken. That's getting quite rare in the UK.
- physicsguy 2y agoNot really? I've only really seen people not taking cash at trendy street food stalls and bougie coffee shops, pretty much everywhere else does.
- orf 2y agoIn London it’s really common
- physicsguy 2y agoAt where though? The example given was in 711 which is a nationwide chain a bit like a Tesco Express or Sainsbury's Local, both of which still accept cash nationwide in the UK too.
- xdennis 2y agoAldi, apparently. That's where Piers Corbyn couldn't buy strawberries with cash. https://www.mirror.co.uk/news/uk-news/piers-corbyn-splits-opinion-latest-30601946 https://www.mirror.co.uk/news/uk-news/piers-corbyn-splits-op...
- deleted 2y ago[deleted]
- bamboozled 2y agoThe world just became a slightly better place.
- deleted 2y ago[deleted]
- rwmj 2y agoCan someone explain what Crowdstrike actually is? Reading Wikipedia it seems to be some sort of anti-virus software?
- evgen 2y agoIt is one of the best systems available for realtime protection of windows systems against various threat actors. Prior to today you could probably have said 'no one gets fired for recommending Crowdstrike as the security tool for the company.' It is everywhere and in particular if you are a large org with a lot of Windows seats you are likely a Crowdstrike customer.
- rwmj 2y ago> realtime protection of windows systems against various threat actors So it's AV + a firewall? What does it actually do?
- zelos 2y agoConsume a lot of CPU and occasionally delete development build artifacts?
- steve1977 2y agoSounds just like your average antivirus then.
- lizknope 2y agoWhat the heck is it doing? My work laptop fan always seems to be blasting air whether it is 10pm or 3am. It's in a reboot loop now so I just shut it off. All my Linux machines are all quiet when nothing is running. In contrast I go to the bathroom at 10pm or 3am and the work laptop fan is blasting. I've logged and see some other security stuff taking up CPU cycles but it happens at least a few times an hour. I wonder how much electricity the world is wasting with this crap. When I first got the laptop when I started this job 5 years ago I thought it must be infected with malware because it was always running the fan so I put it in a separate VLAN so it can't attack my home Linux machines. IT told me it is security software. Who knew that the cyber attack would come from inside the security software.
- chgs 2y agoIs this just a massive mistake or is it deliberate and cover for something else
- Scoundreller 2y agoGood question. Weaponized/compromised update? Disgruntled employee Logic bomb?
- tkubacki 2y agoIndustry should move to Linux on desktop - we should not rely on single vendor
- WediBlino 2y ago2024!!! The year of Linux on the desktop!
- alex_suzuki 2y agoFinally! And here I thought it would never come.
- pi-rat 2y agoWe got day or maybe even week of linux on the desktop at least :)
- GardenLetter27 2y agoAt least Linux servers, this situation is crazy.
- oldmanyells68 2y agoDon't forget about the BSD's. We should not rely on a single Finnish man.
- tgshaik 2y agohttps://www.dailymail.co.uk/news/article-13650411/Microsoft-Crowdstrike-outage-Woolworths-seveneleven.html https://www.dailymail.co.uk/news/article-13650411/Microsoft-...
- junto 2y agoPlease at least an archive link. That website is trash.
- techie128 2y agoThis is good and bad. This showcases the importance of CrowdStrike. This is a short term blip but in the long run they will learn from this and prevent this type of an issue in the future. On the flip side, they have a huge target on their back for the U.S. government to try and control them. They are also a huge target for malicious actors since they can clearly see that CS is part of critical US and western infra. Taking them down can cripple essential services. On a related note, this also demonstrates the danger of centralized cloud services. I wish there were more players in this space and the governments would try their very best to prevent consolidation in this space. Alternatively, I really wish the CS did not have this centralized architecture that allows for such failure modes. Software industry should learn from great & age old engineering design principles. For example, a large ships have watertight doors that prevent compartments from flooding in case of a breach. It appears that CS didn't think the current scenario was not possible therefore didn't invest in anything meaningful to prevent this nightmare scenario.
- choeger 2y agoOr, and that maybe a radical idea, YOU DON'T INSTALL THIS FUCKING SNAKE OIL IN THE FIRST PLACE. The idea of antivirus software is laughable when Adobe cannot implement a safe and secure PDF parser then how can Crowdstrike while simultaneously supporting the parsing of a million other protocols? Everyone involved: Vendor, operator, and auditors who mandate this shit are responsible and should be punished. YOU HAVE TO MINIMIZE THE ATTACK SURFACE, NOT INCREASE IT.
- anigbrowl 2y agoI'm not that confident that they're going to be around to recover from after their stock price falls into the toilet and they get sued out the yin-yang. I don't think 'read the EULA terms lol' is gonna cut it here.
- alt227 2y ago> This is a short term blip No security engineer in the world is going to trust the words CrowdStrike after this.
- 2y ago
- ranjanprj 2y agoHigh time to stop using Microsoft Windows/Azure which is full of security tech debt, that you need all these tools which themselves brick the computer
- onion2k 2y agoCrowdStrike Falcon has a Linux product line for 'cloud security'.
- sofixa 2y agoIf anyone feels like disagreeing about Azure, here's a comment of mine from a few months ago: A random selection of serious security incidents from Azure: just from Wiz from the past 2-3 years, and of course they aren't the only ones: https://www.wiz.io/blog/secret-agent-exposes-azure-customers-to-unauthorized-code-execution https://www.wiz.io/blog/secret-agent-exposes-azure-customers... https://www.wiz.io/blog/storm-0558-compromised-microsoft-key-enables-authentication-of-countless-micr https://www.wiz.io/blog/storm-0558-compromised-microsoft-key... https://www.wiz.io/blog/azure-active-directory-bing-misconfiguration https://www.wiz.io/blog/azure-active-directory-bing-misconfi... https://www.wiz.io/blog/omigod-critical-vulnerabilities-in-omi-azure https://www.wiz.io/blog/omigod-critical-vulnerabilities-in-o... https://www.wiz.io/blog/chaosdb-explained-azures-cosmos-db-vulnerability-walkthrough https://www.wiz.io/blog/chaosdb-explained-azures-cosmos-db-v... Of course Microsoft AI researchers sucking at security: https://www.wiz.io/blog/38-terabytes-of-private-data-accidentally-exposed-by-microsoft-ai-researchers https://www.wiz.io/blog/38-terabytes-of-private-data-acciden... Nice overview from Corey Quinn that predates some of those but things were already horrifically bad: https://www.lastweekinaws.com/blog/azures-terrible-security-posture-comes-home-to-roost/ https://www.lastweekinaws.com/blog/azures-terrible-security-... Go and look for similar things for AWS and GCP, and there's nothing on this level (cross-tenant, trivial to exploit). Oh and there's also this, them selling your usage patterns to partners (hopefully they've stopped): https://twitter.com/QuinnyPig/status/1359769481539506180 https://twitter.com/QuinnyPig/status/1359769481539506180 Oh and another one where they bungled the response: https://twitter.com/QuinnyPig/status/1536868170815795200 https://twitter.com/QuinnyPig/status/1536868170815795200 I find it impossible to believe that Azure as a whole organisation takes security seriously. There might be individuals that do, but definitely nobody with decision making power. Half of the above described exploits are trivial and should have never passed any sort of competent review process.
- oldmanyells68 2y agoSounds like a good time to buy Red Hat stock
- trollstrike 2y agoSounds like a time to buy SPX (always has been). Joke being I have given up trying to time markets :-). With some rare exceptions to the rule (once in 10 year type things).
- 120bits 2y agoFunny how I got rejected today from crowdstrike because I couldn’t code a hard leetcode problem under 40mins. I guess leetcode isn’t true software engineering after all.
- edf13 2y agoThis is a testing and deployment issue rather than coding... mistakes and bugs happen - but most serious businesses have routines setup to catch them before rolling them out globally!
- resonious 2y agoSo maybe they should swap out the Leetcode for a testing and deployment test during their interview!
- saagarjha 2y agoYou probably weren't good enough to take down half the world's Windows systems.
- hackeraccount 2y agoAlternately if he had gotten the job maybe he would have take down three quarters of the world's Windows systems.
- ssss11 2y agoBreaking patches is? …how did this get through QA with a big enough issue that it breaks many many windows machines
- monksy 2y agoI'm about willing to bet they don't have a qa team.
- CapeTheory 2y agoEvidently you dodge bullets well, you should consider running for office.
- qalmakka 2y agoWhen will people learn? 1. Stop putting mission critical systems on Windows, it's not the reliable OS it once was since MS has cut off most of its QA 2. AV solutions are unnecessary if you properly harden your system, AV was needed pre-Vista because Windows was literally running everything as Administrator. AV was never a necessity on UNIX, whatever MS bundles in is usually enough 3. Do not install third party software that runs in kernel mode. This is just a recipe for disaster, no matter how much auditing is done beforehand by the OEM. Linux has taught multiple times that drivers should be developed and included with the OS. Shipping random binaries that rely on a stable ABI may work for printers, not for mission critical software.
- jmcgough 2y agoNone of this advice is useful for massive organizations like banks and hospitals who got hit by this. They cannot switch off of windows for a number of reasons.
- gloosx 2y agoThey are not willing perhaps? Why cannot they?
- cromka 2y agoBecause reasons!
- aenis 2y agoI'd attempt an answer. 1/ imagine running >1000 legacy applications, some never updated in 20 years 2/ imagine a byzantine mix of local data centers, VPCs in aws/gcp/azure 3/ imagine a IT departament run by a lot of people who have never learned anything new since they were hired That would be your typical large, boring entity such as a bank, public utility or many of the big public companies. Yeah, there is no law of physics preventing this, but it's actually nearly impossible to disentangle an organization from decades of mess.
- 2y ago
- jmcgough 2y agoTook down our entire emergency department as we were treating a heart attack. 911 down for our state too. Nowhere for people to be diverted to because the other nearby hospitals are down. Hard to imagine how many millions of not billions of dollars this one bad update caused.
- ibeff 2y ago[flagged]
- deutschlerner 2y ago[flagged]
- dang 2y agoIt's against the site guidelines to post like this, and we have to ban accounts that to it repeatedly, so if you'd please review https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.
- mcs5280 2y ago[flagged]
- bhaney 2y ago> Surely none of the medical devices needed to treat a heart attack are Windows PCs connected to the internet? Wouldn't that be nice
- jmcgough 2y agoWithout access to Epic we can't place med orders, look up patient records, discharge patients from the hospital, enter them into our system, really much of anything. Every provider in the emergency department is on their computer placing orders and doing work when not interacting with a patient. Like most hospitals in this country, our entire workflow depends on Epic. We couldn't even run blood tests because the lab was down too. The STEMI was stabilized, it's more that it was scary to lose every machine in the department at once while intubating a crashing patient. You're flying blind in a lot of ways.
- tamimio 2y agoI just skimmed through the news. A lot of airports, hospitals, and even governments are down! It's ironic how people are putting their eggs in one basket, trying to avoid downtime caused by malware by relying on a company that put their system down. A lot of lessons will be learned after this for sure.
- phatfish 2y agoUnless you run half your devices on one security vendor and half on another surely there is no way round it? Companies install this stuff over "Windows Defender" so they can point fingers at the security vendor when they get hacked, this is the other side of the coin. It has happened before where security software has unwanted effects, can't say i remember anyone else managing to blue screen Windows and require a safe mode boot to fix the endpoints though.
- whyoh 2y agoRelying on easy-install "security vendors" is the problem. It's one thing to run an antivirus on a general purpose PC that doesn't have a qualified human admin. But many of the computers affected here are single-purpose devices, which should operate with a different approach to security.
- pageandrew 2y agoI don’t know Windows systems. I’ve read it’s causing Blue Screen of Death. I take that to mean that systems can’t even boot. Right? Can this be fixed over the air?
- techbrovanguard 2y agoi've seen photos of the bsod from an affected machine, the error code is `PAGE_FAULT_IN_NONPAGED_AREA`. here's some helpful takeaways from this incident: 1) mistakes in kernel-level drivers can and will crash the entire os 2) do not write kernel-level drivers 3) do not write kernel-level drivers 4) do not write kernel-level drivers 5) if you really need a kernel-level driver, do not write it in a memory unsafe language
- dboreham 2y ago0) don't load a new driver into your working kernel.
- YawningAngel 2y agoBasically all AV either runs as root or uses a kernel driver. I guess the former is preferable
- happyweasel 2y ago5) Well how much of those kernel-level drivers we rely upon ARE written in a memory unsafe language ??? Like 99% ? And we are not crashing and dying every day? Sure, Rust is the way to go. it just took Rust 18 years to mature to that level. Also, quite frankly, if your unwrap() makes your program terminate because an array out of bounds isn't that exactly the same thing ? (program terminates) But IMHO if we are hopping along a minefield at this moment every second of every day, well... If this is the worst case scenario, yeah it's not that worse after all.
- techbrovanguard 2y ago> Well how much of those kernel-level drivers we rely upon ARE written in a memory unsafe language ??? Like 99% ? And we are not crashing and dying every day? we shouldn't discount the consequences of memory safety vulnerabilities just because flights haven't physically been grounded. > Also, quite frankly, if your unwrap() makes your program terminate because an array out of bounds isn't that exactly the same thing ? (program terminates) this is a strawman, if you were writing a kernel-level driver in rust you'd configure the linter to deny code which can cause panics. here's a subset: - https://rust-lang.github.io/rust-clippy/master/index.html#/unwrap_used https://rust-lang.github.io/rust-clippy/master/index.html#/u... - https://rust-lang.github.io/rust-clippy/master/index.html#indexing_slicing https://rust-lang.github.io/rust-clippy/master/index.html#in...
- radiator 2y agoSo Crowdstrike protects your computers from cyber attacks. But who is going to protect you from Crowdstrike?
- choeger 2y agoIt's eye-opening how bad our crucial IT infra is nowadays. Running in-kernel third-party tools (AV) on critical infrastructure on Windows? Central banks? Control towers? Seriously? We should fire everyone involved and start IT from scratch. This level of negligence cannot be fixed.
- attentive 2y agoI am most annoyed by built-in RCE. Who thought that to be a good idea?
- ryandv 2y agoAbsolutely shameful display of how the cure can be worse than the disease. It's nonsense snake oil and security theater such as this that throws the cyber"security" industry into disrepute. One may as well have just installed McAfee Anti Virus.
- stefan_ 2y agoThis has been the story of the antivirus "industry" all along. They simultaneously seem to employ actual bona-fide security researchers while also making sure none of their software products are ever touched by people you could even refer to as "developers". I can't even imagine the noise at Microsoft from all the crash reports solely caused by antivirus software written by utter clowns injecting into other programs and, as here, into the kernel. Previous: https://infosec.exchange/@wdormann/112530285189478825 https://infosec.exchange/@wdormann/112530285189478825 Previous: https://thehackernews.com/2022/05/chinese-hackers-caught-exploiting.html https://thehackernews.com/2022/05/chinese-hackers-caught-exp... Previous: https://www.ftc.gov/news-events/news/press-releases/2024/02/ftc-order-will-ban-avast-selling-browsing-data-advertising-purposes-require-it-pay-165-million-over https://www.ftc.gov/news-events/news/press-releases/2024/02/... Previous: https://www.fortiguard.com/psirt/FG-IR-24-015 https://www.fortiguard.com/psirt/FG-IR-24-015 ...
- ryandv 2y ago"We're only backdooring your machines for your own good! We are the good guy experts, we promise."
- aa-jv 2y ago>snake oil So much software - especially in the 'operating system' sphere of things - really is just snake oil. Its just, very functional oil, in many cases - and highly toxic and slippery in many, many other cases. >cyber "security" industry Yes, I agree this is a market of smoke and mirrors, lies and propaganda. The reason is, operating systems are broken. Pretty much all of them. Only, some of them work well enough to get a lot of work done, most of the time. Of the 99.9995% of the time it works, its great. But, here's a thing I feel needs broader attention and discussion - It is my firm opinion that "Operating Systems Vendors" are a very poor, ragged class of professionals these days. The decisions made at Microsoft - and other OS vendor corporations - have really lost the plot. I can prove this by asking the golden question among the general public, and categorically get a standard response: "does this feature benefit the user, or does it benefit an advertiser?" "No, this all seems to be some sort of setup. Windows doesn't feel like its for us, any more." I mean, how many 3rd-party vendors do I need, secretly installing crippling 'updates' in my production systems, before I realize that there is no security, so write better software that doesn't need all this utter junk. I mean this sincerely, operating systems vendors are treasonous to the user if 3rd parties are of more relevance to production runtime, than the thing the user very definitely needs to be operating. The cloud is for backups, encrypted. It is made of snake oil. Always run your own machines.
- cromka 2y agoThis is a sample of what Y2K would look like if not for the countermeasures.
- mrinfinitiesx 2y agoActually really scary to see/read the comments on. Like Die Hard's 'Fire Sail"
- qeternity 2y ago*sale
- deleted 2y ago[deleted]
- Eji1700 2y agoWelp this fucked my night. A toast to the rest of you who are waaaaaay more screwed than me
- gloosx 2y agoThanks god all the critical infrastructure in my country is still on MS DOS!
- LeoPanthera 2y agoThat school running their HVAC infra on an Amiga must be pretty happy.
- jaza 2y agoHahaha, you mean all the CRITIC~1.INF ?
- hulitu 2y ago> Hahaha, you mean all the CRITIC~1.INF ? Kids those days. It shall be CRITIC~1.COM
- woodylondon 2y agoCyberStrike offers a temporary solution for crashed systems Cyberstike has given users a potential way to fix their systems. Boot Windows into Safe Mode or the Windows Recovery Environment (you can do that by holding down the F8 key before the Windows logo flashes on screen) Navigate to the C:WindowsSystem32driversCrowdstrike directory Locate the file matching “C-00000291.sys” file, right click and rename it to “C-00000291.renamed” Boot the host normally.
- oldmanyells68 2y agoGreat. now I'll do this 10,000 times.
- 0x1ceb00da 2y agoPractice makes a man perfect.
- rosege 2y agofun doing that on thousands of machines
- steve1977 2y agoIt's easy, the users can do it themselves. Just send them an e-mail with the inst... oh wait...
- junto 2y agoOne of the downsides of WFH. How do you contact your IT support via Microsoft Teams when you don’t have your laptop or Microsoft Teams.
- steve1977 2y agoYup. Well in our case (and we are thankfully not affected), they could call IT support. But then again, if IT support themselves cannot boot their PCs...
- silamay 2y agoI’m trying to refresh to get the latest update …
- berkaydumaner 2y agoHi guys, what is the KB code of this update?
- nickdothutton 2y agoThe only surprising thing is that this doesn't happen every month. Nobody understands their runtime environment. Most IT org's long ago "surrendered" control and understanding of it, and now even the "management" of it (I use the term loosely) is outsourced.
- ssss11 2y ago“Cloud” is so great huh
- nickdothutton 2y ago"I'm going to install an agent from Company X, on this machine, which it is essential that they update regularly, and which has the potential to both increase your attack surface and prevent not just normal booting but also successful operation of the OS kernel too". I am not going to provide you with a site specific test suite, you're going to just have to trust me that it wont interrupt your particular machine".
- yard2010 2y ago"And ofc, you pay me a shitload of money for this, I don't have to tell you why, am I?"
- swozey 2y agoThis is mostly physical machines in person, kiosks and pos terminals, office desktops and things like that. Windows is a tiny portion of GCP and AWS and the web in general. I'm 100% "cloud" with tens of thousands of linux containers running and haven't been affected at all.
- patates 2y agoNowadays it seems like everyone is running stuff inside VMs because IT removes all the rights in the host system.
- jkells 2y agoMaybe they do perform canary deployments and Australia was the canary? Certainly feels like it's disproportionately affecting us down under.
- bobnamob 2y agoWe’re just the only ones awake to feel it
- bobnamob 2y agoAnecdotal evidence: the global mega corp I unfortunately work for is definitely feeling this globally
- tomashubelbauer 2y agoIt looks like the whole world is the canary, but they will have the release ready and in top shape for the Mars deployment.
- rochak 2y agoI haven’t seen a simultaneous outage as big as this in my entire life. I’m just hoping this gets enterprises to move off of Windows.
- BartjeD 2y agoExcept it hasn't got much to do with Windows... its a faulty kernel software package from a commercial vendor unrelated to the OS. Philosophically it's always good to have diversity, precisely to avoid such disruptions. But the real issue here is: A) Apparently half the world runs CloudStrike... So everything is disrupted. B) Apparently CloudStrike didn't test their update properly. I'm very curious what will happen to CloudStrike. This seems like a huge liability?
- lachlanj 2y agoI'm confused, is this an issue with Windows or with Crowdstrike software installed on Windows?
- pi-rat 2y agoWith a crowdstrike kernel driver, so technically not a microsoft/windows issue.
- dboreham 2y agoFor a while I've joked with family and colleagues that software is so shitty on a widespread basis these days that it won't be long before something breaks so badly that the planet stops working. Looks like it happened.
- drooopy 2y agoI've picked the perfect day to return from vacation. Being greeted by thousands of users being mad at you and people asking for your head on a plate makes me reconsider my career choice. Here's to 12 hours of task force meetings...
- jeffrallen 2y ago"Gentlemen! You can't fight in here, this is the War Room!" Have a nice day, anyway.
- urbandw311er 2y agoHuge sympathies to you. If it's any consolation, because the scale of the outage is SO massive and widely reported, it will quickly become apparent that this was beyond your control, and those demanding your 'head on a plate' are likely to appear rather foolish. Hang in there my friend.
- drooopy 2y agoTo their credit, the stakeholder that asked for my head personally came to me and apologised once they realised that entire airports have been shut down worldwide. But yeah, not a Friday/funday hahaha
- defrost 2y agoMost organisations seem to have a section of management that dissolve into batshit crazy teapots at the first hint of a crisis.
- rightbyte 2y agoYe and these types make any problem worse. Any technical problem also becomes a social problem to deal with these lunatics and keep the house of cards from crumbeling.
- CoastalCoder 2y agoI don't it's a management thing, per se. I see it more as a personality trait for how one handles stressful situations. I think some people can improve upon it with time and effort.
- adzm 2y agoWorkaround fixed it for me, thankfully I had access to the bitlocker recovery keys. This will be a bad day for IT people worldwide.
- cromka 2y agoBBC reports: “ The cause is not known - but Microsoft says it's taking mitigation action”. Most of the media I found say it’s because “cloud infrastructure”. I am yet to see any major source actually factually report this is caused by a bad patch in Crowdstrike software installed on top of Windows. Gets to show how little competency there is in journalism nowadays. And begs a question how often they misinterpret and misreport things in other fields?
- spuz 2y agoOn BBC news a few minutes ago, an expert did describe the problem as affecting Microsoft Azure cloud systems as well as Windows systems running Crowdstrike due to an "update gone wrong".
- cromka 2y agoWell, in BBC’s live coverage, just minutes ago, their technology editor said: “ There have been reports suggesting that a cybersecurity company called Crowdstrike, which produces antivirus software, issued a software update that has gone horribly wrong and is bricking Windows devices - prompting the so-called "blue screen of death" on PCs. Now, whether these two issues are the same thing, or whether it's a perfect storm of two big things happening simultaneously - I don't yet know. It certainly sounds like it's going to be causing a lot of havoc.” What two issues? Two major independent outages? This is seriously bad and purely speculative.
- gedw99 2y agoIronically SolarWinds court case happened yesterday. SEC won. SolarWindows was fraudulent to say their software way “secure”. They should rename a side channel attack a “Tom and Jerry”, because its getting like a game of Cat and Mouse
- gedw99 2y agoRock me Amadeus. At least the central flight booking system is up I guess. Google brought it years ago and it's a mainframe. Hence why google flights is so tapped in :)
- deleted 2y ago[deleted]
- reegnz 2y agoCybersecurity company secures computers worldwide by not allowing them to be turned on. - not the onion
- mro_name 2y agoWhat do card houses do for a living?
- vlugovsky 2y agoCrowdstrike is a perfect name for a company that could cause a worldwide outage.
- lizknope 2y agoYeah, I've always thought it was a bad name. I see them during Formula 1 advertisements because they sponsor the Mercedes team. They might as well have named themselves "cluster bomb" as they have done a huge amount of damage today and for the next few days.
- Lammy 2y agoMakes me think of flystrike which is also a perfect analogy https://en.wikipedia.org/wiki/Myiasis https://en.wikipedia.org/wiki/Myiasis
- jiehong 2y agoIMO, having a mix of servers would help in mitigating issues like that. Like run stuff on Linux, windows and freebsd servers, so that you have OS redundancy should an issue affect one in particular (kernel or app). Just like you want more than a single server handling your traffic, you’d want 2 different base for those servers to avoid impacting them both with an update.
- romanovcode 2y agoNot using this crap security software would mitigate this issue.
- tanelpoder 2y agoI'm curious why this post is still not the 1st (but 2nd after an ebook reader announcement), despite all the upvotes.
- saagarjha 2y agoLots of people commenting drags down the position on the front page.
- bgnn 2y agoThis is a manifestation of almost everything wrong about software development and marketing practices. I work in hardware development and such a failure is almost impossible to imagine. It has to work, always. It puzzles me why this isn't the casebfor software. My SWE colleagues often get mad at us HW guys because we want to see their test coverage for the firmware/drivers etc.. The focus is having something which compiles and pushing the code to production as fast as possible and then regressing in production. Most of HW problems are a result of this. I found it's often better to go over the firmware myself and read line by line to understand what the code does. It saves so much time from endless debugging sessions later. It pisses of firmware guys, but hey, you have to break some eggs to make an omelette.
- aa-jv 2y ago> It puzzles me why this isn't the case for software In my anecdotal experience, its because corporate software projects are not typically run by people who are good at building safe things - but rather, just building things quickly. There's a huge issue with the mentality of "it works, ship it" being propagated. I build systems software for safety-critical and mission-critical markets, and I can say without a doubt that if there aren't at least two quality stages in your process workflow (and your workflow isn't waterfall), then you're going to be in for a rough time, rookies. Always, always delay your releases, and always, always, eat your own dog food by testing your delayed releases in your own customer-like environment, which is to say, never release a developers build ..
- bgnn 2y agoThis is also my experience. And the worst was always being lectured by SW project managers about being agile and having to move quick and release early. I won't release anything without making sure everything works in every possible condition. This is why it takes years to build a complex chip (CPU, Fpga, any SoC really). Their firmware is often squeezed into months, and often the developers are handling like 10 different projects. So, no focus, no time to understand the details of the design. At the end it's common to have firmware issues in the first year after release. It's kind of expected even.
- 2y ago
- silamay 2y agoI’m trying to refresh to get latest update… let’s keep posting
- mehh 2y agoDon’t be fooled, it’s Skynet, head to the bunkers!
- steve1977 2y agoUsername "mehh" has been noted.
- openrisk 2y agoThe details (the particular companies / systems etc) of this global incident don't really matter. When the entire society and economy are being digitized AND that digitisation is controlled and passes through a handful of choke points its an invitation to major disaster. It is risk management 101, never put all your digital eggs in one (or even a few) baskets. The love affair with oligopoly, cornered markets and power concentration (which creates abnormal returns for a select few) is priming the rest of us for major disasters. As a rule of thumb there should be at least ten alternatives in any diversified set of critical infrastructure service providers, all of them instantly replaceable / forced to provide interoperability... Some truths will hit you in the face again and again until you acknowledge the nature of reality.
- deleted 2y ago[deleted]
- pcblues 2y agoCentralisation vs decentralisation. Cost-savings vs localisation of disaster. It's a swinging pendulum of decisions. And developers know that software/hardware provision is a house of cards. The more levels of dependency, the more fragile the system is. Absolutely horrible when lives will be lost, but determining the way our global systems are engineered and paid for will always be a moving target based on policy and incentive. My heart goes out to life and death results of this. There are no perfect tech solutions.
- asib 2y ago> It is risk management 101, never put all your digital eggs in one (or even a few) baskets. The fact it's widespread is because so many individual organisations individually chose to use CrowdStrike, not because they all got together and decided to crown CrowdStrike as king, surely? I agree with you in principle, but the only solution I can think of would be to split up a company with reach like CrowdStrike's. The consequences of having to do that are up for debate.
- alex-nt 2y agoIt's never that simple. There is a strong herd mentality in the business space. Just yesterday I've been in a presentation from the risk department and they described the motives around choosing a specific security product as `safe choice, because a lot of other companies use it in our space, so regulator can't complain`...the whole decision structure boiled down to: `I don't want to do extra work to check the other options, we go with whatever the herd chooses`. Its terrifying to hear this...
- ssss11 2y agoI believe that today they struck the entire crowd… (or should that be cloud)
- chucke1992 2y agoI guess Microsoft can now offer some similar to a Crowstrike solution for Azure users.
- teeheelol 2y agoThrowaway account... CrowdStrike in this context is a NT kernel loadable module (a .sys file) which does syscall level interception and logs then to a separate process on the machine. It can also STOP syscalls from working if they are trying to connect out to other nodes and accessing files they shouldn't be (using some drunk ass heuristics). What happened here was they pushed a new kernel driver out to every client without authorization to fix an issue with slowness and latency that was in the previous Falcon sensor product. They have a staging system which is supposed to give clients control over this but they pissed over everyone's staging and rules and just pushed this to production. This has taken us out and we have 30 people currently doing recovery and DR. Most of our nodes are boot looping with blue screens which in the cloud is not something you can just hit F8 and remove the driver. We have to literally take each node down, attach the disk to a working node, delete the .sys file and bring it up. Either that or bring up a new node entirely from a snapshot. This is fine but EC2 is rammed with people doing this now so it's taking forever. Storage latency is through the roof. I fought for months to keep this shit out of production because of this reason. I am now busy but vindicated. Edit: to all the people moaning about windows, we've had no problems with Windows. This is not a windows issue. This is a third party security vendor shitting in the kernel.
- jakebleiberg 2y agoHello: I'm a reporter with Bloomberg News covering cybersecurity. I'm trying to learn more about this Crowdstrike update potentially bypassing staging rules and would love to hear about your experience. Would you be open to a coversation? I'm reachable by email at jbleiberg2@bloomberg.net or on Signal at JakeBleiberg.24. Here's my Bloomberg author page: https://www.bloomberg.com/authors/AWuCZUVX-Pc/jake-bleiberg https://www.bloomberg.com/authors/AWuCZUVX-Pc/jake-bleiberg. Thank you. Jake
- tamimio 2y ago> give clients control over this but they pissed over everyone's staging and rules and just pushed this to production. Interesting.. > We have to literally take each node down, attach the disk to a working node.. Probably the easiest solution for you is to go back in time to a previous scheduled snapshot, if you have that setup already.
- anal_reactor 2y ago[flagged]
- wasteduniverse 2y ago[dead]
- jpl56 2y agoObligatory XKCD : dependency [0] [0] https://xkcd.com/2347/ https://xkcd.com/2347/
- esskay 2y agoSitting in our work slack feeling pretty smug that I forced the migration to only Linux servers and Linux or macOS work computers now.
- altdataseller 2y agoI dont understand? Linux servers dont need endpoint protection? I venture the vast majority of servers with crowdstrike are linux
- kozziollek 2y agoI'm assuming Linux implementation is different and the bug is not present there.
- CaliforniaKarl 2y agoYes. CrowdStrike for Linux has had bugs in the past, though. This time the Windows version is affected.
- drpossum 2y agoThe bug only affects Windows machines
- kasabali 2y ago> endpoint protection That's a polite way to say malware
- Medox 2y agoUntil the VPN is affected by the Domain Controller that is on Windows.
- esskay 2y ago100% nix based here so thankfully zero systems affected. Everything from routers to devices, we have a total blanket ban on any Windows based software.
- pulkitsh1234 2y agoWorkaround steps: https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/comment/ldvwkbn https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/commen...
- ndsipa_pomu 2y ago> CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes. > Workaround Steps: > Boot Windows into Safe Mode or the Windows Recovery Environment > Navigate to the C:\Windows\System32\drivers\CrowdStrike directory > Locate the file matching “C-00000291*.sys”, and delete it. > Boot the host normally.
- tjoff 2y agoWas thinking about a bootable usb-stick that would do that automagically. But I guess it is harder to boot from a usb-stick in these environments than the actual fix. I guess more feasible and even neater to do it if you have network boot or similar.
- entuno 2y agoThere's potentially a huge issue here for people using BitLocker with on-prem AD, because they'll need the BitLocker recovery keys for each endpoint to go in an fix it. And if all those recovery keys are stored in AD (as they usually are), and the Domain Controllers all had Crowdstrike on them...
- mschuster91 2y agoAssuming that they also have a regular Bitlocker password, there's hope with a bit manual effort. https://news.ycombinator.com/item?id=41003893 https://news.ycombinator.com/item?id=41003893
- entuno 2y agoMost of the large deployments I've seen don't use pre-boot PINs, because of the difficulty of managing them with users - they just use TPM and occasionally network unlock. So might save a few people, but I suspect not many.
- aaron695 2y ago[dead]
- dvdkon 2y ago[flagged]
- deleted 2y ago[deleted]
- jpl56 2y agoWas watching TV this morning in France (TF1, 8:00 CET), the weather forecast map system was out. The journalist just gave us the information as if he was on the radio, telling he was sorry for the system to be failing.
- netsharc 2y agoThey sponsor the Mercedes F1 team https://crowdstrikeracing.com/f1/about-partnership/ https://crowdstrikeracing.com/f1/about-partnership/ , who have a race this weekend and practice sessions today. It'd be funny if their cars can't go on track because their computers are down...
- netsharc 2y agoThey did go down! https://www.reddit.com/r/formula1/comments/1e71dtn/mercedes_cloudstrike_pitwall_bsod/ https://www.reddit.com/r/formula1/comments/1e71dtn/mercedes_... But someone probably fixed it, and the cars were able to go out on track for the first practice session.
- deleted 2y ago[deleted]
- shubhamjain 2y agoNaive question, if it’s a blue screen of death with a boot loop, how are they going to restore things? Don’t tell me the answer is going to every system manually.
- chgs 2y agoGo to every system manually, boot to safe mode, rename the sys files, run a fix. Easier to just rebuild from the image. For every windows machine your company has. lol.
- CaliforniaKarl 2y agoLots of systems (not all) are able to reboot, and have CrowdStrike download the fix before the bad code is able to crash things. But otherwise, yes, you have to go to systems manually.
- ogurechny 2y agoWell, it seems that Windows is not yet accessible remotely when it crashes. If system administrator had too much free time, and configured every system to probe network on booting, and there is no encryption, it is possible to boot from a minimal Linux image with a script that automatically renames the driver and restarts. The corporate version of the same approach uses Intel AMT (or however else it is called), but it is only available on licensed hardware from big suppliers. Otherwise, you can distribute flash drives with the same auto-executing fix to everyone who is able to enter firmware setup, and boot from USB. If it's not available for security reasons, more manual work is required. But what happens next? If Crowdstrike handled all the security measures, and there was no additional firewall rules, address checks, and so on, your network is now as open as it can be. I suppose certain groups have been celebrating, and uploading gigabytes of data from networks whose detection systems became severed.
- misswaterfairy 2y agoGoing to every system manually, then delete a file via command line in Windows' recovery environment.
- muixoozie 2y ago
- mro_name 2y agoheavy clouds this morning. Maybe time to reconsider how solid a ground clouds are.
- ch_123 2y agoWould this issue not affect bare metal as well?
- uitgewis 2y agohttps://archive.is/AWVuv https://archive.is/AWVuv
- r2vcap 2y agoThis is why I don't use Windows and refuse any SWE jobs that require Windows machines. Additionally, I believe kernel-level game anti-cheat software should be banned.
- deleted 2y ago[deleted]
- clydethefrog 2y ago10 hours ago someone posted a critical post about CrowdStrike on the "wallstreetbets" subreddit. https://old.reddit.com/r/wallstreetbets/comments/1e6ms9z/crowdstrike_is_not_worth_83_billion_dollars/ https://old.reddit.com/r/wallstreetbets/comments/1e6ms9z/cro...
- deleted 2y ago[deleted]
- elorant 2y agoHow can an antivirus update affect Azure's servers?
- dboreham 2y agoTBF although I worried about this possibility the first time the IT dude wandered into my office in 1989 holding a floppy he said he wanted to put into all the PCs we had (we had no PCs), it has actually taken a very long time for the shit to hit the fan.
- nullify88 2y agoThe sheer coverage of this outage across multiple businesses and industries, the impact must be greater than some of the malicious cyber attacks from ransomware, worms etc.
- fudged71 2y agoThis title doesn’t nearly describe the breadth and severity of the problem…
- solidninja 2y agoAh the "move fast and break things" philosophy gets a demonstration.
- topspin 2y agoThat's misplaced. Windows is an ancient platform. CrowdStrike is ubiquitous and routinely updated. There was no "move fast" here, at least on the part of the people operating these systems.
- endstart 2y agoThe workaround suggests removing a file with .sys extension. What does the file do normally? If removed, what happens to the state of security on that system?
- saagarjha 2y ago.sys files on Windows are typically drivers and driver-related files
- codeulike 2y agoMicrosoft are going to be pissed that this is widely being discussed as a Microsoft outage. Do AV vendors like Crowdstrike need a license or something from Microsoft to push these kernal driver based things? Or is it just like anyone can make one?
- 1oooqooq 2y agoyes, and they have two. it is a windows problem.
- scopeh 2y agoHow many people still believe the "cloud" was worth it? Maybe we should go back to the days of buying software and running it ourselves with our own infrastructure. I know, I'm dreaming.
- ogurechny 2y agoThey all should have used some expensive corporate-and-government-level product that promises protection against exactly that kind of large scale attack on infrastructure.
- fx1994 2y agoThat piece of "AV software" slowed down my brand new corporate i7 Lenovo to shit so I switched to M2 Pro. Best decision ever.
- sensanaty 2y agoUntil corporate decides to install new MDM software on the usually-blazingly-fast apple silicon chips :( I can't even open files larger than 500 lines without my whole system slowing to a crawl because of the insanely aggressive and slow "antivirus" bloatware the MDM forces on me.
- qxmat 2y agoTheir Windows sensor has made development almost unworkable. Not sure why but I haven't noticed the OSX sensor slow things down appreciably. I suspect my Windows profile is configured to be more aggressive?
- WatchDog 2y agoThis was apparently caused by a faulty "channel file"[0], which is presumably some kind of configuration database that the software uses to identify malware. So there wasn't any new kernel driver deployed, the existing kernel driver just doesn't fail gracefully. [0]: https://x.com/brody_n77/status/1814185935476863321 https://x.com/brody_n77/status/1814185935476863321
- vesinisa 2y agoWhy on earth don't they have staged rollouts for updates?
- tehlike 2y agoEverytime i look into such catastrophic issues, it always boils down to lack of robust canarying mechanisms. They have enough client base that they can even run an A/B test on the whole binary level, but no.
- malkia 2y agoAlso, why not have some sort of graceful degradation (well kind of), like: OS Boots, loads CS driver, the driver loads some new feature/config, and before/after new recent thing ("runtime flag") marked whether it successfully worked, and if not on the next reboot that thing gets either disabled, or the previous known good config (obviously some combination of things might cause another issue), but instead of blindly rebooting to the same state....
- ta345346633 2y agoI think pfsense does this (from memory, been a while using it). Basically dual-partitions, and if it failed to come up on the active partition after an update it'd revert. Granted you need to have the space to have two partitions, but for a small partition/image not so bad. What surprises me is if its a content update, and the code fell over when dealing with it - just basically bad release engineering isn't it not to cater for that in the first place? i.e. some tests in the pipeline before releasing the content update would've picked it up given it sounds like 100% failure rates.
- roschdal 2y agoUSA can no longer be trusted to supply import technology for the world.
- deleted 2y ago[deleted]
- pjmlp 2y agoYet another good example why liability in software should already be a common thing.
- patates 2y agoThis company has post-apocalyptic style photos to make you panic-buy their solution. https://ibb.co/Bc6n527 https://ibb.co/Bc6n527 "62 minutes could bring your business down" I guess they could bring all the businesses down much quicker. edit: link https://www.crowdstrike.com/en-us/#teaser-79minutes-adversary-1 https://www.crowdstrike.com/en-us/#teaser-79minutes-adversar...
- lamp_book 2y agoBoy, that is some corny branding.
- bob1029 2y agoI agree but I've also personally witnessed how effective this crap is on a certain cohort of IT managers. You can see the 3 or 4 gears grinding together in their head... something like "oh my goodness look at all the things I get for one purchase order!".
- lamp_book 2y agoI could certainly see that! Haha.
- deleted 2y ago[deleted]
- hulitu 2y ago> "62 minutes could bring your business down" > I guess they could bring all the businesses down much quicker. It is because the buyer does not get the message. And, when they get it, it is too late.
- sofixa 2y agoAnd for much longer.
- joenot443 2y agoTheir "Statement" is remarkably aloof for having brought down flights, hospitals, and 911 services. "The issue has been identified, isolated and a fix has been deployed." Maybe I'm misunderstanding what I read elsewhere, but is the machine not BSODing upon boot, prior to a Windows Update service being able to run? The "fix" I see on reddit is roughly: Workaround Steps: 1. Boot Windows into Safe Mode or the Windows Recovery Environment 2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory 3. Locate the file matching “C-00000291*.sys”, and delete it. I'm horrified at the thought of tens of thousands of novice Windows users digging through System32 to delete driver files; can someone set my mind at ease and assure me this will eventually be fixed in an automated fashion? https://www.crowdstrike.com/blog/statement-on-windows-sensor-update/ https://www.crowdstrike.com/blog/statement-on-windows-sensor...
- Jyaif 2y agoMy employers pays Crowdstrike to double my build times. Quite astounding really.
- guenthert 2y agoafaiu google (and I presume other operators of large number of computers) deploy updates to their software first to a small set of nodes and only if after a given time the update has been deemed successful, continue to update an increasingly larger set til complete. Isn't this done as well with automatic updates of end user software or embedded systems and if not, why not?
- ivxvm 2y agoThings like hospitals, airlines, 911, should have multiple systems with different software stacks and independent backends running in-parallel, so that when one infra goes down they can switch to another.
- judgementalfish 2y agoThere is an enormous cost associated with the kind of redundancy you're talking about. Capitalism prevents us from being set up in the way you're describing. Why invest in company A if company B can run the same business with half the operational expenses? Shareholder profit above all.
- SkyBelow 2y agoIs company B allowed to take the full brunt of all the problems when there is a failure, or does government protect it by limiting damages? If company B's cheaper choice leads to harm and lets people and estates sue company B into the ground, then company A is a safer investment even if it has lower returns. If government interaction limits such recovery options, then that is what leads to company B's higher returns not also having higher risks, so they'll be the better investment. But that is a result of government intervention, not the economic system in play.
- tflinton 2y agoFor some areas of our critical systems we have three independent software groups program the same exact system on different infrastructure. Just for moments like these...
- lucasRW 2y agoGood day for OSINTers, APTs, redteamers, to find out who uses Crowdstrike on their endpoints.
- gnuser 2y agodumb techbro c-suites: what, why would you have an issue with a proprietary closed source app that frequently self updates and sends tons of data to a third party while essentially being a backdoor? We said we wanted security and this has Security(tm) all over the literature! Look we even have dashboards for the gui-ninjas like the security team!
- attentive 2y agoIt's backdoor as a feature. It even has a cute name - "CrowdStrike Real Time Response".
- AndyMcConachie 2y agoFinally the crowd has struck! ba-dum ching!
- commercialnix 2y agoAll my customers endpoints are on Linux based endpoints. Because our users' Windows apps run in vdi with disposable instances based off snapshots and highly restrictive networking on the Linux endpoints, none of our users are effected. Running Windows on bare-metal was always obviously very stupid. The consequences of such stupidity are just being felt now.
- personalityson 2y agoThis is what AI's first strike will look like
- dbergamin 2y ago[flagged]
- deleted 2y ago[deleted]
- dmarble 2y agoPlot twist: The * in C-00000291*.sys is "-block-ultron" Premature deployment of Crowdstrike AGI disaster response plan.
- KingOfCoders 2y agoAll down had no backup plan.
- deleted 2y ago[deleted]
- pharos92 2y ago1. This is why kernel modules are a bad idea 2. This is why centralism is a bad idea 3. This is why sacrificing stability for security is a bad idea 4. Security still needs to factor in security of supply - not just data safety
- imdsm 2y agoCentralisation in a nutshell. Monopolies so big that they become globally fragile. CloudFlare outages break a lot of the internet, and now we can see, Windows-based updates bricking machines across the world. We've all pushed bad updates but how was this not tested?
- 2-3-7-43-1807 2y agoi've never heard of crowdstrike ever but it (co-)runs half of the essential IT infrastructure, worldwide? (also, great choice of name i must say)
- BoredPositron 2y agoI only know them because their CEO is a relatively good amateur racing driver lol.
- 2-3-7-43-1807 2y agoseems like he's also a relatively good amateur kernel driver developer - or at least his team is and he's the former CTO of mcafee?
- _8j50 2y agoTheir stock price will suffer but they can waive license fees for a year or so for every endpoint affected (~$50). They better pin this on a rogue employee, but even then, force pushing updates shouldn't be in their capability at all! They must guarantee removal of that capability. Lawsuits should be interesting. They offer(ed?) $1 mil breach insurance to their customers, so if they were to pay only that much per customer this might be compensation north of $10B. But to be honest, wouldn't surprise me if they can pay up without going bankrupt. The sad situation is, as twitter people were pointing out, IT teams will use this to push back against more agents for a long time to come. But in reality, these agents are very important. Crowdstrike Falcon alone is probably the single biggest security improvement any company can make and there is hardly any competition. This could have been any security vendor, the impact is so widespread because of how widely used they are, but there is a reason why they are so widely used to begin with. Oh and just fyi, the mitigation won't leave you unprotected, when you boot normal, the userspace exe's will replace it with a fixed version.
- InsideOutSanta 2y ago> the single biggest security improvement Clearly not, unless you don't count a world-wide economic and societal disruption of unprecedented scale a security incident. > This could have been any security vendor ...that apparently deploys Kernel Extensions to millions of Windows devices at once, without any staggering. > there is a reason why they are so widely used to begin with Because companies need to check a box, and purchasing CrowdStrike checks that box.
- akho 2y ago> Crowdstrike Falcon “Cybersecurity’s AI-native platform for the XDR era”. I hope there’s a blockchain somewhere in it.
- _kb 2y agoAssuming this event itself isn't malicious, what an excellent POC for something that is. I sure hope every org out there with this level of market reach has good security in place. It's certainly going to be getting some probing after this.
- nyx_land 2y agoworded badly whatever
- dark-star 2y agoThat's what you get for letting a company install a root kit on your servers and desktops ;-) I mean, don't they do canary updates on CrowdStrike too? Every Windows admin has done this for the last 5+ years, test Windows updates on a small number of systems to see if they are stable. Why not do the same for 3rd party software?
- kingkongjaffa 2y agoIt's kind of surprising so much infra was using windows servers or windows cloud VMs for these things. I assumed these systems would all be Linux VMS in Azure/AWS/GCP at this point. on https://azure.status.microsoft/en-gb/status https://azure.status.microsoft/en-gb/status the message is currently: > We have been made aware of an issue impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon agent, which may encounter a bug check (BSOD) and get stuck in a restarting state.
- junto 2y agoWelcome to the enterprise. Where “lift and shift” was sold to corporate CTO’s as better than maintaining their own IT infrastructure.
- patates 2y agoThis company has post-apocalyptic style photos to make you panic-buy their solution. https://ibb.co/Bc6n527 https://ibb.co/Bc6n527 "62 minutes could bring your business down" I guess they could bring all the businesses down much quicker. https://www.crowdstrike.com/en-us/#teaser-79minutes-adversary-1 https://www.crowdstrike.com/en-us/#teaser-79minutes-adversar... (Repeating my comment because other story is duped)
- Thorentis 2y agoPerversely, this may make many companies no longer invest in this type of cyber security software. Which may lead to a whole host of other problems...
- mnau 2y agoYeah, these events will be fun once new product liability directive (that includes sw) comes into force.
- deleted 2y ago[deleted]
- nynyny7 2y agoCrowdstrike marketing slogan on their website: "A radical new approach proven to stop breaches". I'll give them that: Putting all Windows computers within a company into an endless BSOD loop is a very radical approach to stop breaches. :)
- CoastalCoder 2y ago"We breach your systems to hackers can't!"
- tm-guimaraes 2y agoHow does such a huge company do “full deploys” like this? At this number of endpoints, only a few % should have been updated (and faced the problems) before a full rolout This is not a small startup with some SaaS, these guys are in most computers of too many huge companies. Not rolling out the updates to everyone at the same time seems just too obvious
- ndsipa_pomu 2y agoThis incident definitely makes a good case for staggered deploys of patches.
- dorkwood 2y agoIs this something that could be solved by building AI code review directly into git clients? I can't help thinking Claude 3 would have caught this.
- machinekob 2y agoYes AI already solved it, my god why people are so high on LLM's solving everything
- redrove 2y agoLMAO Yeah let's throw an LLM at the C++ kernel driver and auto-push to prod
- tslocum 2y agoPoe's law in action.
- tehlike 2y agoRolling out updates in an A/B test slowly is the only way to reduce the occurrence of such issues _significantly_. There's no other way, literally, nothing.
- alibarber 2y agoI have been told 'not to worry' because it isn't a cyber attack. Yet the outcomes we are seeing feel a lot like the doomsday predictions of what a cyberattack would do. It is almost as if we are experiencing the cybersecurity/warfare equivalent of 'friendly fire'.
- elorant 2y agoFor years now antivirus solutions have ridiculous amount of control over the OS. I accidentally installed an adware antivirus the other day that was bundled-up with a third party software, and I had to boot to Linux to manage to completely remove the damn thing from Windows. The uninstall option left a process running that couldn’t be forcefully killed. Microsoft needs to take control and forbid anyone and anything from running software with that kind of behavior.
- deleted 2y ago[deleted]
- simonjgreen 2y agoCrowdStrike should have learned the lesson from the more seasoned players in the industry to slow roll their updates and observe.
- echoangle 2y agoWhy would they roll out this update globally to all users immediately? Isn’t it normal to do gradual rollouts? Or did this update contain some critical security fix they wanted everyone to have as fast as possible?
- madisp 2y agoif I'm reading this correctly the short interest for the stock doubled over June? :) https://www.nasdaq.com/market-activity/stocks/crwd/short-interest https://www.nasdaq.com/market-activity/stocks/crwd/short-int...
- nicholasbraker 2y agoThis article seems more relevant than ever and was posted a few days ago: https://ea.rna.nl/2024/07/12/no-it-really-no-i-t/ https://ea.rna.nl/2024/07/12/no-it-really-no-i-t/
- pxc 2y agoVendors of tools like this drive the cybersecurity industry discourse, so 'defense in depth' often practically sorta means 'add more software that does more things'. But maybe this kind of thing can actually impart the lesson that loading your OS up with always-on, internet-connected agents that include kernel components in order to instrument every little thing any program does on the system is, uh, kinda risky. But maybe not. I wonder if we'll just see companies flock to alternative vendors of the exact same type of product.
- zmmmmm 2y agoSo CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer starting up and then in the critical path of every network connection the computer makes. The risk of ZScaler being a central point of failure is not considered. But - the risk of failing the compliance checkbox it satisfies is paramount. All over the place I'm seeing checkbox compliance being prioritised above actual real risks from how the compliance is implemented. Orgs are doing this because they are more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting. So we need to hold regulatory bodies accountable as well - when they frame regulation such that organisations are cornered into this they get to be part of the culpability here too.
- wparad 2y agoThat for sure is a great point, it's actually something Dorota brought up in her talk this year https://vimeo.com/940487124 https://vimeo.com/940487124
- SuperSandro2000 2y agoSounds familiar
- RedShift1 2y agoDon't put your eggs in one basket, I use multiple anti-virus products so that if one blows up at least not all computers are affected. Looks like my old wisdom is still new wisdom. Clarification: I mean that every computer has one anti-virus product, but not every computer has the same anti-virus product. I'm not installing multiple anti-virus products on the same computer.
- cqqxo4zV46cp 2y agoThis is much easier applied personally than it is to a 30k person organisation. No need to be condescending.
- YoboDev 2y agoMy team and I have begun to refer to this issue as CrowdStroke
- naizarak 2y agothis is really microsoft's fault for handing out kernel access to random 3rd parties, none of which are doing anything special that microsoft couldn't implement themselves (AV, anti-cheat, security)
- immibis 2y agoYes, Microsoft should just grant itself a monopoly on all those markets "for your own safety" and see what happens with their lawyers.
- commandersaki 2y agoOr do what Apple does, disallow kernel extensions, and provide rigid kernel faciltiies for VPN clients, EDR agents, etc. to use, so they don't have to implement custom code resident in the kernel.
- immibis 2y agoApple can disallow kernel extensions because it fully controls the entire hardware and software stack. Everything that would need to be an extension is already in the kernel and Apple knows all of those things.
- dschuetz 2y agoI wonder who exactly messed up the update, microsoft or crowdstrike. Usually, there is pre-rollout update testing AND some companies use N-1 version staging for critical/production systems. For me it feels much more complex a failure than just "it's crowdstrike's fault". Everybody involved must have done something wrong.
- dschuetz 2y agoThat was quick: Microsoft is blaming "3rd party" and announced that "a fix is forthcoming". Very curious indeed. https://techcrunch.com/2024/07/19/banks-airlines-brokerage-houses-report-widespread-outages-across-the-globe/ https://techcrunch.com/2024/07/19/banks-airlines-brokerage-h...
- runningmike 2y agoGuess we will never read the real facts. Truth is RMS was right. Again. Closed source security software is too often malware by design. We need open solutions we can truly trust.
- wcski 2y ago> Closed source security software is too often malware by design. Can you be more specific? Genuinely curious what you mean here.
- lizknope 2y agoCrowdstrike is closed source security software. What's the difference between malware and what Crowdstrike has done to the world today? We might as well reclassify Crowdstrike as malware and remove it from all computers to avoid this situation in the future.
- wcski 2y agoThe difference is that the intent of malware is to disrupt. Is gasoline useless just because it explodes when you light a match next to it? edit to add: OSS is not inherently more secure than closed source.
- lizknope 2y agoGasoline is very useful. We also take a lot of precautions when using it. We also have things like inspections and financial penalties if you were storing it in an unsafe manner. It's clear we need to take more precautions before using Crowdstrike. More testing, ability by IT departments to not push updates, ability to rollback updates.
- AndrewDucker 2y agoThis is, of course, why they should be doing phased rollouts. 1% of their customers, then 10%, then all the rest.
- 0x1ceb00da 2y agoOr just test your code?
- AndrewDucker 2y agoBoth. No testing is going to catch everything. And if it only hits 1 computer in 100 then your local testing will probably miss it
- Dentrax 2y agoI'm just curious, don't they have something like "gradual rollout" to update their app? They just bulk-update simultaneously across entire agents? No way. Something is a bit off for me. But there are good lessons to learn for sure.
- deleted 2y ago[deleted]
- Jamie9912 2y agoMy company stays multiple versions behind latest for this exact reason, but we were still affected
- bravetraveler 2y agoI'm a bit unfamiliar with this stuff anymore... supposedly it was a content update, not the agent itself :/
- bobo_legos 2y agoSurely though these content updates must go through some kind of regression testing right? Right?
- bravetraveler 2y agoParty 'try not to cry', me and you
- modestygrime 2y agoI read that they pushed a new configuration file, so possibly they don't consider that a "software update" and pushed it to everyone. Which is obviously insane. If I am publishing software, it doesn't matter if I've changed a .py file or a .yaml file. A change is a change and it's going to be tagged with a new version.
- Capricorn2481 2y ago
- felix_kirkine 2y agoJust gonna leave this here: https://news.ycombinator.com/item?id=32548671 https://news.ycombinator.com/item?id=32548671
- grahar64 2y agoI wonder what the rollout procedure is for CrowdStrike. I put $100 down that this was a minor update they decided was so minimal it didn't need extensive testing. So many places use the "emergency break glass rollout procedure" on every deploy because it doesn't require all the hassle
- jaza 2y agoConsidering what CrowdStrike's software does, I'd say the majority of their updates could be quite easily argued as being "emergency" updates, so yeah, quite possibly they've gotten into the habit of "omg URGENT must break glass" way too often.
- vinay_ys 2y agoIf you are IT team for a large impactful organization, you have to control updates to your organization's fleet. You cannot let vendors push updates directly. You have to stage those updates and test them and then do a gradual rollout to your whole organization. Plus, for your critical communication systems, you must have a disaster recovery plan that actually helps you recover quickly in minutes, not hours or days. And you have to exercise this plan regularly. If you are crowd strike, shame on you for not testing your product better. You failed to meet a very low bar. You just shipped a 100% reproducible widely impactful bug. Your customers must leave you for a more diligent vendor. And I really hope the leadership teams in every software engineering organization learn a valuable lesson from this – listen to that lone senior engineer in your leadership team who pushes for better craft and operational rigor in your engineering culture; take it seriously - it has real business impact.
- latexr 2y agoThere’s already a Wikipedia page on the outage. https://en.wikipedia.org/wiki/July_2024_global_cyber_outages https://en.wikipedia.org/wiki/July_2024_global_cyber_outages
- omoikane 2y agoIt now redirects to a new page that names CrowdStrike specifically. https://en.wikipedia.org/wiki/2024_CrowdStrike_incident https://en.wikipedia.org/wiki/2024_CrowdStrike_incident
- agilob 2y agoDo we have any estimates how many machines are affected?
- bidikburger 2y agohow can billion dollar company push update before testing?
- ghoshbishakh 2y agoNo worries for us. https://pinggy.io/ https://pinggy.io/ is working like a charm :)
- StrLght 2y agoFrom the guidelines: > Don't solicit upvotes, comments, or submissions. Users should vote and comment when they run across something they personally find interesting—not for promotion.
- crypt1d 2y agoThis never would have happened if all these orgs used a blockchain. /sarcasm /but is it really?
- hellajack3d 2y agoI guess this article might need some updating soon: https://www.crowdstrike.com/resources/reports/total-economic-impact-of-crowdstrike-falcon-complete/ https://www.crowdstrike.com/resources/reports/total-economic...
- jeffrallen 2y ago> ...delivers -407% ROI... FTFY.
- deleted 2y ago[deleted]
- JonChesterfield 2y ago"Falcon Complete managed detection and response (MDR) delivers 403% ROI, zero breaches and zero hidden costs"
- clwg 2y agoI'm always curious on how security software can provide a ROI. I had McAfee tell me one time that the hackersafe logo on our website would increase sales by 10%, this was at a Fortune 50 doing billions in sales online every year. I was pretty hyped because it would have done wonders for my career, but then they walked it back and wouldn't explain it to me. I wasn't mad, I was disappointed.
- adrr 2y agoI ran an AB test on 2012 not sure its relevant now, we tested the McAfee logo and conversion was boosted by 2%. Bigger boost was a lock icon, 3%. It kept increasing the more locks we added an topped at 5% after 5 lock icons.
- uticus 2y agoThe intersection of ROI and human psychology! 1 lock: “looks safe, I buy” 2 locks: “wow really safe, I buy more” 50 locks: “I’m being lied to”
- w4rh4wk5 2y agoI love the name! Really tells you what's going on ^^
- nvarsj 2y agoWas involved in a "security mandated" mandatory rollout of Crowdstrike at my prior company. This software was utter shit, and broke stuff all over the place. And installs itself as basically malware into critical paths everywhere. We objected to ever using it as a SPOF, but was overruled. So yeah, not remotely surprised this happened. Any kind of middleware/dynamic agent is highly suspect in my experience and to be avoided.
- mrinfinitiesx 2y agoHalf of the world's computers are down. The biggest tech failure of our time. Airports. Banks. NYSE. 298 of the fortune 500 companies. RIP.
- mulmen 2y ago> The biggest tech failure of our time. Is it? Is this even the first time this year that all the airlines are down? How many times a day is a hospital a victim of ransomware?
- mads_quist 2y agoI'm actually very fond of "fail fast" and "no blame" culture, but someone needs to get fired for this!
- bsodfriday 2y agohttps://isitbsodfriday.com/ https://isitbsodfriday.com/
- mrkramer 2y agoWindows breaking computers since 1985.
- RadixDLT 2y agodoes Russia has something to do with this?
- birracerveza 2y agoFeels like what people imagined the millennium bug would have been like, just short of PCs catching on fire.
- xorcist 2y agoWe often read about how organizations are so bad because they don't spend enough on security. That slope is particularly slippy. Crowdstrike is very expensive.
- jaza 2y agoToday's incident shows that the real problem is actually that organisations spend too much (money, but too little time / manpower) on security. Hey, third-party vendor, I'll give you all the money you want, I'll let you pwn all my systems, I'll be your little bitch, just make me secure, I don't have time for all that security shit, kthxbye.
- piva00 2y agoIt's bizarre reading all the headlines about companies offline, flights canceled, banks not working because of a piece of antivirus software in 2024. Mostly because I lived through Y2K and every fear about Y2K just materialised but because of Crowdstrike instead. I can't imagine the amount of wasted work this will create, not only the lost of operations across many industries but recovery will be absolute hell with Bitlocker. How many corporate users have access to their encryption keys? And when stored centrally, how many of the servers have Crowdstrike running and just got stuck in a boot loop now? I don't envy the next days/weeks for Windows IT admins of the world...
- simonjgreen 2y agoCrowdStrike have finally posted publicly on it: https://www.crowdstrike.com/blog/statement-on-windows-sensor-update/ https://www.crowdstrike.com/blog/statement-on-windows-sensor...
- alibarber 2y agoBrilliant press release: - Not apologetic - An 'issue' in a 'single' release I bet they were originally planning on starting with a '[reported by a] small number of customers' too.
- user3939382 2y ago“We’ve encountered a small single issue in which.. cough we’ve crashed all the computers in the world…”
- alibarber 2y ago"A specific and limited subset of computers on a single globe"
- Daedren 2y agoTheir legal team would definitely not allow nothing else.
- DebtDeflation 2y agoThis is pretty wild. I woke up to a news alert on my phone stating a "global IT outage" took down banks, airlines (who were calling for a global ground stop for all flights), hospitals, emergency services, etc. Expected it to be some sort of Tier 1 Network issue. Nope, a failed update for some third party Windows security app.
- sidmkp96 2y agoThe thing that amazes me is how they've rolled out such a buggy change at such a scale. I would assume that for such critical systems, there would be a gradual rollout policy, so that not everything goes down at once.
- mavamaarten 2y agoThis. I can see such an update shipping out for a few users. I mean I've shipped app updates that failed spectacularly in production due to a silly oversight (specifically: broken on a specific Android version), but those were all caught before shipping the app out to literally everybody around the world at the same time.
- CubsFan1060 2y agoThe only thing I can think of is they were trying to defend from a very severe threat very quickly. But... it seems like if they tested this on one machine they'd have found it.
- escapecharacter 2y agoThat’s the most charitable hypothesis, and I agree could be possible! I myself have ninja-shipped a fix a minor problem, but then caused a worse problem since I rushed it.
- pvillano 2y agoI pushed a one-character fix and broke it a second time
- trusche 2y ago"works on my machine" at Internet scale. What a scary thought
- altdataseller 2y agoUnless that threat was a 0day bug that allows anyone to SSH to any machine with any public key, it was not worth pushing it out in haste. Full stop. No excuses.
- roca 2y agoWhat I'm curious about: other than checkbox compliance, how does Crowdstrike convince companies to buy their product? Do they present evidence that their product is effective at protecting customers? Because certainly Crowdstrike customers still get hacked.
- bdd8f1df777b 2y agoI would imagine that their best weapon is that so many other big organizations are using CS, so choosing CS gives the decision maker the best shield from responsibilities, similar to "nobody gets fire by choosing IBM". Of course, how they started from small was completely different.
- bob1029 2y agoI've watched it occur countless times. Often times the people making the purchase decision are largely incompetent. They usually come out and take your team to a nice lunch. Then they run you through a fancy slide deck and convince you to let them run some scaremongering reporting tool over your infra. By the end of the day, most of your leadership is convinced they need the solution. Rinse and repeat hundreds of times and you have the 3rd party vendor hodgepodge hellscape that constitutes most large corporations' IT infrastructure.
- FullMetalBitch 2y agoSo what are going to be the consequences of this? In my country some healthcare institutions and emergency systems are working.
- artk42 2y agoI guess all the blamed EuroCommission will again have to do their job to bring anti-oligo/monopoly regulations, which everyone will hate but still slightly work. Architecting technical systems is MUCH WAY easier than architecting social-economical systems. I hope one day all those tech-savvy web3 wannabe revolutionaries will start to do the real job a designing socially working systems, not only technically barely working cryptographically strong hamster-tapping scams
- DebtDeflation 2y agoAt one point overnight airlines were calling for an "international ground stop for all flights globally". Planes in the air were unable to get clearance to land or divert. I don't believe such a thing has ever happened before except in the immediate aftermath of 9/11.
- acomjean 2y agoFlights have been delayed/canceled. Not as critical as the OP hospital, but not good. Effecting airlines/brodcasters and banks. https://www.nytimes.com/live/2024/07/19/business/global-tech-outage?unlocked_article_code=1.8U0.snej.23Wwd3Ej68kq&smid=url-share https://www.nytimes.com/live/2024/07/19/business/global-tech...
- DebtDeflation 2y agoIf a plane is in the air, and can't get permission to land anywhere, well, they only have a finite amount of fuel onboard.
- spectrumero 2y agoIn the case of an emergency (which low fuel most definitely is) the captain has the ultimate authority and can tell ATC "I'm landing anyway".
- langsoul-com 2y agoThat might be fine for a single or a few planes. But given the magnitude of the outage, what if a single airport had dozens of planes landing anyway. There's a very good reason airport traffic control exists.
- silisili 2y agoI'd be highly surprised if ATC systems were affected by this, but if anyone wants to correct me please do.
- Renaud 2y agoWhen you see the size if the impact across the world, the number of people who will die because hospital, emergency and logistics systems are down… You don’t need conventional war any more. State actors can just focus on targeting widely deployed “security systems” that will bring down whole economies and bring as much death and financial damage as a missile, while denying any involvement…
- netsharc 2y agoSeeing all the cancelled and delayed flights, it makes me think a hacking kind of climate activism/radicalism would be more useful than gluing hands to roads, or throwing paint on art.
- mrguyorama 2y agoActivism is mostly about awareness, because generally you believe your position to be the one a logical person will accept if they learn about it, so doing things that get in the news but only gets you a small fine or month in jail are preferred. Taking destructive action is usually called "ecoterrorism" and isn't really done much anymore.
- treesknees 2y ago-
- muzani 2y agoI'm not so sure it's hyperbole: https://news.ycombinator.com/item?id=41002977 https://news.ycombinator.com/item?id=41002977
- btbuildem 2y agoWe are far past that point. So many critical systems are running on autopilot, with people who built and understood them retiring, and a new batch of unaware, aloof, apathetic people at the helm. There's no real need for some Bad Actor -- at some point, entropy will take care of it. Some trivial thing somewhere will fail, and create a cascade of failures that will be cataclysmic in its consequences. It's not fear-mongering, it's kind of a logical conclusion to decades of outsourcing, chasing profit above and over anything else, and sheer ignorance borne of privilege. We forgot what it took to build the foundations that keep us alive.
- crooked-v 2y agoI was going to buy some put options against CRWD with spare pocket money, but it turns out that the service I have my investment money is in broken right now. I wonder if that's because of Crowdstrike.
- deleted 2y ago[deleted]
- alkhimey 2y agoAn update to the internal database. It still did not sunk to developers that data has equivalent risk as code. A400 crashed because of an XML file update. I have witnessed my share of critical bugs caused by "innocent? updates to "data" which were treated less seriously because of this. Management and devs alike should change their conception about this.
- heraldgeezer 2y ago[flagged]
- Numerlor 2y agoIt's ridiculous how many people here are blaming Windows for something that's really completely unrelated
- avidphantasm 2y agoHow long before companies start consciously de-risking by replacing general-purpose systems like Windows with newer systems with smaller attack surfaces? Why does an airline need to use Windows at all for operations? From what I’ve seen, their backend systems are still running on mainframes. The terminals are accessed on PCs running Windows, but those could trivially be replaced with iPadOS devices that are more locked down than Windows and generally more secure by design.
- ricardobayes 2y agoOften, the cost of switching is too high or too complex to justify. On top of that, many applications commonly run in manufacturing etc., simply does not run on any other OS.
- fransje26 2y agoIt's true that a multi-billion dollars screw-up, with possible deaths, is a cost that is much easier to justify..
- sabas123 2y agoAnd you think that any realistic alternative (which does require appropriate funding) does not have similar risks?
- fransje26 2y agoThe billions that have been lost, and the lives that have been lost, have, in the blink of an eye, rendered the "too costly to implement" argument moot. For bean-counting purposes, it's just really convenient that the burden of that cost was transferred onto somebody else, so that the claim can continue to be made that another solution would still be too costly to implement. Accepting the status-quo that got us here in the first place, under the pseudo-rational argument that there are not realistic alternatives, is simply putting ones head in the sand and careening, full steam ahead, to the next wall waiting for us. That there might not be an alternative available currently does not mean that a new alternative cannot be actively pursued, and that it is not time for extreem introspection.
- jeffchien 2y agoPhenomenal timing: https://old.reddit.com/r/wallstreetbets/comments/1e6ms9z/crowdstrike_is_not_worth_83_billion_dollars/ https://old.reddit.com/r/wallstreetbets/comments/1e6ms9z/cro...
- krembo 2y agoCoincidence?
- bryancoxwell 2y agoProbably.
- andrelaszlo 2y agoI'm curious about investing and economy, and I always wonder about P/E ratios like Crowdstrike's (currently 450-something, was over 500 last week). Some P/E ratios for today, for some companies I find interesting: - Shopify: 615.12 - Crowdstrike: 455.70 - Datadog: 341.98 - Palantir: 212.34 - Pinterest: 187.67 - Uber: 99.0 - Broadcom: 77.68 - Tesla: 58.33 - Autodesk: 52.36 - Adobe: 49.23 - Microsoft: 37.97 What's going on here? Do investors expect Shopify, for example, to increase their earnings by an order of magnitude despite already having done extraordinarily well in a very competitive market? Can anyone ELI5?
- blackbear_ 2y agoThe greater fool theory: https://en.m.wikipedia.org/wiki/Greater_fool_theory https://en.m.wikipedia.org/wiki/Greater_fool_theory Essentially, investors buy as long as they think they will be able to sell at a higher price in the future, regardless of economic fundamentals.
- chii 2y ago> regardless of economic fundamentals. not regardless, but only if. The future is unknown, so their bet is also based on that unknown. Is it foolish? Who knows. Did nvidia seem foolish if somebody made that bet before their ai boom?
- 999900000999 2y agoI'd bet my career CS isn't spending enough on QA. It's always the first thing to be cut, no one cares about QA when everything is going well, but when things go wrong...
- bkj512 2y agoLol we were using Symantec software so thankfully no affect.
- DuckHunt 2y agoSo apparently "The issue has been identified, isolated and a fix has been deployed" https://x.com/George_Kurtz/status/1814235001745027317 https://x.com/George_Kurtz/status/1814235001745027317 Yet the chaos seems to continue. Could it be that this fix can't be rolled out automatically to affected machines because they crash during boot - before the Crowdstrike Updater runs?
- CubsFan1060 2y agoCorrect. Many just end up in an endless loop and never actually boot. It's about as bad as it gets.
- jwitthuhn 2y agoYeah, you need to manually fix each affected system by booting in safe mode. Not possible to do remotely.
- rudasn 2y agoOuch!
- martyvis 2y agoAnd you will need your bitlocker recovery key to access your encrypted drive in safe mode. I luckily had mine available offline There's going be a lot of handholding to get end users through this.
- fifteen1506 2y agoYou can enable safemode for next boot without the recovery key and then you can delete the offending file on that next boot.
- kiririn 2y agoThat requires being able to boot in the first place
- alphabetting 2y agoGoogle spending a boatload for Wiz looks smarter now
- bkj512 2y agoWow
- bandrami 2y agoWhat a fun time to be less than 48 hours out from a transcontinental flight
- monkeydust 2y agoSo where can I buy an ETF of companies specializing in software Quality Assurance?
- higeorge13 2y agoNobody does that anymore. :)
- butler14 2y agoI’m guessing it’s completely incidental that the CEO of crowdstrike was critical of China earlier this year, and that China is somehow unaffected by this ‘global’ issue!
- scrollaway 2y agoI mean… yeah? almost everyone in the west except for some highly corrupt people is critical of china, and china doesn’t deploy crowdstrike.
- cja 2y agoSorry to be dense, but what is CrowdStrike and do I have it on my computer?
- vtemian 2y agoWhat's the actual magnitude of this outage? Is there a way to estimate how many machines were down?
- zteppenwolf 2y agoI guess people who continue to use Windows in 2024 arguably deserve this, particularly those utilizing it in a production environment.
- cedws 2y agoIt's truly horrifying how many critical systems run Windows.
- dopylitty 2y agoIt was Windows in this case but nothing is stopping it from happening with any other widely used system that gets online updates. CrowdStrike has root on Linux/MacOS as well after all. The problem is relying on networked computers for critical infrastructure with no contingency plan. This sort of thing will happen whether because of a bug or because of ransomware. The software and hardware industries are incapable of producing reliable and safe products in our economic system. Important services such as hospitals, groceries, water treatment plants, and electric grids should be able to operate in offline mode when this sort of thing inevitably happens.
- x86x87 2y agowhat about all the people that use services provided by people that use windows? should there be some sort of pushback here?
- deleted 2y ago[deleted]
- eitland 2y agoSome Canonical guy I think many years ago mentioned this as their sales strategy a few year ago after a particularly nasty Windows outage: We don't ask customers to switch all systems from Windows to Ubuntu, but to consider moving maybe a third to Ubuntu so they won't sit completely helpless next time Windows fail spectacularly. While I see more and more Ubuntu systems, and recently have even spotted Landscape in the wild I don't think they were as successful as they hoped with that strategy. That said, maybe there is a silver lining on todays clouds both WRT Ubuntu and Linux in general, and also WRT IT departments stopping to reconsider some security best practices.
- 51Cards 2y agoExcept further up this thread another poster mentions that CrowdStrike took down their Debian servers back in April as well. As soon as you're injecting third party software into your critical path with self-triggered updates you're vulnerable to the quality (or lack of) that software despite platform. Honestly your comment highlights one of the few defenses... don't sit all on one platform.
- noveltyaccount 2y agoHopefully they won't botch the update for two operating systems at the same time. But yeah. Hope.
- potatolicious 2y agoYeah, I see a lot of noise on social media blaming this on Microsoft/Windows... but AFAIK if you install a bad kernel driver into any major OS the result would be the same. The specific of this CrowdStrike kernel driver (which AFAIK is intended to intercept and log/deny syscalls depending on threat assessment?) means that this is badnewsbears no matter which platform you're on. Like sure, if an OS is vulnerable to kernel panics from code in userland, that's on the OS vendor, but this level of danger is intrinsic to kernel drivers!
- gjsman-1000 2y ago
- scrollaway 2y agoThose focusing on QA and staged rollouts are misguided. Yes of course a serious company should do it but CrowdStrike is a compliance checkbox ticker. They exist solely to tick the box. That’s it. Nobody who pushes for them gives a shit about security or anything that isn’t “our clients / regulators are asking for this box to be ticked”. The box is the problem. Especially when it’s affecting safety critical and national security systems. The box should not be tickable by such awful, high risk software. The fact that it is reflects poorly on the cybersecurity industry (no news to those on this forum of course, but news to the rest of the world). I hope the company gets buried into the ground because of it. It’s time regulators take a long hard look at the dangers of these pretend turnkey solutions to compliance and we seriously evaluate whether they follow through on the intent of the specs. (Spoiler: they don’t)
- lopkeny12ko 2y agoIsn't a Windows BSOD the equivalent of a kernel panic? I don't understand how this is CrowdStrike's fault. Vanilla userspace operations shouldn't cause a kernel panic--that's a bug in the OS, not a bug in some user software. If anything, we should be blaming Windows here?
- jhfdbkofdchk 2y agoCrowdStrike is not a vanilla userspace program, it hooks deeply into the operating system.
- EvanAnderson 2y ago> Vanilla userspace operations shouldn't cause a kernel panic... The component Crowdstrike says you need to remove to restore functionality is a ".sys" file. That's a kernel-mode driver. The fault is happening on the kernel side.
- kidbomb 2y agoLessons learned from this: - CS: Have a staging (production-like) environment for proper validation. It looks like CS has one of these bu they have just skipped it - IT Admins: Have controlled roll-outs, instead of doing everything in a single swoop. - CS: Fuzz test your configuration Anything I have missed?
- totaldude87 2y agoI hope the narrative of , install crowd strike and pass the audit or else changes after this . but being in the industry for so long , I don't expect any changes whatsoever, it's either CS or some other tool
- fnord77 2y agoI take it patching remote machines is going to be difficult or impossible? I haven't used windows in years, but from what I read you need to be in safe mode to delete a crowdstrike file in a system directory, but you need some 48 char key to get into safe mode now if it is locked down?
- mattbrewsbytes 2y agoDoes crowdstrike work similarly on MacOS? I have to imagine the "walled garden" doesn't allow for 3rd parties to insert themselves into the OS kernel but I could be wrong.
- checker659 2y agoI mean, you can always disable SIP and install your own extensions.
- frankohn 2y agoThe Windows ecosystem typically deployed in corporate PCs or workstations is often insecure, slow, and poorly implemented, resulting in ongoing issues visible to everyone. Examples include problems with malware, ransomware, and Windows botnets. In corporate environments, IT staff struggle to contain these issues using antivirus software, firewalls, and proxies. These security measures often slow down PCs significantly, even on recent multi-core systems that should be responsive. Microsoft is responsible for providing an operating system that is inherently insecure and vulnerable. They have prioritized user lock-in, dark patterns, and ease of use over security. Apple has done a much better job with macOS in terms of security and performance. The corporate world is now divided into two categories: 1. Software-savvy companies that run on Linux or BSD variants, occasionally providing macOS to their employees. These include companies like Google, Amazon, Netflix, and many others. 2. Companies that are not software-focused, as it's not their primary business. These organizations are left with Microsoft's offerings, paying for licenses and dealing with slow and insecure software. The main advantage of Microsoft's products is the Office suite: Excel, Word and Powerpoint but even Word is actually mediocre. EDIT: improve expression and fix errors:
- Darvon 2y agoIf you were ready to ditch corpomicrosoft why would you go to corpoapple instead of something foss like debian tho
- nextos 2y agoI'd say something implementing the ideas of NixOS, i.e. immutable versioned systems and declarative system definitions, is poised to replace the current deployment mess, which is extremely fragile. With NixOS, you can upgrade without fear, as you can always roll back to a previous version of your system. Regular Linux distributions, macOS, and Windows make me very nervous because that is not the case.
- throwaway48476 2y agoIdeally there would be a usable security first os based on something like sel4 with a declarative package system for slow to change mission critical appliances.
- gchamonlive 2y agoI know I have the benefit of hindsight in this regard, but how isn't there redundant checks and tests that would prevent a mishap of this magnitude? I mean, there should be extensive automated testing using many different platforms and hardware combinations as a prerequisite for any rollout. I guess this is what we get when everything is opaque, not only the product and the code, but also the processes involved in maintaining and evolving the solution. They would think twice about not investing heavily in testing their deployment pipelines if everyone could inspect their processes. It might also be the case that they indeed have a thorough production and testing process deployed to support the maintenance of crowdstrike solutions, but we are only left to wonder and to trust whatever their PR will eventually throw at us, since they are a closed company.
- gz5 2y agoSeems CS themselves may have been hacked? For example, seems unlikely that both: 1. CS normally pushes global updates to entire user base simultaneously? 2. This made it through their testing. Not only 'just' QA but likely CS employees internally run a version or two ahead of their customer base? Just speculation - folks who know either answer can validate or debunk.
- mmiyer 2y ago(they confirmed there was no hack) I think you have too much faith in minimal software development practices being applied at companies.
- Kye 2y agoCrowdstrike seems like the kind of thing that's sold to CEOs at conferences, forced on IT against objections, and the subject of a lot of discussion at Defcon.
- red_admiral 2y agoI can't wait for rachelbythebay's comments on this.
- EvanAnderson 2y agoI wonder what Crowdstrike's opsec is like re: malicious actors gaining control of their automated update servers. This incident certainly highlights the power of that type of attack, even if this one just ends up being typical human incompetence-based.
- luismedel 2y agoProbably a stupid question but, how can the Windows kernel recover so well after a graphics driver crash and at the same time being unable to do the same for other kind of drivers.
- red_admiral 2y agoFrom reddit: > I'm in Australia. All our banks are down and all supermarkets as well so even if you have cash you can't buy anything. I hope the national security/defense people are looking at this closely. Because you can bet the bad guys are. What's the saying, civilisation is only ever three days away from collapse or something? I am pretty convinced this is a fuckup not an attack, but if Iran or someone managed something like this, there would be hell to pay.
- misswaterfairy 2y agoYou can bet a substantial amount of money on CloudStrike inadvertently painting a huuuuge hacker target on their back over this...
- jimberlage 2y agoFor $150/hour, I will spend today consulting for businesses who need someone in the St Louis area to go reboot a remote workers’ machine.
- lencastre 2y agoLots of issues in Spain and Germany.
- anchochilis 2y agoWe routinely implement phased / canary deployments in server-side systems to prevent faults from rolling out globally. How is it possible that CrowdStrike and/or Windows does not have a similar system built in for large, institutional customers? This is outrageous.
- Tylast 2y agoOh, the foresight of the 1st episode of Connections. https://www.youtube.com/watch?v=XetplHcM7aQ https://www.youtube.com/watch?v=XetplHcM7aQ
- pelasaco 2y agoIf i was North Korea, I would say that was me. That would be however a crazy story if Russia and China had done anything about it.
- whoknowsidont 2y agoSomewhere out there, there is an engineer with the biggest "I told you so" shit eating grin scrolling through every social media site and basking in the glory.
- gquere 2y agoThere's supposedly a fix being deployed (https://x.com/George_Kurtz/status/1814235001745027317 https://x.com/George_Kurtz/status/1814235001745027317). Since it's a channel update I'm assuming that it would be downloaded automatically? Has anyone received it yet? Does the garbage driver disappear or is it replaced? Edit: got in touch with an admin: C-00000291-00000000-00000029.sys SHA256 1A30..4B60 is the bad file (timestamp 0409 UTC) C-00000291-00000000-00000030.sys SHA256 E693..6FAE is the fix (timestamp >= 0527 UTC) Do not rely on the hashes too much as these might vary from org to org I've read.
- cryptica 2y agoI've been warning about the coming software apocalypse for years. This isn't a one-off, this is the beginning of a pattern. Tech recruitment is broken, software is more complex than ever, more and more people are turning to hacking, people are growing increasingly dissatisfied with the status quo...
- cryptica 2y agoThis is what happens when you entrust software security to ex-hackers. Hackers love complexity because that's the kind of environment they thrive in; yet when they start working for the other side as security consultants, they still love complexity. Complexity ought to be the security consultant's worst enemy. Ex-hackers often talk about security as if it's something you need to add to your systems... Security is achieved through good software development practices and it's about minimalism. You can't take intrinsically crappy, over-engineered, complex software and make it more secure by adding layers upon layer of complex security software on top.
- lobochrome 2y agoThis is how I would start a war… surreal. I hope it’s just a bug.
- Melatonic 2y agoGood luck everyone. I just spent all night fixing my shit and we caught it early
- lifeisstillgood 2y agoWhat do we do next week? So assuming everyone uses sneaker-net to restart what’s looking like millions of windows boxes, there comes recriminations but then … what? I think we need to look at minimum viable PC - certain things are protected more than others. Phones are a surprisingly good example - there is a core set of APIs and no fucker is ever allowed to do anything except through those. No matter how painful. At some point MSFT is going to enforce this the way Apple does. The EU court cases be damned. For most tasks for most things it’s hard to suggest that an OS and a webbrowser are not the maximum needed. We have been saying it for years - what I think we need is a manifesto for much smaller usable surface areas
- andrewia 2y agoIsn't that basically the point of WinRT and Windows 10 S Mode? The problem is getting developers to adopt the new more secure APIs.
- 999900000999 2y agoIn this case even dockerized environments would allow you to redeploy with ease. But that's too much work, many of these systems are running docker resistant software. Management doesn't want to invest in modernization - it works this quarter, it's someone else's problem next quarterly. You're basically proposing Windows 12 to radically limit what software and drivers can do. Even then eventually someone will probably still break it with weird code. I'm actually amazed these updates are being tested in prod. Do they have no QA environments ? Do I personally need to create a startup company called Paranoia... We actually run a clone of your prod environment minus any sensitive data, then we install all the weird and strange updates before they hit your production servers... As an upsell we'll test out privileges, to take sure your junior engineers can't break prod. Someone raise a seed round, I'm down to get started this week.
- quasse 2y ago> In this case even dockerized environments would allow you to redeploy with ease. Not if the CIO mandated that your bare-metal OS hosting Docker has to run a rootkit developed by bozos.
- harimau777 2y agoIt seems like this would indirectly tell us what systems use Cloudstrike. Could that in of itself be information that could help an attacker? I know the security team at work is adamant about not leaking details of our system.
- smithington 2y agoThere's already somebody trying to cash in on this problem: https://fix-crowdstrike-apocalypse.com https://fix-crowdstrike-apocalypse.com
- steveBK123 2y agoWild that a piece of software so integral to basic function has such bad release discipline. A/B, Blue/Green, Canary, Rolling, etc.. I've worked on 4 person software teams that at least followed basic user group rolling release system.
- guax 2y agoIntegral? I would argue that is wild that a piece of software so useless for the basic and correct function gets so much privilege.
- steveBK123 2y agoWell I mean integral here in that your PC can't boot and do anything useful if the software breaks. It's not some solitaire app or saas website. Completely nuked 1000s of companies ability to operate for a day, and the thing is auto-updating with apparently global big bang release push method..
- JackC 2y agoCrowdstrike did this to our production linux fleet back on April 19th, and I've been dying to rant about it. The short version was: we're a civic tech lab, so we have a bunch of different production websites made at different times on different infrastructure. We run Crowdstrike provided by our enterprise. Crowdstrike pushed an update on a Friday evening that was incompatible with up-to-date Debian stable. So we patched Debian as usual, everything was fine for a week, and then all of our servers across multiple websites and cloud hosts simultaneously hard crashed and refused to boot. When we connected one of the disks to a new machine and checked the logs, Crowdstrike looked like a culprit, so we manually deleted it, the machine booted, tried reinstalling it and the machine immediately crashes again. OK, let's file a support ticket and get an engineer on the line. Crowdstrike took a day to respond, and then asked for a bunch more proof (beyond the above) that it was their fault. They acknowledged the bug a day later, and weeks later had a root cause analysis that they didn't cover our scenario (Debian stable running version n-1, I think, which is a supported configuration) in their test matrix. In our own post mortem there was no real ability to prevent the same thing from happening again -- "we push software to your machines any time we want, whether or not it's urgent, without testing it" seems to be core to the model, particularly if you're a small IT part of a large enterprise. What they're selling to the enterprise is exactly that they'll do that.
- nicklad 2y agoThis seems to be misinformation? The CrowdStrike KB says this was due to a Linux kernel bug. --- Linux Sensor operating in user mode will be blocked from loading on specific 6.x kernel versions Published Date: Apr 11, 2024 Symptoms In order to not trigger a kernel bug, the Linux Sensor operating in user mode will be prevented from loading on specific 6.x kernel versions with 7.11 and later sensor versions. Applies To Linux sensor 7.11 in user mode will be prevented from loading: For Ubuntu/Debian kernel versions: 6.5 or 6.6 For all distributions except Ubuntu/Debian, kernel versions: 6.5 to 6.5.12 6.6 to 6.6.2 Linux sensor 7.13 in user mode will be prevented from loading: For Ubuntu 22.04 running the 6.5 kernel: 6.5.0 - 6.5.0-1014-aws 6.5.0- 6.5.0-1015-azure 6.5.0 - 6.5.0-1014-gcp 6.5.0 - 6.5.0-24-generic 6.5.0 - 6.5.0-1015-oem Ubuntu kernel version: 6.6 to 6.6.2 For Debian kernel version: 6.5 to 6.5.12 6.6 to 6.6.2 For all distributions except Ubuntu/Debian, kernel versions: 6.5 to 6.5.12 6.6 to 6.6.2 Linux Sensors running in kernel mode are not affected. Resolution CrowdStrike Engineering identified a bug in the Linux kernel BPF verifier, resulting in unexpected operation or instability of the Linux environment. In detail, as part of its tasks, the verifier backtracks BPF instructions from subprograms to each program loaded by a user-space application, like the sensor. In the bugged kernel versions, this mechanism could lead to an out-of-bounds array access in the verifier code, causing a kernel oops. This issue affects a specific range of Linux kernel versions, that CrowdStrike Engineering identified through detailed analysis of the kernel commits log. It is possible for this issue to affect other kernels if the distribution vendor chooses to utilize the problem commit. The commit where the kernel bug was introduced is seen at https://github.com/torvalds/linux/commit/fde2a3882bd07876c144f2e00f7ae6893c378180 https://github.com/torvalds/linux/commit/fde2a3882bd07876c14... and the commit that resolves the issue is seen at https://github.com/torvalds/linux/commit/4bb7ea946a370707315ab774432963ce47291946 https://github.com/torvalds/linux/commit/4bb7ea946a370707315... To avoid triggering a bug within the Linux kernel, the sensor is intentionally prevented from running in user mode for the specific distributions and kernel versions shown in the above section These kernel versions are intentionally blocked to avoid triggering a bug within the Linux kernel. It is not a bug with the Falcon sensor. Sensors running in kernel mode are not affected. No action required, the sensor will not load into user mode for affected kernel versions and will stay on kernel mode. For Ubuntu 22.04 the following 6.5 kernels will load in user mode with Falcon Linux Sensor 7.13 and higher: 6.5.0-1015-aws and later 6.5.0-1016-azure and later 6.5.0-1015-gcp and later 6.5.0-25-generic and later 6.5.0-1016-oem and later If for some reason the sensor needs to be switched back to kernel mode: Switch the Linux sensor backend to kernel mode sudo /opt/CrowdStrike/falconctl -s --backend=kernel
- tedajax 2y agoThe first time I experienced crowdstrike in a corporate environment it seemed obvious that something like this would eventually happen.
- ezoe 2y agoThose EDR software is implemented as a kernel driver. A third party closed source Windows kernel driver that can't be audited. It gathers massive amount of activities and send back to the central server(which can be sold) as well as execute arbitrary payload from the central server. It became single point of failure to your whole system. If an attacker gain control of the sysadmin PC, it's over. If an attacker gain administrator privilege on EDR-installed system, it run the same privilege with EDR so attacker can hide their activities from EDR. There aren't many EDR products in the world it can be done. I'd like to call it "full trust security model".
- ngneer 2y agoSecurity technology harming security? Shocker. We need less monoculture. Trouble is monoculture pays. Write the software once, deploy it everywhere - free money. I manage a simple Tier-4 cloud application on Azure, involving both Windows and Linux machines. Crowdstrike, OMI, McAfee and endpoint protection in general has been the biggest thorn in my side.
- snailb 2y agoOn a positive note, I'm in morocco and getting money from ATM wasn't working for the whole day I believe because of this outage. I was at the till in a supermarket and people started asking if they can chip in to pay for some food I bought because I didn't have the cash. Humanity 1 - Technology 0 Edit: Outage of all ATM's in Morocco was yesterday not today. so not sure how the two are related.
- hughw 2y agoSo, why did our little company's (little used) two Windows machines not BSOD overnight? They were just sitting idle. They run CS Falcon sensor. Did the update force a restart? Didn't seem to happen here.
- Kye 2y agoIt looks like a configuration file update is the culprit. The software presumably picks up the update, then BSODs.
- egberts1 2y agoI am quite sure that they have had three precious timezone hours to detect a total failure of telemetry after their fateful midnight upgrade. Like the most useful Canary Island in the Coal Mine.
- KingOfCoders 2y ago2024 years after 2k we have 2k.
- Kye 2y agoMaybe a silly question, but: why hasn't this affected Linux? I assume it uses a proprietary kernel module just like it does on Windows. I guess this will come out in a post-mortem if they publish one, but it's been on my mind. edit: aha https://news.ycombinator.com/item?id=41005936 https://news.ycombinator.com/item?id=41005936 They did do this to Linux, but in the past. Maybe whatever they did to deal with it saved Linux this time around
- ngneer 2y ago"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. It demands the same skill, devotion, insight, and even inspiration as the discovery of the simple physical laws which underlie the complex phenomena of nature." "The most important property of a program is whether it accomplishes the intention of its user." C.A.R. Hoare
- uticus 2y agoAgreed but have you been in the industry lately? Nobody hires assembly programmers anymore. Want money you must work at wobbly top of abstraction mountain.
- ngneer 2y agoI am well aware, but the quotes are timeless for a reason. Not to be cheeky, but "Want money" is exactly how you get to the many routinely broken endpoint solutions that wind up reducing reliability and at times increasing the attack surface. Wherever you are in the stack, please make it more robust and easier to reason about. No matter how far from the assembly.
- thih9 2y agoIt’s not just about the tech abstraction mountain, it’s about the app logic and dev process too. A react native JS app with a clear spec and a solid release process can be more reliable than bloated software that receives an untested hotfix, even if the latter was handwritten in assembly.
- sans_souse 2y agoWhy would you name your company "CrowdStrike" anyway? What does Crowd Strike even mean?
- josephd79 2y agoYear of Linux
- markus_zhang 2y agoIn pre-market, CRWD is 14% down. I think investors are a bit scared that THIS time there is going to be some consequences.
- rjmunro 2y agoI'm amazed it's just 14%, not more like 75%-80%. Surely a lot of customers are going to uninstall and move to competitors. The remainders are at least going to demand much cheaper service with better guarantees going forward.
- markus_zhang 2y agoYeah, and now recovered to -9.39%. Let's see what happens. I guess CrowdStrike is backed by enough powerful people to NOT lose too much business.
- ddgflorida 2y agoDo you suppose they test before pushing updates out?
- thomasjudge 2y agoWhy are "security" patches not tested before they are deployed?
- egberts1 2y agoYet Lennart Pottering and Redhat (spelled that way as I am one of the original pre-IPO investor of RedHat via Alex Brown/Deutsche Bank) wants to put networking of Linux into UEFI this quarter, inside the most sacrosanct PID 1. They still won’t learning anything from Crowdstrike’s mistakeS! Maybe it is time for me to ditch that stock.
- integricho 2y agoIronic that the software intended to prevent exactly these kinds of outages ends up causing it.
- integricho 2y agoThis should at the very least put them out of business by causing each and every client to abandon them as their security solution.
- sytelus 2y agoGenuine question: How the heck crapeware like CloudStrike got into all critical systems from 911 to hospitals to airlines? My understanding was that all these critical systems are just super lazy to upgrade or install anything at all. I would love to know all the sales tactics CS used to get into millions of systems for money!
- ncr100 2y agoReading other comments here, sorry I don't have the link, one crowd strike salesperson threatened to cancel them as a Client, yes you read that right, if the client wasn't easier to work with. So they're bullies or at least that one salesperson in crowd strike is a bully. Another article talked about crowd strike being required for compliance, people here talking about checkbox compliance. So there's a systemic requirement from perhaps insurers for there to be some kind of comprehensive near real-time updated antivirus solution. Furthermore, the haste makes waste philosophy seems to not be honored, in my opining mind, by the minds who drive The impacted sectors of our economy. Hospitals, Banks, airlines. This kind of vulnerability should not have been accepted. It's a single point of failure. Even on crowdstrike's website they have this kind of like radar ring hotspot Target kind of graphic, where they show at the very center one single client app .. theirs, as if that one single client is the thing that's going to save us?
- sytelus 2y agoThis is amazing sales tactics! So, you buddy up with insurance, they create a checkbox and recommend you for a revenue cut! Now you suddenly have millions of customers out of nowhere and your product gets installed on billions computers before you even know it. I have seen this tactic get used for many mediocre products. For example, 3rd party dishwasher soap recommended by dishwasher company. Amazingly powerful. I don’t think most of CrowdStrike employees even knew they were in more than billion computers with paid service. The CEO was just busy doing brutal marketing of this pointless product.
- convivialdingo 2y agoHere’s my take as a security software dev for 15 years. We put too much code in kernel simply because it’s considered more elite than other software. It’s just dumb. Also - if a driver is causing a crash MSFT should boot from the last known-good driver set so the install can be backed out later. Reboot loops are still the standard failure mode in driver development…
- erichocean 2y agoNot possible in this situation, the "driver" is fine, it's a file the driver loads during startup that is bad, causing the otherwise "good" driver to crash. Going back to an earlier version—since the driver is "good—would just re-load the same driver, loading the updated file, and then crashing again.
- convivialdingo 2y agoA driver that crashes with bad input is not “fine.” Bad design, bad configuration loading and crap input validation. Did they even fuzz the code? We’d spend 20x development time on kernel code because BSOD is never an option. I get that this was a bad release - but IMHO it’s incredible that they pushed this out to a billion devices before the red flags went up.
- amai 2y agoIt seems monocultures are not only bad for resilience in agriculture, but also in IT.
- jacobgorm 2y agoThe great clownstrike.
- whoisstan 2y agoCan someone with experience explain how integration tests did not detect that?
- amai 2y agoDid Crowdstrike forget the rule, that one does not simply deploy on Friday? https://www.reddit.com/r/ProgrammerHumor/comments/f79iag/dont_deploy_on_fridays/ https://www.reddit.com/r/ProgrammerHumor/comments/f79iag/don...
- dev1ycan 2y agoCrazy isn't it, I had no issues because my group policy updates have been off since last year, guess the "everyone must forcefully update" for "security reasons" ended up backfiring, who could've thought
- daemonologist 2y agoMy company has some bios bitlocker extension installed which prompts for a password on boot, so automatic updates (one of which tried to install last night) just get stuck there in jet engine mode. Normally this is extremely annoying but today I count myself lucky - aside from a couple of people with Chromebook thin clients I am the only person showing as online in Teams right now.
- kaladin-jasnah 2y agoAnecdote: my first job was IT at a small org. We had somehow gotten a 15 minute remote meeting with Kevin Mitnick, and asked him several questions about security best practices and software recommendations. I don't remember a lot about that meeting, but I do remember his strong recommendation of Crowdstrike. Interesting to see it brought up again in this context.
- farceSpherule 2y agoI absolutely abhor these end point solutions that "auto update for your convenience and safety." I can control and manage my own systems. I do not need nanny state auto updating for me. Crowdstrike should be held liable for financial losses associated with this nonsense.
- deleted 2y ago[deleted]
- axelthegerman 2y agoLooks like crowdstrike are just delivering what their name promised, striking crowds around the world
- aktuel 2y agoGermany is not affected since it's Krautstrike only.
- GrumpyNl 2y agoHo do they test this before they roll it out? Looks like a bug thats easy to spot. I would presume they test it at several configurations and when it passes the test ( a reboot), they roll it out. Has this been tested?
- xyst 2y agoCRWD dropped $50/share at market open. Wild. Is this specific to only Windows machines “protected” with CS or is this impacting Linux/macOS as well?
- remram 2y agoI can't wait to see the CloudFlare traffic report after this. All those computers going down must have affected traffic worldwide. Even from Linux systems as their owners couldn't run jobs from their bricked Windows laptops.
- aeyes 2y agoIt looks quite normal so far: https://radar.cloudflare.com/traffic https://radar.cloudflare.com/traffic DE-CIX traffic is also often a good indicator during global events, looks normal: https://www.de-cix.net/en/locations/frankfurt/statistics https://www.de-cix.net/en/locations/frankfurt/statistics
- remram 2y agoInteresting! Thanks for that. I guess most servers and consumer endpoints are fine, and those are driving all the traffic.
- charlie0 2y agoNext up: CrowdStrike Sued for Alleged Negligence. I hate lawyers, but this is the reason why companies outsource. Why take the blame (and spend the money) when you can blame the vendor?
- Geezus_42 2y ago"Incidents of this nature do occur in a connected world that is reliant on technology." - Mike Maddison, CEO, NCC Group Until I see an explanation of how this got past testing, I will assume negligence. I wasn't directly affected, but it seems every single Windows machine running their software in my org was affected. With a hit rate that high I struggle to believe any testing was done.
- zzhelezc 2y agoFrom the BBC's cyber correspondent Joe Tidy [1]: > A "content update" is how it was described. So, it wasn’t a major refresh of the cyber security software. It could have been something as innocuous as the changing of a font or logo on the software design. He can't be serious, right? Right? [1] https://www.bbc.co.uk/news/live/cnk4jdwp49et?post=asset%3Abd501d28-fe49-4e4e-8605-194da98eeb6c#post https://www.bbc.co.uk/news/live/cnk4jdwp49et?post=asset%3Abd...
- Beijinger 2y agoSpeaking of security. I got an email yesterday that I need a different system now to log into my social security account. This one: https://www.id.me/government https://www.id.me/government It is for social security, taxes, unemployment benefits, whatever. And running under a foreign TLD, .ME for Montenegro. I am not a security specialist. But I think this is asking for trouble. By the way, do you remember when fuck.yu became fuck.me ?
- rajeshivivek 2y agoDO NOT REDEEM SAARRRRRRRRSSSSS! BLODDY BASTARDS INVALID FORMATING SARRRRRRSSSSS!!
- stevetron 2y agoWorking late Thursday night in Florida, USA. I have someone in Australia wanting me to write a quick script in LSL for an object in Second Life. We were interrupted: Second Life kept running, but Discord went down, telling me to 'try another server' which doesn't make sence when you are 1-on-1 with someone. All my typing in Discord turned red. Additionally, I couldn't log into the email portal for outlook.com: I got a screen of tiny-fonted text all clinging to the left edge of the display, unreadable, unusable. Second Life, though, stayed online and kept working for me, but then I'm on Windows 7. My friend who had requested the collaboration froze in Second Life on his Windows 10 system, and I don't know what his Discord was doing. I ended the session since I couldn't get a no/no-go out of him for the latest script version.
- _def 2y agoWow I didn't know second life was still a thing. Literally yesterday I looked at a 20 year old archived version of a freeware portal which also listed a version of second life.
- dboreham 2y agoLooks like it affected the Crowdstrike stock, but not Microsoft.
- upofadown 2y agoPerhaps a dumb question for someone who actually knows how Microsoft stuff works... Why would an anti-malware program be allowed to install a driver automatically ... or ever for that matter? Added: OK, from another post I now know Crowdstrike has some sort of kernel mode that allows this sort of catastrophe on Linux. So I guess there is a bigger question here...
- atoav 2y agoMaybe I am in the minority, but it always puzzled me that anybody in IT would think a mega-priviledged piece of software that looks into all files was a good idea. If there is any place that historically was exploited more than all other things it was broken parsers. Congratulations if such an exploited file is now read by your AV-software it now sits now at a position where it is allowed (expected) to read all files and it would not surprise me if it could write them as well. And you just doubled the number of places in which things can go wrong. Your system/software that reads a PNG image might do everything right, but do you know how well your AV-software parses PNGs? This is just an example, but the question we really should ask ourselves is: why do we have systems where we expect malicous files to just show up in random places? The problem with IT security is not that people don't use AV software, it is that they run systems that are so broken by design that they are sprinkled on top. This is like installing a sprinkler system in a house full of gasoline. Imagine gasoline everywhere including in some of the water piping — in the best case your sprinkler system reacts in time and kills the fire, in the worst case it sprays a combustive mix into it. The solution is of course not to build houses filled with gasoline. Meanwhile AV-world wants to sell you ever more elaborate, AI-driven sprinkler systems. They are not the ones profiting from secure systems, just saying..
- Sohcahtoa82 2y ago> but it always puzzled me that anybody in IT would think a mega-priviledged piece of software that looks into all files was a good idea. Because otherwise, a piece of malware that installs itself at a "mega-privileged" level can easily make itself completely invisible to a scanner running as a low-priv user. Heck, just placing itself in /root and hooking a few system calls would likely be enough to prevent a low-priv process from seeing it.
- rajeshivivek 2y agoSARRRRRRRSSSSS!
- purpleblue 2y agoDo all the machines need to be manually fixed? It doesn't seem like an automatica update will work here...
- snappr021 2y ago“To err is human, but to really fuck things up requires a computer.” ~ Len Beattie
- resters 2y agoAny company that inserts itself so heavily into US politics cannot be counted on as a solid engineering organization.
- localfirst 2y agoThis is the first time I'm hearing about crowdstrike, what is it and why is this such a big deal?
- casey2 2y agoIsn't Crowdstrike the same company the heavily lobbied to get make all their features a requirement for government computers? https://www.opensecrets.org/federal-lobbying/clients/summary?cycle=2021&id=D000084079 https://www.opensecrets.org/federal-lobbying/clients/summary... They have plenty of money for congress, but it seem little for any kind of reasonable software development practices. This isn't the first time crowdstrike has pushed system breaking changes.
- WillPostForFood 2y agoSince we are in political season here in the US, they are also well known as the company that investigated the Russian hack of the DNC. https://www.crowdstrike.com/blog/bears-midst-intrusion-democratic-national-committee/ https://www.crowdstrike.com/blog/bears-midst-intrusion-democ...
- pjot 2y agoThe DNC has since has implemented many layers of protection including crowdstrike, hardware keys, as well as special auth software from Google. They learned many lessons from 2016.
- laidoffamazon 2y agoIf I were to hazard a guess I think the OP is attempting to say they are incompetent and wrong in fingering the GRU as the cause of the DNC hacks (even though they were one of many groups that made that very obvious conclusion).
- pjot 2y agoWhat? No.
- laidoffamazon 2y agoNot you, the person you were responding to.
- steelframe 2y agoWow, this hits close to home. Doing a page fault where you can't in the kernel is exactly what I did with my very first patch I submitted after I joined the Microsoft BitLocker team in 2009. I added a check on the driver initialization path and didn't annotate the code as non-paged because frankly I didn't know at the time that the Windows kernel was paged. All my kernel development experience up to that point was with Linux, which isn't paged. BitLocker is a storage driver, so that code turned into a circular dependency. The attempt to page in the code resulted a call to that not-yet-paged-in code. The reason I didn't catch it with local testing was because I never tried rebooting with BitLocker enabled on my dev box when I was working on that code. For everyone on the team that did have BitLocker enabled they got the BSOD when they rebooted. Even then the "blast radius" was only the BitLocker team with about 8 devs, since local changes were qualified at the team level before they were merged up the chain. The controls in place not only protected Windows more generally, but they even protected the majority of the Windows development group. It blows my mind that a kernel driver with the level of proliferation in industry could make it out the door apparently without even the most basic level of qualification.
- bonestamp2 2y ago> without even the most basic level of qualification That was my first thought too. Our company does firmware updates to hundreds of thousands of devices every month and those updates always go through 3 rounds of internal testing, then to a couple dozen real world users who we have a close relationship with (and we supply them with spare hardware that is not on the early update path in case there is a problem with an early rollout). Then the update goes to a small subset of users who opt in to those updates, then they get rolled out in batches to the regular users in case we still somehow missed something along the way. Nothing has ever gotten past our two dozen real world users.
- rvnx 2y agoOr it could be made that Windows stops loading drivers that are crashing. Third-party driver/module crashed more than 3 times in a row -> Third-party driver/module is punished and has to be manually re-enabled.
- CKMo 2y agoThis is a good example of why you don't want ring0 level access for clients. Or just, you don't want client-based solutions. The provider just becomes another threat vector.
- swozey 2y agoI know there's a better word to be used here, but what initially looked like a massive cyberattack turning out to be a massive defender foot-broom is chefs kiss. I saw it was Windows and went to bed. What a great feeling. I'm sorry to those of you dealing with this. I've had to wipe 1200 computers over a weekend in a past life when a virus got in. Did I receive any appreciation? Nope. I was literally sleeping under cubicle desks bringing up isolated rows one by one. I switched everything in that call center to linux after that. Ironically it turns out it was a senior engineers ssh key that got leaked somehow and was used to get in and dig around servers in our datacenter outside of my network. My filesystem logging (in Windows, coincidentally) alerted me. IT is fun.
- cloin 2y agoI'm confused as to how this issue is so widespread in the first place. I'm unfamiliar with how Crowdstrike works, do organizations really have no control over when these updates occur? Why can't these airlines just apply the updates in dev first? Is it the organizations fault or does Crowdstrike just deliver updates like this and there's no control? If that's just how they do it, how do they get away with this?
- rboyd 2y agoCompanies operate on a high level of fear and trust. This is the security vendor, so in theory they want those updates rolled out as quickly as possible so that they don't get hacked. Heh.
- prpl 2y agoI mean, they pay a lot of money to crowdstrike. A failure this widespread is a Crowdstrike dev issue.
- commandlinefan 2y agoCan somebody summarize what CrowdStrike actually is/does? I can't figure it out from their web page (they're an "enterprise" "security" "provider", apparently). Is this just some virus scanning software? Or is it some bossware/spyware thing?
- kube-system 2y ago> Is this just some virus scanning software? Essentially, yes. It is fancy endpoint protection.
- noduerme 2y agoIt's both. Antivirus along with spyware to also watch for anything the user is doing that could introduce a threat, such as opening a phishing email, posting on HN, etc.
- moffkalast 2y ago
- stainablesteel 2y agoits strange how people who work in professions that are considered crucial infrastructure are held to such a high standard but there's always some tech problem that cripples them the hardest
- queuebert 2y agoAnd they all invariably use Windows instead of a high-reliability OS.
- cypress66 2y agoWindows is high reliability. The problem here is what's basically a third party backdoor.
- DeepYogurt 2y ago> Windows is high reliability. lol no
- bestouff 2y ago"Windows" is the combination of the OS per se and all the things needed for it to run properly. That thing is a mess of proprietary drivers and pieces of software cobbled together. It can't be called "high reliability" with a straight face.
- lupire 2y agoCrowdstrike is a multiplatform malware that chronically damages computers on all major desktop OSes. This is a Crowd strike problem and an admin problem.
- BuckRogers 2y agoThat's a hell of a take that should not be taken seriously. Perhaps if you hold everything else to the same standard. Anything used on macOS or Linux or whatever else fully and completely represents that core platform, then I'd agree. Anecdotally, I have zero stability problems on my non-ECC consumer-grade 11th gen Intel Windows 11 system. It'll stay up for months, until I decide to shut it down. I had a loose GPU power cable that was causing me problems at a point, but since I reseated everything I haven't had a single issue. That was my fault, things happen. The system is great. More significantly, I see no difference in stability between our Windows Server platform and Red Hat Enterprise (Oracle) server platform at work either. Work being one of the top 3 largest city governments in the USA.
- rboyd 2y agoSeems like a modern operating system would have an automatic rollback mechanism for cases like this.
- Kye 2y agoWindows has restore points that do this in the event of a failed update, but this wasn't Windows.
- tgtaptarget 2y agoIn my org, none of the essential systems went down (those used by labor). However all of management's individual PCs went down which got me wondering... Is this the beginning (or continuation) of whittling down what is "essential" human labor versus what could be done remotely (or eliminated completely)? Or perhaps Microsoft is just garbage and soon will be as irrelevant as commercial real estate office parks and mega-call centers
- eatsyourtacos 2y ago[flagged]
- Jabrov 2y agoSure this isn't Windows' fault but it's hardly a "high reliability OS"
- consteval 2y agoThe reason people "can't learn" how to operate alternative software is because we don't give software the weight it deserves. We don't consider it crucial, but evidently it is. When a new surgical technique is standardized, we don't tell doctors "well don't worry about it - we can't expect you to learn cryptic things!" Because we understand the gravity of what they're doing and how crucial that technique may be. For whatever reason, software is still treated like the wild west and customers/employees are still babied. They're told it's all optional, they don't need to learn more. We still tell Windows users its fine to download executables online, click "okay!" and have them run as Admin. And that's the root cause of why we're in this mess. We have safer computing environments - just look at iOS or Mac. Even Microsoft is slowly trying to faze this out with the new Windows store. But alas, we cannot expect anyone to change anything ever, so we still use computers like it's 1995.
- avhception 2y agoAssuming that other operating system, whether high-reliability or not, are necessarily "cryptic" and unnecessarily impair people in their ability to "get shit done" is naive at best and disingenuous at worst.
- dang 2y agoPlease don't cross into the flamewar style in HN comments. It's not what this site is for, and destroys what it is for. We detached this subthread from https://news.ycombinator.com/item?id=41007791 https://news.ycombinator.com/item?id=41007791.
- kubov 2y agoWere cloud providers (AWS and azure) so heavily impacted because they use CS internally or because so many users use CS?
- raphar 2y agoI want to see the internal postmortem of why this happened to CrowdStrike (if they are still in business)
- gonzo41 2y agoDo people not have test environments?
- tbatchelli 2y agoThis event is predicted in Sydney Dekker’s book “Drift into Failure”, which basically postulates that in order to prevent local failure we setup failure prevention systems that increase the complexity beyond our ability to handle, and introduce systemic failures that are global. It’s a sobering book to read if you ever thought we could make systems fault tolerant.
- COGlory 2y agoWe need more local expertise is really the only answer. Any organization that just outsources everything is prone to this. Not that organizations that don't outsource aren't prone to other things, but at least their failures will be asynchronous.
- bjelkeman-again 2y agoFunny thing is that for decades there were predictions about how there was a need for millions of more IT workers. It was assumed one needed local knowledge in companies. Instead what we got was more and more outsourced systems and centralized services. This today is one of the many downsides.
- downrightmike 2y agoTwo weeks ago it was just about all car dealers
- smsm42 2y agoThe problem here would be that there's not enough people who can provide the level of protection a third-party vendor claims to provide, and a person (or persons) with comparable level of expertise would be much more expensive likely. So companies who do their own IT would be routinely outcompeted by ones that outsource, only for the latter to get into trouble when the black swan swoops in. The problem is all other kinds of companies are mostly extinct by then unless their investors had some super-human foresight and discipline to invest for years into something that year after year looks like losing money.
- cjbgkagh 2y agoDue to the scale I think it’s reasonable to state that in all likelihood many people have died because of this. Sure it might be hard to attribute single cases but statistically I would expect to see a general increase in probability. I used to work at MS and didn’t like their 2:1 test to dev ratio or their 0:1 ratio either and wish they spent more work on verification and improved processes instead of relying on testing - especially their current test in production approach. They got sloppy and this was just a matter of time. And god I hate their forced updates, it’s a huge hole in the threat model, basically letting in children who like to play with matches. My important stuff is basically air-gapped. There is a gateway but it’ll only accept incoming secure sockets with a pinned certificate and only a predefined in-house protocol on that socket. No other traffic allowed. The thing is designed to gracefully degrade with the idea that it’ll keep working unattended for decades, the software should basically work forever so long as equivalent replacement hardware could be found.
- Supermancho 2y ago> And god I hate their forced updates, My windows machine notified me of the update, asked me to restart. I was busy, so I didn't. Then the news broke, then the update was rolled back.
- vel0city 2y agoIt wasn't a Windows update. If you got a notification for an update, it wasn't the update that did this.
- dtech 2y agoI don't see what this has to much do with MS. A bad proprietary kernel module can crash any OS.
- cjbgkagh 2y agoI don’t know the specifics of this case, but formal verification of machine code is an option. Sure it’s hard and doesn’t scale well but if it’s required then vendors will learn to make smaller kernel modules. If something cannot be formally verified at the machine code level there should be a controls level verification where vendors demonstrate they have a process in place to achieving correctness by construction. Driver devs can be quite sloppy and copy paste bad code from the internet, in the machine code Microsoft can detect specific instances of known copy and pasted code and knows how to patch it. I know they did this for at least one common error. But if I was in the business of delivering an OS I want people to rely on my OS this stuff formal verification at some level would be table stakes.
- nimbius 2y agoI work for a diesel truck repair facility and just locked up the doors after a 40 minute day :( . - lifts wont operate. - cant disarm the building alarms. (have been blaring nonstop...) - cranes are all locked in standby/return/err. - laser aligners are all offline. - lathe hardware runs but controllers are all down. - cant email suppliers. - phones are all down. - HVAC is also down for some reason (its getting hot in here.) the police drove by and told us to close up for the day since we dont have 911 either. alarms for the building are all offline/error so we chained things as best we could (might drive by a few times today.) we dont know how many orders we have, we dont even know whos on schedule or if we will get paid.
- shoebham 2y agowow, why do lifts require an OS?
- warkdarrior 2y agoHow else are you going to update your grocery list while operating the lift?
- kulikalov 2y agothe question is - why lifts require windows?
- GirishSharma643 2y agoWho is responsible for this billion dollar mistake?
- vlan0 2y agoAnything that has root/kernel access is a risk. It always has been. When will we learn. Probably never. Because money runs this world. So sad. Time to open a bakery and move on from this world.
- Sohcahtoa82 2y agoConsidering what Crowdstrike is intended to do, it's not really possible for it to work without running at the kernel level.
- steine65 2y agoHere's a visual representation of flight cancellations and delays at major US airports https://www.flightaware.com/miserymap/ https://www.flightaware.com/miserymap/
- irusensei 2y agoCan we end the whole “loading a kernel rootkit” thing? AFAIK Apple already shuns kernel extensions. What’s preventing Microsoft to do the same? As a bonus shit like anti cheat will go away too.
- _yb2s 2y agoWhy are so many mission critical hardware connected systems connected to the internet at all or getting automatic updates? This is just basic IT common sense. You only do updates during a planned outage, after doing an easily reversible backup, or you have two redundant systems in rotation and update and test the spare first. Critical systems connected to things like medical equipment should have no internet connectivity, and need no security updates. I follow all of this in my own home so a bad update doesn’t ruin my work day… how do big companies with professional IT not know this stuff?
- gedy 2y ago> Why are so many mission critical hardware connected systems connected to the internet at all or getting automatic updates? Because it lets them "scale" by having fewer and cheaper offsite IT and contractors to manage vs hiring pesky onsite employees.
- basch 2y agoYou do that for antivirus definition updates?
- mckn1ght 2y agoProbably, implicitly. Have automated regular backups, and don’t let your AV automatically update, or even if it does, don’t log into all your computers simultaneously. If you update/login serially, then the first BSOD would maybe prevent you from doing the same thing on the other (or possibly, send you running to the other to accomplish your task, and BSODing that one too!) But yeah this is one reason why I don’t have automatic updates enabled for anything, the other major one being that companies just can’t resist screwing with their UIs.
- _yb2s 2y agoI’m not an IT professional, but I don’t use antivirus software on my personal macs and linux machines- I do regular rotated physical backups, and only install software digitally signed by trusted sources and well reviewed Pirate Bay accounts (that's a joke :-). My only windows machine is what I would classify as a mission critical hardware connected/control device, an old Windows 8 tablet I use for car diagnostics- I do not connect it to the internet, and never perform updates on it. I am an academic and use a lot of old multi-million dollar scientific instruments which have old versions of windows controlling them. They work forever if you don't network them, but the first time you do, someone opens up a browser to check their social media, and the entire system will fail quickly.
- franczesko 2y agoI just wanted to mention that Microsoft has 3 tiers of Windows beta releases before changes are pushed to production. I can't comprehend how this wasn't noticed before.
- bloopernova 2y agoIt didn't come from Microsoft or Windows Update. It was pushed by Crowdstrike to their corporate security kernel extension.
- franczesko 2y ago3rd party got unsupervised access to kernel? I'd say it's even worse then
- jimt1234 2y agoI watched a presentation by someone representing "I Am The Cavalry" at B-Sides, Las Vegas, a few years ago. Very interesting stuff, gave me a whole new perspective on "cyber security". https://iamthecavalry.org https://iamthecavalry.org
- rootforce 2y agoAWS has posted some instructions for those affected by the issue using EC2. [AWS Health Dashboard](https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status) "First, in some cases, a reboot of the instance may allow for the CrowdStrike Falcon agent to be updated to a previously healthy version, resolving the issue. Second, the following steps can be followed to delete the CrowdStrike Falcon agent file on the affected instance: 1. Create a snapshot of the EBS root volume of the affected instance 2. Create a new EBS volume from the snapshot in the same Availability Zone 3. Launch a new instance in that Availability Zone using a different version of Windows 4. Attach the EBS volume from step (2) to the new instance as a data volume 5. Navigate to the \windows\system32\drivers\CrowdStrike\ folder on the attached volume and delete "C-00000291*.sys" 6. Detach the EBS volume from the new instance 7. Create a snapshot of the detached EBS volume 8. Create an AMI from the snapshot by selecting the same volume type as the affected instance 9. Call replace root volume on the original EC2 Instance specifying the AMI just created"
- rahkiin 2y agoThat is a lot of steps. Can this not be scripted?
- timoteostewart 2y agoFWIW, I find the high-level overview more useful, because then I can write a script tailored to my situation. Between `bash`, `aws` CLI tool, and Powershell, it would be straightforward to programmatically apply this remedy. Here's something quick that ChatGPT ginned up: https://chatgpt.com/share/293ea9d5-b7ac-4064-b870-45f8266aea63 https://chatgpt.com/share/293ea9d5-b7ac-4064-b870-45f8266aea...
- CalumSult 2y agoYes it can, that's what I ended up writing at 4am this morning, lol. We manage way more instances then is feasible to do anything by hand. This is probably too late to help anyone, but you can also just stop instance, detach root, attach it to another instance, delete file(s), offline drive, detach, reattach to original instance, and then start instance. You need a "fixer" machine in the same AZ.
- PaulHoule 2y agoPeople at my workplace were affected but I dodged the bullet because I left my computer turned on overnight because I always want to be able to RDP in the next morning in case I decide to stay home.
- janalsncm 2y agoThis outage may be more expensive and cause more damage than any cyberattack in history.
- rs999gti 2y agoSo can crowdstrike be classified as malware now? Currently waiting in line for 2 hours + waiting for Delta to tell me when my connecting leg can be booked. My current flight is delayed 5 hours.
- Zaskoda 2y agoI want to add something to the discussion but it's difficult for me to accurately summarize and cite things. In a nutshell, there appears to be a lot of tomfoolery with CrowdStrike and the stuff that happened with the DNC during the 2016 election. Here's some of what I'm talking about: There's a strong link between the DNC, Hillary, and CrowdStrike. Here's once piece that links a cofounder of CrowdStrike with Hillary pretty far back: https://www.technologyreview.com/innovator/dmitri-alperovitch/ https://www.technologyreview.com/innovator/dmitri-alperovitc... This 2017 piece talks about doubt behind CrowdStrike's analysis of the DNC hack being the result of Russian actors. One of the groups disputing CrowdStrike's analysis was Ukraine's military. https://www.voanews.com/a/crowdstrike-comey-russia-hack-dnc-clinton-trump/3776067.html https://www.voanews.com/a/crowdstrike-comey-russia-hack-dnc-... This detailed analysis of CrowdStrike's explanation of the DNC hack goes so far as to say "this sounded made up" https://threatconnect.com/resource/webinar-guccifer-2-0-the-dnc-hack-and-fancy-bears-oh-my/ https://threatconnect.com/resource/webinar-guccifer-2-0-the-... The Threat Connect analysis is also discussed here: https://thehill.com/business-a-lobbying/295670-prewritten-guccifer-20-remarks-read-by-confederate-at-security-conference/ https://thehill.com/business-a-lobbying/295670-prewritten-gu... "For one, the vulnerability he claims to have used to hack the NGP VAN ... was not introduced into the code until an update more than three months after Guccifer claims to have entered the DNC system." Noted at the end of this story they mention that CrowdStrike installed it's software on all of the DNC's systems: https://www.ft.com/content/5eeff6fc-3253-11e6-bda0-04585c31b153 https://www.ft.com/content/5eeff6fc-3253-11e6-bda0-04585c31b... Finally, there's this famous but largely forgotten story of the time Bernie's campaign was accused to accessing Hillary's data: https://www.npr.org/2015/12/18/460273748/bernie-sanders-campaign-locked-out-of-key-voter-file-after-data-breach https://www.npr.org/2015/12/18/460273748/bernie-sanders-camp... "This was a very egregious breach and our data was stolen," Mook said. "We need to be sure that the Sanders campaign no longer has access to our data." "This bug was a brief, isolated issue, and we are not aware of any previous reports of such data being inappropriately available," the company said in a blog post on its website. (edited for spelling)
- energy123 2y agoOn the plus side this will help us develop an immune system against cyber attacks in any future war. Businesses will start thinking of contingencies.
- Kye 2y agoThere's a workaround: reboot 10-15 times. I've seen two people say it independently, so maybe it's for real.
- tonymet 2y agoNo rolling updates? How could a 100% repro BSOD pass QC? I'm more concerned about the deployment process than the crash itself. Everyone experiences a bad build from time to time. How did this possibly go live?
- Key89 2y agois there an ELI5 on how can this happen? Like i get its a boot loop, but what did crowdstrike do that cause it? How can non malicious code trigger boot loop?
- monocasa 2y agoKernel driver bug that essentially defaults, then on reboot loads the same driver early on segfaults and reboots again, ad nauseum.
- baryphonic 2y agos/defaults/segfaults/ # stupid autocorrect
- monocasa 2y agoThanks for catching that.
- sieste 2y agoBest ELI5 ever!
- __MatrixMan__ 2y agoCrowdstrike needs to be the first person in the room so that they can act like the boss. If other people show up before crowdstrike, there's a possibility that they'll somehow prevent crowdstrike from being the boss. For this reason, crowdstrike integrates with the boot process in ways that most software doesn't. Their ability to monitor and intervene against all software on the system also puts them in a position to break all software on the system. more accurately: s/boss/most informed spy/g
- fsloth 2y agoI would not call Crowdstrike ”non-malicious”. It’s incredibly incompetently implemented kit that’s sold to organizations as snakeoil that ”protects them from cybercrime”. It’s purpose is to give incompetent IT managers to ”implement something plausible” against cyberincidents, and when an incident happens, it gives them the excuse that ”they followed best practices”. It craps the users PC while at it too. I hope the company burns to the ground and large organizations realize it’s not a really great idea to run a rootkit at every PC ”just because everyone else does it”.
- piuantiderp 2y agohow come does anyone still use crowdstrike?
- worldwidelies 2y ago[dead]
- 1024core 2y agoRead on Mastodon: https://infosec.exchange/@littlealex/112813425122476301 https://infosec.exchange/@littlealex/112813425122476301 The CEO of Crowdstrike, George Kurtz, was the CTO of McAfee back in 2010 when it sent out a bad update and caused similar issues worldwide. If at first you don't succeed, .... ;-) j/k
- BobbyTables2 2y agoI didn’t understand why in 2010, it didn’t seem to make most news… Took out the entire company where I worked. People thought it was a worm/virus — few minutes after plugging in laptop, McAfee got the DAT update, quarantined the file; which caused Windows to start countdown+reboot (leading to endless BSODs).
- localfirst 2y agoKurtz response is ridiculous blaming the customer on X. He will probably find another company to hire him as CEO tho. Just an upside down world in the C-suite world.
- BaldricksGhost 2y agoDon't forget the golden parachute. These guys always seem to fail upward.
- jeffrallen 2y agoThat guy is gonna fail all the way right up to the top. Sheesh.
- deleted 2y ago[deleted]
- bloopernova 2y agoWorth $3.7B, paid $148M in 2022. Edited to add: I wonder what the economic fallout from this will be? 10x his monetary worth? 100x? (not trying to put a price on the people who will die because of the outage; for that he and everyone involved needs to go to jail)
- belter 2y agoIt's not the first time they pull something similar...1 month ago: "CrowdStrike bug maxes out 100% of CPU, requires Windows reboots" - https://www.thestack.technology/crowdstrike-bug-maxes-out-100-of-cpu-requires-windows-reboots/ https://www.thestack.technology/crowdstrike-bug-maxes-out-10... 75 Billion dollars valuation, CNBC Analysts praising the company this morning on how well the company is run!...When in reality they can't master the most basic of the phased deployment methodologies known for 20 years... Hundreds of handsomely paid CTO's, at companies with billions of dollars in valuations, critical healthcare, airlines, who can't master the most basic of the concepts of "Everything fails all the time"... This whole industry is depressing....
- monkmartinez 2y agoThis borked our dispatch/911 call center then as well. However, it wasn't as bad as this one. This outage put our entire public safety system into the stone age and with that we were at stone age efficiency.
- BaldricksGhost 2y agoI work IT at a regional 911 center. We're fine but I sympathize with those who are back to pen and paper dispatching. Hard for most current dispatchers to realize the way we did it back in the day.
- chronid 2y agoWhat I find definitely depressing is the fact we used to roll out progressively even OS upgrades (I guess now that is done through intune?) and was one point in favor of windows (on Linux you had to do things yourself at the time AFAIK, I don't think the situation has improved much). Nowadays we get mandated random software upgrading at once on the entire company fleet and no one bats an eye - I counted more than a dozen agents installed for "security" and "monitoring" purposes in my previous company servers, many of those with hooks in the kernel obviously, and many of those installed with random policies to tick yet another compliance box...
- 2y ago
- nu11ptr 2y agoThis whole thing likely would have been averted had microkernel architectures caught on during the early days (with all drivers in user mode). Performance would have likely been a non-issue, not only due to the state of the art L4 designs that came later, but mostly because had it been adopted everything in the industry would have evolved with it (async I/O more prevalent, batched syscalls, etc.). I will admit we've done pretty well with kernel drivers (and better than I would have ever expected tbh), but given our new security focused environment it seems like now is the time to start pivoting again. The trade offs are worth it IMO.
- lizknope 2y agoNot disagreeing with you but we need operating systems with snapshots before updates and a trivial way to rollback the update. Linux has some immutable OS versions and also btrfs snapshots and booting a specific snapshot from the GRUB bootloader
- crote 2y agoI wonder if for critical applications we'll ever go back to just PXE booting images from a central server: just load a barebones kernel and the app you want to run into a dedicated memory segment, mark everything else as NX, and you don't even have to worry about things like viruses and hacks anymore. Run into an issue? Just reboot!
- deleted 2y ago[deleted]
- chinathrow 2y agoYet their stock tanked only a couple of dollars. They (and their customers) should face some rather unpleasant lawsuits. If you let others own your systems, you should not be allowed to provide critical infrastructure.
- rabidonrails 2y agoI don't think you're going to see as many lawsuits are you think. Most of these contracts probably state that they had to follow reasonable precautions for business continuity and data recovery. Having Crowdstrike in the path seems to have been a reasonable and potentially best practice before today's outage. I don't think that companies are going to be held liable at all.
- akdev1l 2y agoCrowdstrike should be held liable and sued out of existence.
- medion 2y agoThey won’t. If crowdstrike was an individual or a state there would be repercussions. But this will all be a forgotten memory in two weeks or less.
- fsloth 2y agoI do hope some organizations realize though it’s not a great idea to have a half-baked rootkit as your lucky charm against cybercrime.
- rabidonrails 2y agoEh. I think you're underestimating how overmatched these IT depts are when it comes to cybersecurity. Either sign a contract with a best-in-class (even if in name only) vendor who says that they'll do all of this for us or we need to become "experts" in cybersecurity and potentially still use them. The CIO is overmatched here so they're making the decision that protects them and their clients in _almost all_ cases.
- Ringz 2y agoBy chance, I watched a few episodes of 911 and kept thinking that it was all completely unrealistic nonsense. Then there's an episode where the entire emergency call system for LA goes down, and even though there were different reasons in the episode (a transformer fire), I couldn't have imagined that it was actually possible to completely disable the emergency call system (and what else) of a city.
- qwerty456127 2y agoWTF is CrowdStrike and why is it affecting so many people and companies? I've never heard of it before. And apparently it isn't anything relevant to all Windows users as it didn't affect any computer of any person I personally know.
- etc-hosts 2y agoMission critical systems should be running something like ChromeOS. Too bad ChromeOS seems be on the way out at Google.
- nothercastle 2y agoI love how their company name foreshadows this exact event. It’s malware pretending to be a security suite.
- satisfice 2y agoI want to say the problem is that the industry has systematically devalued software testing in favor of continuous delivery and the strategy of hoping that any problems are easy to roll back. But it's deeper than that: the industry realizes that, once you get to a certain size, no one can hurt you much. Crowdstrike will not pay a lasting penalty for what has just happen, which means executives will shrug and treat this as a random bolt of lightning.
- 01100011 2y agoThis might be a good time for folks to go back and watch the first episode of James Burke's Connections: The Trigger Effect https://www.youtube.com/watch?v=NcOb3Dilzjc https://www.youtube.com/watch?v=NcOb3Dilzjc Interconnected systems can fail spectacularly in unforeseen ways. Strange that something so obvious is so often dismissed or overlooked.
- rincebrain 2y agoIf you realize something horrific, your options are to decide it's not your problem (and feel guilty when it blows up), carefully forget you learned it, or try to do something to get it changed. Since the last of these involves overcoming everyone else's shared distress in admitting the emperor has no clothes, and the first of these involves a lot of distress for you personally, a lot of people opt for option B.
- HPsquared 2y agoOption C is to quietly distance yourself from it.
- __MatrixMan__ 2y ago> overcoming everyone else's shared distress in admitting the emperor has no clothes I don't disagree, but why do we do we react this way? Doesn't knowing the emperor has no clothes instill a bit of hope that things can change? I feel for the people who were impacted by this, but I'm also a little bit excited. Like... NOW can we fix it? Please?
- AmpsterMan 2y agoGoing off of that framework: Because doing three requires convincing a bunch of people that are currently doing number two, to do number one instead.
- johnklos 2y agoIt's a nice idea, but has that worked historically? Some people will make changes, but I think we'd be naive to think that things will change in any large and meaningful way. Having another I-told-you-so isn't so bad, though - it does give us IT people a little more latitude when we tell people that buying the insecurity fix du jour increases work and adds more problems than it addresses.
- coderinsan 2y agoCrowdStrike today has shown why it's absolutely crucial to test code before deployment, say no to YOLO deployments with LLM powered software testing https://github.com/codeintegrity-ai/mutahunter https://github.com/codeintegrity-ai/mutahunter
- LrnByTeach 2y agowhat many people of not taking is that why we are here: one simple reason: all eggs in one Microsoft PC basket why in one Microsoft PC basket? - most corporate desktop apps are developed for Windows ONLY Why most corporate desktop apps are developed for Windows ONLY? - it is cheaper to develop and distribute since, 90% of corporations use Windows PCs ( Chicken and Egg problem) - alternate Mac Laptops are 3x more expensive, so corporations can't afford - there are no robust industrial grade Linux laptops from PC vendors (lack of support, fear of Microsoft may penalize for promoting Linux laptops etc.) 1/ Most large corporations (Airlines, Hospitals etc..) can AFFORD & DEMAND their Software vendors to provide their ' business desktop applications' both in Windows and Linux versions and install mix of both Operating systems. 2/ majority of corporate desktop applications can be Web applications (Browser based) removing the single vendor Microsoft Windows PC/Laptops -
- ahachete 2y agoLenovo and Dell have some laptops with Linux, and they are very good ones. (not sure if you meant rugged ones, that may not be the case, but I guess this is a tiny percentage of the market)
- linuxlizard 2y agoCrowdstrike also has an Ubuntu Linux version. We're required to install it at work.
- trhway 2y agoSolarWinds story was quickly forgotten, and this one will be too, and we'll continue to build such special single points of global catastrophic failure into our craftly architected decentralized highly robust horizontally scaled multi-datacenter-region systems
- rtev 2y agoSolarwinds is still dealing with the reputation damage and fallout today from that breach. People don’t forget about this stuff. the lawsuits will likely be hitting crowdstrike for years to come
- rewgs 2y agoThe most concerning thing about this is the realization of just how many incredibly critical systems run on Windows.
- apantel 2y agoThis just in ‘CrowdStrike Strikes Crowd’
- insane_dreamer 2y agoHow is it that these major companies aren't rolling out vendor updates to a small number of computers first to make sure that nothing broke, and then rolling out to the entire fleet? That's deployment 101.
- rustcleaner 2y agoThank Chronos I switched to Qubes OS almost two years ago!
- siliconc0w 2y agoThe postmortem will should interesting, can't imagine how even just basic integration testing didn't catch this. Much less basic best practice like canarying.
- low_tech_punk 2y agoCrowdstruck
- m3kw9 2y agoIt’s that easy. A hacker that controls the update terminal at crowdstrike controls the world?
- cjbgkagh 2y agoEdit; it appears my comment has been moved to a top level comment, i.e. peer with the parent without any way of telling what happened - so now there is the whole other pointless branch polluting the relevance of the tree. Previously; It appears that someone was able to take my previous comment in this thread completely off hacker news, it's not even listed as flagged. It was at 40pts before disappearing, perhaps there is some reputation management going on here. If it was against the site rules it would be helpful to know which ones. Edit; the link is https://news.ycombinator.com/item?id=41007985 https://news.ycombinator.com/item?id=41007985 it was a high up comment that no longer appears even though flagged comments do appear. I checked if it has been moved but the parent comment is still the same. This feels like hellbanned in that there isn't an easy way for me to see if I've been shadowbanned. But I really don't know. I was commenting in good faith.
- nine_k 2y agoOh, nevermind, it's just a third-party cybersecurity tool running on the server detected a potential threat and quarantined the offending database record, just in case!
- cjbgkagh 2y agoI was looking forward to spending the day talking to people about cyber security but if my comments are going to disappear like that then maybe hacker news is not the site for me. A shame really. Edit; I don't know for sure but this is possibly the last straw for me on hacker news. It really has gone downhill. If good faith discussions from experts are being secretly deleted for what I can only now assume are for nefarious reasons then I can't trust what I find here is in anyway representative. It's unfortunate in that there really isn't anywhere else to go. Now my best discussions are in small WhatsApp groups / Discords with friends. It's ok for me where I have had a career to get to know people personally and have such groups but if public forums are tainted in this way then younger people in this field will end up only talking to each other.
- oblio 2y ago
- deleted 2y ago[deleted]
- darkhorn 2y agoWhy they don't use Windows' own anti-virus?
- Nemo_bis 2y agoThat's called [Microsoft Defender for Endpoint](https://learn.microsoft.com/en-us/defender-endpoint/ https://learn.microsoft.com/en-us/defender-endpoint/), which is used even on Linux servers in big corporations. (Largely because it's the easiest way to complete box ticking exercises with Windows servers: once you have it, it's easy to decide to extend it to non-Windows machines as well.) The binary self-upgrades and runs in highly privileged mode, so it might not be immune from the kind of failure CrowdStrike had here. Though apparently there's at least a way to use a local mirror so you have some control on the updates: https://learn.microsoft.com/en-us/defender-endpoint/linux-support-offline-security-intelligence-update https://learn.microsoft.com/en-us/defender-endpoint/linux-su...
- pfortuny 2y agoThis is the Irish potato famine (essentially due to the farming of a single species of potato) equivalent in IT infrastructure: a single vendor.
- mindcrash 2y agoThis has all the hallmarks of a SSCA (Software Supply Chain Attack). Either that or Crowdstrike is testing critical software meddling in ring zero so poorly, causing crashes and bootloops out in the wild on 100% of the deployments, that they need to get sued out of existence. I hope for their sake its the former.
- felipesabino 2y agoDoes anyone know how to proceed if I do not have administrator level access to the computer? I do not have access to c:\windows\system32\drivers\crowdstrike folder to delete the corrupted .sys file I was able to boot on recovery mode with network, after waiting 30 min, I rebooted and BSOD persisted. Are there other alternatives on how to recover?
- victor9000 2y agoCreate a bootable USB stick from Ubuntu, then see if you can mount the windows drive and delete the file that way.
- einpoklum 2y agoIf I weren't an atheist I would say this is god's punishment for installing malware on your employees' machines, on one hand, and for being a spineless patsy for management by letting them install that crap on your work machine.
- jonplackett 2y agoHow do so many super critical things rely on… windows? I wouldn’t trust windows to run a laptop reliably but here it is running prettty ucy everything. I guess that’s why they need crowdstrike.
- jonplackett 2y agoAll the crazy people banging the drum for war with Russia and/or China. Imagine what our IT systems would look like with someone _intentionally_ messing with them.
- awahab92 2y agoclosed source will always fuck you in the ass
- zx10rse 2y agoThere is hardly a better time if you write software to watch - "The Mess We're In" by Joe Armstrong - https://www.youtube.com/watch?v=lKXe3HUG2l4 https://www.youtube.com/watch?v=lKXe3HUG2l4 I am not sure in which one of his talks he briefly mentioned that one of his concerns is that we are basically building a digital Alexandria library, and if it burns, well ... Even more devastating events like this will happen in the future. We stand on the shoulders of giants and yet we learned nothing.
- outside1234 2y agoTexas, where software goes to die. Or maybe that is where killer software is developed?
- dboreham 2y agoNow things are serious: I can't place a mobile order at Starbucks.
- kelembu 2y agoIs there a way to estimate number of affected devices? 10 million? 100 million?
- photonbeam 2y agoIt should be obvious to everyone now that kernel extensions for ‘security’ is not worth it
- wufufufu 2y agoAnyone have a technical writeup of the actual bug? I'm trying to explain how this could happen to people who think this is related to AI or cyber attacks. What happened to the QA testing, staggered rollouts, feature flags, etc.? It's really this easy to cause a boot loop? To me, BSOD indicates kernel level errors, which I assume Crowdstrike would be able to cause because it has root access due to being a security application. And because it's boot-looping, there's not a way to automatically push out updates?
- nrr 2y agoI don't have a technical writeup to offer, but your assessment around the BSOD seems correct enough. Without having an affected machine but knowing how NT loads drivers like this, I'd hazard a guess that the OS likely isn't even getting to the point where smss.exe starts before the kernel bugchecks. This means no userspace, which almost certainly means no hope of remotely remediating the problem.
- rtkwe 2y agoI'd hate to be on Microsoft's teams today. They're catching a lot of stray blame for this in the public eye where it's entirely not their fault.
- _hcuq 2y agoOk... Would a Linux based infrastructure be more resilient. Does Linux require Crowdstrike style AV software?
- tryauuum 2y agotechnically no os requires such software, it's people who make a decision to run it Someone in comments described crowd strike bringing down their fleet of Linux servers in April
- sytelus 2y agoCloudStrike had managed to invade into StarBucks IT. All of the online order taking systems are down.
- exabrial 2y agoWhy are people still using Windows?
- OWMYT 2y ago[dead]
- OWMYT 2y ago[dead]
- sagebird 2y agoIt is interesting that operating systems exist for server applications at all. What is the problem they are solving? What is the difference between what an operating system contains and can do and what you need it to do? Why would I want to rent a server to run a program that performs a task, and also have the same system performing extra tasks - like intrusion detection, intrusion detection software updates, etc. I just don't understand why compiled program that has enough disk and memory would ever be asked to restart for a random fucking reason having nothing to do with the task at hand. It seems like the architecture of server software is not created intelligently.
- banku_brougham 2y agoThe IT security chief at my co (paraphrasing): >talked to pres of Crowdstrike. His forthrightnes was refreshing. He said “We got it wrong.” >They are working with Microsoft to understand why this happened. Pretty much the message minus even more boilerplate talk.
- wojo1206 2y agoWhy Crowdstrike doesn't follow standard deployment strategies such as canary or rolling? Gradual update would uncover this bug before reaching critical mass. Doing all-at-once update is unacceptable to critical systems.
- nineteen999 2y agoSpeaking as somebody who manages a large piece of a 911 style system for first responders and has done so for 10 years (and is not affected by this outage) - this is why we do not allow third parties to push live updates to our systems. It's unfortunate, the ambulances are still running in our area of responsibility, but it's highly likely that the hospitals they are delivering patients to are in absolute chaos.
- downrightmike 2y agoThis is what happens when you treat IT as a cost center.
- golemiprague 2y ago[dead]
- ijidak 2y agoHow do you patch software causing a BSOD? It seems like a chicken and egg problem. I ran a team that developed a remote agent, and this was my nightmare scenario.
- grumpyprole 2y agoHopefully now people might wake up to the idea that these tech monopolies are not leading to safe, secure and reliable systems. They will wonder how a third party component could cause such breakage. I expect many will be calling for regulation.
- badgersnake 2y agoI don’t really understand why AV updates aren’t tested before being pushed out to critical systems and I don’t understand why every system would run the same AV. But also I don’t understand why this corporate garbageware is still a thing in 2024 when it adds so little value.
- xyst 2y agowhy the fuck is our critical infrastructure running on WINDOWS. Fuck the sad state of IT. CIOs and CTOs across the board need to be fired and held accountable for their shitty decisions in these industries. yes CRWD is a shitty company but seems they are a "necessity" by some stupid audit/regulatory board that oversees these industries. But at the end of the day, these CIOs/CTOs are completely fucking clueless as to the exact functions this software does on a regular basis. A few minions might raise an issue but they stupidly ignore them because "rEgUlAtOrY aUdIt rEqUiReS iT!1!"
- lnrd 2y agoThe OS doesn't matter, the question should be why is critical infrastructure online and allowed to receive OTA updates from third parties.
- teej 2y agoThe OS absolutely matters
- deleted 2y ago[deleted]
- sixothree 2y agoThe alternatives aren't in a position fill the roles needed for the tasks at hand.
- rbanffy 2y agoWhat are the tasks in question?
- jeremycarter 2y agoThis is true. Linux large fleet management is still missing some features large enterprises demand. Do they need all those features, idk, but they demand them if they're switching from Windows.
- 2y ago
- wolfspaw 2y ago(FORCE) Pusheedd to Prod on FRIDAYY -- Burneeeddd by its Sins
- 2OEH8eoCRo0 2y agoThey should be sued into bankruptcy
- fargle 2y agosuch stupidity. our $$$ corporate geniuses mandate multiple so-called security software which is: - unaccountable black boxes - of questionable, and un-auditable, quality - requires kernel modules, drivers, LocalSystem, root access, etc. - updates at random times with no testing - download these updates from where? and immediately trust and run that code at high privilege. using unaccountable-black-box crypto to secure it. - all have known patterns of bad performance, bugs, and generally poor quality all in the name of security. let's buy multiple "solutions" and widely deploy them to protect us from one boogeyman, or at least the shiny advertisements say. while punching all sorts of serious other holes in security. why even look for a Windows ZeroDay when we can look for a McAfee or Crowdstrike zero day?
- mensetmanusman 2y agoStopped by a gas station in rural Wisconsin leaving from MSP. Thank God we were on a full tank when we left, nothing was operational except the bathrooms (which is why we stopped). I left thinking about how anti-anti-fragile our systems have become. Maybe we should force cash operations…
- accra4rx 2y agoNever heard mainframe going down
- GabeIsko 2y agoSo - what is the lesson learned? The only clear message for me is that critical programs that also demand kernel level access maybe shouldn't update themselves.
- mikewarot 2y agoWhy are we still running ANY operating systems based on Ambient Authority, as part of our infrastructure? DoD shouldn't have given up on MULTICS. That premature optimization is going to sink the US and the Free World. Personally, I'm still waiting for Genode to be my daily driver.
- jpgvm 2y agoIf you run RATs like these on your machines then I'm sorry, this is just a case of fucking around and finding out. Just don't do it. Windows Defender is a thing, it does just fine. For everything else there is least-privilege and group policy.
- watersb 2y agoI wonder how unknown (yet) malware will be wiped out or enabled before this is over.
- ckemere 2y agoIt seems that an unexplored weirdness here is the prevalence of virtual Windows in the medical world. It seems that this has approach has become commonplace for HIPAA reasons (though it's unclear that it makes the world better versus using secure applications to handle HIPAA data). In the case of this Crowdstrike outage, one would think that virtual machines would simplify getting things up and running again, but instead there seems to be just the opposite going on, where lack of hardware access is limiting restoring them. Any insight from those affected?
- Group_B 2y agofun times...
- smcleod 2y agoI mean... installing what is essentially a 3rd party enterprise rootkit that not only has root access to all files and network activity but also a self-update mechanism ... who could have seen this coming?
- gpderetta 2y agoThe machine stops.
- tigerlily 2y agoMeanwhile the linux desktop just keeps on truckin'.
- Bluestein 2y agoI think we have reached and inflection point. I mean we have to make an inflection point out of this.- This outage represents more than just a temporary disruption in service; it's a black swan célèbre of the perilous state of our current technological landscape. This incident must be seen as an inflection point, a moment where we collectively decide to no longer tolerate the erosion of craftsmanship, excellence, and accountability that I feel we've been seeing all over the place. All over critical places.- Who are we to make this demand? Most likely technologists, managers, specialists, and concerned citizens with the expertise and insight to recognize the dangers inherent in our increasingly careless approach to ... many things, but, particularly technology. Who is to uphold the standards that ensure the safety, reliability, and integrity of the systems that underpin modern life? Government? Historically, the call for accountability and excellence is not new. From Socrates to the industrial revolutions, humanity has periodically grappled with the balance between progress and prudence. People have seen - and complained about - life going to hell, downhill, fast, in a hand basket without brakes since at least Socrates.- Yet, today’s technological failures have unprecedented potential for harm. The Crowdsource outage killed, halted businesses, and posed serious risks to safety—consequences that were almost unthinkable in previous eras. This isn't merely a technical failure; it’s a societal one, revealing a disregard for foundational principles of quality and responsibility. Craftsmanship. Care and pride in one's work.- Part of the problem lies in the systemic undervaluation of excellence. In pursuit of speed and profit uber alles. Many companies have forsaken rigorous testing, comprehensive risk assessments, and robust security measures. The very basics of engineering discipline—redundancy, fault tolerance, and continuous improvement—are being sacrificed. This negligence is not just unprofessional; it’s dangerous. As this outage has shown, the repercussions are not confined to the digital realm but spill over into the physical world, affecting real lives. As it always has. But never before have the actions of so few "perennial interns" affected so many.- This is a clarion call for all of us with the knowledge and passion to stand up and insist on change. Holding companies accountable, beginning with those directly responsible for the most recent failures.- Yet, it must go beyond punitive measures. We need a cultural shift that re-emphasizes the value of craftsmanship in technology. Educational institutions, professional organizations, and regulatory bodies must collaborate to instill and enforce higher standards. Otherwise, lacking that, we must enforce them ourselves. Even if we only reach ourselves in that commitment.- Perhaps we need more interdisciplinary dialogue. Technological excellence does not exist in a vacuum. It requires input from ethical philosophers, sociologists, legal experts. Anybody willing and able to think these things through.- The ramifications of neglecting these responsibilities are clear and severe. The fallout from technological failures can be catastrophic, extending well beyond financial losses to endanger lives and societal stability. We must therefore approach our work with the gravity it deserves, understanding that excellence is not an optional extra but an essential quality sine qua non in certain fields.- We really need to make this be an actual tuning point, and not just another Wikipedia page.-
- ok123456 2y agoMake a live CD Linux image that mounts the NTFS drives, locates the Windows directories from the bootloader, and deletes the file. Also, you can mount BitLocker partitions from Linux iirc. If it encounters a BitLocker partition, have it read a text file of possible keys off the USB drive.
- clarity20 2y agoHas any thread had 3000 comments before?
- Cyphase 2y agoThis story is about to break into the top 15 of upvoted stories on HN, but it already seems safely within the top 10 by number of comments.
- type0 2y agovideo summary from fireship: https://www.youtube.com/watch?v=4yDm6xNeYas https://www.youtube.com/watch?v=4yDm6xNeYas
- grigy 2y agoI could not imagine so many critical systems run on Windows.
- hughw 2y agoSo, if CrowdStrike licenses didn't say "We're responsible for nothing" and if all affected users sued them, they'd be worth negative 90 trillion dollars or so right now. iow out of business. I can understand the frustration their customers feel. But how could a software company ever bear liability for all the possible damage they can cause with their software? If they built CrowdStrike to space mission standards nobody could afford it.
- JSDevOps 2y agoRemember, there's someone out there right now, without irony, suggesting that AI can fix this. There's someone else scratching their head, wondering why AI hasn't fixed this yet. And there's someone doing a three-week bootcamp in AI, convinced that AI will fix this. I’m not sure which is worse
- beepbooptheory 2y agoThe saddest one is definitely the bootcamp one.
- 1oooqooq 2y agowhen even jsDevOpsv can see the king is naked....
- sys32768 2y agoSMB here. Just spent a nine hour day fixing this. We had two machines that after a couple of reboots just came back up fine. We were trialing CrowdStrike and about to purchase next week. If their rep doesn't offer us at least half off, we are going with Sentinel One which was half the price of CS already. The incompetence that allowed this is baffling to me. I assumed with their billions of dollars they'd have tiers of virtual systems to test updates with. I remember this happening once with Sophos where it gobbled up Windows system files. If you had set to Delete instead of Quarantine, you were toast.
- troad 2y ago> We were trialing CrowdStrike and about to purchase next week. If their rep doesn't offer us at least half off, we are going with Sentinel One which was half the price of CS already. You’re still considering them after all this?
- integricho 2y agoIt is baffling to me that you are still considering them.
- utkarsh858 2y agoAdding a comment to make this the most commented piece on hackernews and hence highlight the bad impact a bug can make on lives founded on IT.
- tammer 2y agoWhen I saw 'Global IT Outage' trending I assumed it was another major cloud service failure. Obviously this has far wider impact because of the need for intervention on individual endpoints. The irony is dawning on me that for much of the recent computing era we've developed defenses against massive endpoint outages (worms, etc.) and one of them is now inadvertently reproducing the exact problem we had mostly eradicated.
- unixhero 2y ago4 hour delay at the airport in Los Cabos. At least they have tacos!
- SoftMachine 2y agoAm I supposed to use some AI bot to summarize all this shit? Ain't no one got time to read 3000+ comments. Any good links?
- hansvm 2y agoRandom strangers running unknown, untrusted code on your computers is the worst. It's a good thing we patched that security flaw by letting the _right_ random strangers run unknown, untrusted code on our computers. As something of a friendly reminder, it was Microsoft this time, but it's a matter of "when" not "if" till every other OS with that flavor of security theatre is similarly afflicted (and it happens much more frequently when you consider the normal consequences of a company owning the device you paid for -- kicked out of email forever, ads intruding into basic system functions, paid-in-full device eventually requires a subscription, ...). Be cautious with automatic updates.
- elchief 2y agomaybe this was just an enormous distraction while Spetssvyaz did a bunch of fun stuff
- type0 2y agoclownStrike does make cool commercials though https://www.youtube.com/watch?v=8gatiYW5kWs https://www.youtube.com/watch?v=8gatiYW5kWs https://www.youtube.com/watch?v=p4OGQQPMiXQ https://www.youtube.com/watch?v=p4OGQQPMiXQ
- TexanFeller 2y agoA heuristic that has served me well for years is that anyone who uses the word “cybersecurity” is likely incompetent and should be treated with suspicion. My first encounter with CrowdStrike was overwhelmingly negative. I was wondering why for the last couple weeks my laptop slowed to a crawl for 1-4 hours on most days. In the process list I eventually found CrowdStrike using massive amounts of disk i/o, enough to double my compile times even with a nice SSD. Then they started installing it on servers in prod, I guess because our cloud bill wasn’t high enough.
- Laforet 2y agoCyber- is pretty much a code prefix for anything targeted at the public sector. I too see it as a kind of dirty word TBH.
- gukov 2y ago"Cyber," used on its own, is the worst of them all.
- photochemsyn 2y agoIt rather looks like Crowdstrike marketed heavily to corporate executives using a horror story about the bad IT tech guy who would exfiltrate all their data if they didn't give Crowdstrike universal access at the kernel level to all their machines...? It seems more aimed at monitoring the employees of a corporation for insider threats than for defense against APT actors.
- neverrroot 2y agoThe employees is a very important attack vector, we had multiple incidents, after they downloaded the wrong kind of stuff.
- nbtm 2y agoI heard all Windows PCs at the University of New South Wales were also boot-looping.
- zoom6628 2y agoBy way of a data point for everyone else I live in HongKong and haven't seen any of this level of disruption yet. I also was in Shenzhen China yesterday, probably the words highest density of Win95 machines, and everything was fine. At home we have only one old laptop on win10 that only gets opened when the 8yo gets windows homework - otherwise it's MacOs and Linux on all laptops, desktops and SBCs. If I see some news I will update this comment.
- godelmachine 2y agoI wonder what happens to the engineer who deployed this patch.
- jiggawatts 2y agoWhile initially everyone blamed Microsoft and then quickly pointed the finger at CrowdStrike, I'd like to call out Microsoft especially their Azure division for making the recovery process unnecessarily difficult. 1) A key recovery step requires a snapshot to be take of the disk. The Portal GUI is basically locking up, so scripting is the only way to do this for thousands of VMs. This command is undocumented and has random combinations of strings as inputs that should be enums. Tab-complete doesn't work! See: https://learn.microsoft.com/en-us/powershell/module/az.compute/new-azsnapshotconfig?view=azps-12.0.0 https://learn.microsoft.com/en-us/powershell/module/az.compu... E.g.: What are the accepted values for the -CreateOption parameter? Who knows! Good luck using this in a hurry. No stress, just apply it to a production database server at 1 am in the morning. 2) There has been a long-standing bug where VMs can't have their OS disk swapped out unless the replacement disk matches its properties exactly. For comparison, VMware vSphere has no such restrictions. 3) It's basically impossible to get to the recovery consoles of VMs, especially VMs stuck in reboot loops. The serial console output is buggy, often filled with gibberish, and doesn't scroll back far enough to be useful. Boot diagnostics is an optional feature for "reasons". Etc.. 4) It's absurdly difficult to get a flat list of all "down" VMs across many subscriptions or resource groups. Again, compare with VMware vSphere where this is trivial. Instead of a simple portal dashboard / view, you have to write this monstrous Resource Graph query: Resources | where type =~ 'microsoft.compute/virtualmachines' | project subscriptionId, resourceGroup, Id = tolower(id), PowerState = tostring( properties.extended.instanceView.powerState.code) | join kind=leftouter ( HealthResources | where type =~ 'microsoft.resourcehealth/availabilitystatuses' | where tostring(properties.targetResourceType) =~ 'microsoft.compute/virtualmachines' | project targetResourceId = tolower(tostring(properties.targetResourceId)), AvailabilityState = tostring(properties.availabilityState)) on $left.Id == $right.targetResourceId | project-away targetResourceId | where PowerState != 'PowerState/deallocated' | where AvailabilityState != 'Available'
- vpshastry 2y agoDon’t they have canary deployments? Such huge updates happen al at once?
- cookiengineer 2y agoI'm a little late to the party, but I've uploaded my source codes to GitHub in case anyone needs a more convenient tool to deploy/execute on running machines and/or needs something fast on USB flash drives to run around the office: https://github.com/cookiengineer/fix-crowdstrike-bsod https://github.com/cookiengineer/fix-crowdstrike-bsod Releases section contains prebuilt binaries, but of course, I always recommend to check the source and then build it yourself.
- miffy900 2y agoI'm sure you mean well, but it's not going to be most programmers or devs who will need to apply a fix for this, it'll be sysadmin/network/SREs who'll be doing this and they're not going to download Go to build this random github code repo. Because it affects only Windows systems, it'll be way better writing a bat or powershell script that can non-programmers can read and comprehend before they execute anything in production/live systems.
- 1oooqooq 2y agoWhy aren't we upvoting a list of alternatives to CrowdStrike here?
- dang 2y agoAll: there are over 3000 comments in this thread. If you want to read them all, click More at the bottom of each page, or like this: https://news.ycombinator.com/item?id=41002195&p=2 https://news.ycombinator.com/item?id=41002195&p=2 https://news.ycombinator.com/item?id=41002195&p=3 https://news.ycombinator.com/item?id=41002195&p=3 https://news.ycombinator.com/item?id=41002195&p=4 https://news.ycombinator.com/item?id=41002195&p=4 (...etc.)
- uptownfunk 2y agoThe correct solution is to have IT force push updates only when they deem fit (after they have tested internally on some ghost machines).
- rkagerer 2y agoThis is why I don't like fully automatic updates. I prefer having control over the "deploy" button for the ability to time it when I can tolerate downtime. In mission-critical production systems all updates should go through test staging pipelines that my team controls, not a vendor. Broken updates have cause far more havoc than being a few hours or even days late on a so-called critical patch.
- TeeWEE 2y agoEven if you deploy manually all at once: you have the same problem. A solution is slow rollout. Not a manual deploy button
- rkagerer 2y agoIt's not the same problem at all. Troubleshooting an issue like this when I have the time and am prepared for a potential outage (with human resources hot and standing by for immediate action) is VASTLY different than encountering it the evening before some critical deadline for a multi-million dollar project (as Murphy's Law will be sure to have it). When I have control over the deployment of updates I can push them through my own QA environment first, to uncover many of these kinds of issues before they hit production. Vendors pushing them out on their whim leaves me subject to whatever fast and loose practice they use and prevents me from being able to properly manage my own infrastructure. A slow rollout certainly helps but doesn't satisfy the kind of 9's I demand in the environments I care for.
- TowerTall 2y agoMicrosoft to give Vista kernel access to security firms (2006) https://arstechnica.com/information-technology/2006/10/7998/ https://arstechnica.com/information-technology/2006/10/7998/
- 256_ 2y agoI used to laugh at Dijkstra's idea that all code should be mathematically proven correct. I thought of it as a laughable idea from yet another out-of-touch mathematician. I suppose true genius is seldom understood within someone's lifetime.
- belter 2y ago> I used to laugh at Dijkstra's idea that all code should be mathematically proven correct. That would not help for this outage.
- beardyw 2y agoMicrosoft to give Vista kernel access to security firms (2006) | Hacker News https://news.ycombinator.com/item?id=41014426 https://news.ycombinator.com/item?id=41014426
- cranberryturkey 2y agoSomeone accidentally shut down the planet with a code push -- rofl
- kwhitefoot 2y agoCan someone explain to me why such systems need anti-virus in the first place? Windows has pretty good facilities for locking down the system so that ordinary users, even those with local admin rights, cannot run or install unauthorised code so if nothing can get in why would the system need checking for viruses? So why do most companies not lock down their machines?
- 1oooqooq 2y agoeasier to show a paid bill than to show true due diligence to your insurance when you're hit with ransomware. that's the whole CS business model.
- tomthumb 2y agoSomeone on X has shared the kernel stack trace of the crash The faulting driver in the stack trace was csagent.sys. Now, Crowdstrike has got two mini filter drivers registered with Microsoft (for signing and allocation of altitude). 1) csagent.sys - Altitude (321410) This altitude falls within the range for Anti-Virus filters. 2) im.sys - Altitude (80680) This altitude falls within the range for access control drivers. So, it is clear that the driver causing the crash is their AV driver, csagent.sys. The workaround that CrowdStrike has given is to delete C-00000291*.sys files from the directory: C:\Windows\System32\Drivers\CrowdStrike\ These files being suggested to be deleted are not driver files (.sys files) but probably some kind of virus definition database files. The reason they name these files with the .sys extension is possibly to leverage Windows System File Checker tool's ability to restore back deleted system files. This seems to be a workaround and the actual fix might be done in their driver, csagent.sys and the fix will be rolled out later. Anyone having access a Falcon endpoint might see a change in the timestamp of the driver csagent.sys when the actual fix rolls out.
- rldjbpin 2y agoit is humbling (and lowkey reassuring?) to know that not all large players use the absolute cutting edge approaches in their workflow. it seems and i hope that after all is said and done there is no major life-threatening consequence of this debacle. at the same time, heart goes out to the dev who pushed the troubling code. very easy to point at them or the team's processes, but we need to introspect at our own setup and also recognize that not all of us work in crucial systems like this.
- attentive 2y agothe funny thing it's often labeled "Microsoft IT outage" - theguardian.com as example
- aiman3 2y ago[flagged]
- vlod 2y agoI haven't heard ask this, but would this have happened on linux. Obviously not many people run virus s/w, but would something similar like this have caused this? Are there any protections to prevent repeating reboots?
- peanut-walrus 2y agoI feel we are at a point in the evolution of our digital society where relying on general purpose OS-s is just not an option when moving forward.
- hahamaster 2y agoSomeone at CrowdStrike got fired for this. I'm curious to know who this person is.
- omnee 2y agoThis event raises the question: What is the liability of Crowdstrike given its erroneous update caused the meltdown, and the impact certainly had negative personal or business outcomes globally. See for example 6000 flights cancelled or the many statements posted here regarding it negatively impacting healthcare and other businesses.
- 1oooqooq 2y agowe are bound to see the YouTube ads equivalent of late night spot ads for lawyers with accelerated audio "have you lost someone to the 2024, 2025 or 2029 crowdstrike global hospital outages? if so you may be entitled to compensation. DM law5237 on X to find more"
- ngneer 2y agoThe whole thing needs to be redesigned, so that antivirus and EDR solutions do not require such high privilege. We need a high-performance way for a possibly privileged service to export all the data that is needed for a decision, and then let the AV/EDR do its thing. If the AV/EDR is broken by an update, fine. At least the system won't go down.
- ngneer 2y agoHave we failed as an industry?
- ngneer 2y agoAllow me to give a different, information-theoretic, perspective. How much damage can flipping a single bit cause? How much damage can altering two bits cause? The fanout is a robustness measure on systems. If we can control the fanout we increase reliability. If all it takes is a handful of bits in a 3rd party update to kill IT infrastructure, we are doing it wrong.
- __MatrixMan__ 2y agoAre you suggesting that a 3kb update be tested 3k times to assess the impact of each possible bit flip, and 9M times for the impact of each possible pair bit flips? Because I think that's effort better spent in other ways.
- ngneer 2y agoNot remotely. I am aware of the state space explosion and the difficulty with brute forcing the testing. I am suggesting that the damage a broken antivirus update can do should be restricted.
- tonymet 2y agoQuestion : was this update delivered by Crowdstrike’s update agent or Windows Update ?
- j-bos 2y agoMy read is that Crowdstrike's update agent downloaded new security threat definitions and those definitions exposed a bug in the existing Crowdstrike drivers, causing the disaster.
- tartavull 2y agoThe front page of https://www.accel.com/ https://www.accel.com/ "Fail Fast. Evolve Faster"
- mbrumlow 2y ago> We have collaborated with Intel to remediate affected hosts remotely using Intel vPro and with Active Management Technology. This worries me. Does this mean intel has access to remotely access my machine?!?!
- anvuong 2y agoWhat is with this paranoia on HN? 5s Google says that vPro is the Remote management platform offered by Intel for business, basically a fleet management tool, which is typical for corp
- mbrumlow 2y agoYah. But the way it was worded is that a 3rd party company who goofed was able to go to another company and ask for access to it. I assume that some sort of configuration and opt in step would need to be had. If this had been done prior to the event it seems like this would have been the first thing to do, not a Hail Mary after being down.
- ai4ever 2y agowhat would be funny is if crowdstrike demanded ransom from their castomars. security is a great business - you play on people's fears, your product does not have to deliver the goods. like the lock maker, you sell a lock, the thief breaks it, but it is not your problem, and you sell a bigger badder lock the next year which promptly gets broken. as a business, you dont have any consequences for how your product works or doesnt work, what a great business to be in !!
- meta-level 2y agoWow, CrowdStrike made it to #6 of all time HN threads by now.. https://hn.algolia.com/?q= https://hn.algolia.com/?q=
- meetpateltech 2y agoCrowdStrike’s faulty update crashed 8.5 million Windows devices, says Microsoft https://www.theverge.com/2024/7/20/24202527/crowdstrike-microsoft-windows-bsod-outage https://www.theverge.com/2024/7/20/24202527/crowdstrike-micr...
- Yawrehto 2y agoOh wow, this is #5 for all time already, beating out Steve Jobs.
- porsche4664 2y ago[flagged]
- generatedtruth 2y ago[dead]
- retrocryptid 2y agoMy Commodore 64 never gave me a blue screen of death and my Atari ST never lapsed into a hardcore boot loop. just sayin'
- deleted 2y ago[deleted]
- peter_d_sherman 2y agoGreenspun's tenth rule: "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp." (https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule) Arthur C. Clarke's third law: "Any sufficiently advanced technology is indistinguishable from magic." (https://en.wikipedia.org/wiki/Clarke%27s_three_laws#:~:text=Any%20sufficiently%20advanced%20technology%20is%20indistinguishable%20from%20magic https://en.wikipedia.org/wiki/Clarke%27s_three_laws#:~:text=....) Apparently we now have the following, as well: "Any sufficiently bad software update is indistinguishable from a cyberattack…" (https://x.com/leighhoneywell/status/1814278230704111792 https://x.com/leighhoneywell/status/1814278230704111792)
- minhoryang 2y agoCan we find an uptime(availability) graph for the CrowdStrike agent? Don't you think this graph should be included in the postmortem?
- Haroldteresa 2y ago[dead]
- nurettin 2y agoIf I were a cloud vendor, I would provide a "CrowdStrike recovery" button which queues the recovery image and restores the system for the entire project. Why didn't hetzner, linode, DO, gcp, aws do something like this? Why leave people to their devices? Isn't this a basic application of centralization? It feels to me like this should be easier than managing your data center.