11 ms·
We are a major CS client, with 50k windows-based endpoints or so. All down. There exists a workaround but CS does not make it clear whether this means running
by aenis 2y ago
We are a major CS client, with 50k windows-based endpoints or so. All down.
There exists a workaround but CS does not make it clear whether this means running without protection or not. (The workaround does get the windows boxes unstuck from the boot loop, but they do appear offline in the CS host management console - which of course may have many reasons).
- kurthr 2y agoYeah, the simple renaming .sys files in safe mode does seem like it would inhibit protection.
- Scoundreller 2y agoRenaming it .old or whatever would be what a sane person does. They recommended deleting the thing: https://news.ycombinator.com/item?id=41002199 https://news.ycombinator.com/item?id=41002199
- saagarjha 2y agoWhile you're at it probably delete the rest of the software. Then cancel your contract.
- InsideOutSanta 2y agoThen sue the shit out of them.
- hello_moto 2y agoThen your hardware becomes unprotected, congratulations, you won Cybersecurity award of the year.
- saagarjha 2y agoThere are more options than "you are using Crowdstrike" and "you have no protection".
- hello_moto 2y agoYou're absolutely right, there are more options. Let's say you're a CISO and it's your task to evaluate Cybersecurity solutions to purchase and implement. You go out there and found out that there are multiple organizations that tests (simulate attacks) the EDR capabilities of these Vendors periodically and published grades of these Vendors. You found the top 5 to narrow down your selections and you pitted them in PoC which consists of attack simulations and end-to-end solutions (that's the Response part of EDR). The winner gets the contract. Unless there are tie-breakers... PS: I heard others (and read) said that CS was best-in-class which suggested that they probably won PoC and received high grades from those independent Organizations.
- deleted 2y ago[deleted]
- _8j50 2y agoNo, the userspace program will replace it with a good version.
- blinkersareon 2y agoI think my company has more than 300k+ machines down right now :) SLAs will be breached anyway
- Reason077 2y agoWindows IT admins of the world, now is your time. This is what you've trained for. Everything else has led to this moment. Now, go and save the world!!
- 0xedd 2y agoYeah, manual GUI work. Like any good MS product.
- jylam 2y agoOr don't \o/
- ahazred8ta 2y ago"The world will look up and shout, "Save us!" and I'll whisper "No..." -- Rorschach
- randerson 2y agoOr "log a ticket!"
- mr_mitm 2y agoType in those Bitlocker recovery keys for as long as you can stay awake!
- heraldgeezer 2y ago[flagged]
- btbuildem 2y agoOr rather, go limp and demand to unionize!
- 2y ago
- easytiger 2y agoSurely i'ts not normal practice to allow patches to be rolled out without a staging/testing area on an estate of that size?
- usr1106 2y agoIt's incredibly bad practice, but it seems to be industry normal as we learned today.
- easytiger 2y agoI wonder if the move to hybrid estates (virtual + on prem + issued laptops etc) is the cause. Having worked in only on prem highly secure businesses no patches would be rolled out intra week without a testing cycle on a variety of hardware. I consider it genuinely insane to allow direct updated from vendors like this on large estates. If you are behind a corporate firewall there is also a limit to the impact of discovered security flaws and thus reduced urgency in their dissemination anyway.
- phatfish 2y agoMost IT departments would not be patching all their servers or clients at the same time when Microsoft release updates. This is a pretty well followed standard practice. For security software updates this is not a standard practice, I'm not even sure if you can configure a canary update group in these products? It is expected any updates are pushed ASAP. For an issue like this though Crowdstrike should be catching it with their internal testing. It feels like a problem their customers should not have to worry about.
- InsideOutSanta 2y agoThis is insane. The company I currently work for provides dinky forms for local cities and such, where the worst thing that could happen is that somebody will have to wait a day to get their license plates, and even we aren't this stupid. I feel like people should have to go to jail for this level of negligence.
- the_gipsy 2y agoDoes CS actually offer any real protection? I always thought it was just feel-good software, that Windows had caught up to separating permissions since after XP or so. Either one is lying/scamming, but which one?
- padjo 2y agoEvery company I’ve ever worked at has wound up having to install antivirus software to pass audits. The software only ever caused problems and never caught anything. But hey, we passed the audit so we’re good right?
- fvold 2y agoThe real scam is the audit. Many moons ago, I failed a "security audit" because `/sbin/iptables --append INPUT --in-interface lo --jump ACCEPT` "This leaves the interface completely unfiltered" Since then, I've not trusted any security expert until I've personally witnessed their competence.
- gonium 2y agoThe dreaded exposed loopback interface... I'm an (internal) auditor, and I see huge variations in competence. Not sure what to do about it, since most technical people don't want to be in an auditor role.
- irusensei 2y agoThe companies I had the displeasure of dealing with were basically run by mindless people with a shell script.
- l33t7332273 2y agoI agree completely. It makes me wonder if other engineering disciplines have this same competency issue.
- finaard 2y agoLong time ago I was working for a web hoster, and had to help customers operating web shops to pass audits required for credit card processing. Doing so regularly involved allowing additonal ciphers for SSL we deemed insecure, and undoing other configurations for hardening the system. Arguing about it is pointless - either you make your system more insecure, or you don't pass the audit. Typically we ended up configuring it in a way that we can easily toggle those two states, and reverted it back to a secure configuration once the customer got their certificate, and flipped it back to insecure when it was time to reapply for the certification.
- cm2187 2y agoDoes it require to physically go to each machine to fix it? Given the huge number of machines affected, it seems to me that if this is the case, this outage could last for days.
- ndsipa_pomu 2y agoThe workaround involves booting into Safe mode or Recovery environment, so I'd guess that's a personal visit to most machines unless you've got remote access to the console (e.g. KVM) The info is apparently behind here: https://supportportal.crowdstrike.com/s/article/Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19 https://supportportal.crowdstrike.com/s/article/Tech-Alert-W...
- rhim 2y agoThat's crazy, imagine you have thousands of office PCs that all have to be fixed by hand.
- jaggederest 2y agoPresumably at some point they'll be back to a state where they can boot to a network image, but that's going to be well down the pyramid of recovery. This is basically a "rebuild the world from scratch" exercise. I imagine even the out of band management services at e.g. Azure are running Windows and thus Crowdstrike.
- amlib 2y agoIt gets worse if your machines have bitlocker active, lots of typing required. And it gets even worse if your servers that store the bitlocker keys also have bitlocker active and are also held captive by crowstrike lol
- AgentME 2y agoI've already seen a few posts mentioning people running into worst-case issues like that. I wonder how many organizations are going to not be able to recover some or all of their existing systems.
- fudged71 2y agoA bunch of unprotected endpoints all at once on critical systems… what could possibly go wrong? Hope they role out a proper fix soon…
- CaliforniaKarl 2y agoThey did, around four hours ago.
- ta1243 2y agoA proper fix means that a failure like this causes you a headache, it doesn't close all your branches, or ground your planes, or stop operations in hospitals, or take your tv off air. You do that by ensuring a single point of failure, like virus definition updates, or an unexpected bug in software which hits on Jan 29th, or when leapseconds go backwards, can't affect all your machines at the same time. Yes it will be a pain if half your checkin desks are offline, but not as much as when they are all offline.
- fudged71 2y agoExcept this actually was an opportunity for malicious actors https://www.theregister.com/2024/07/19/cyber_criminals_quickly_exploit_crowdstrike/ https://www.theregister.com/2024/07/19/cyber_criminals_quick...
- drited 2y agoWow that's terrible. I'm curious as to whether your contract with them allows for meaningful compensation in an event like this or is it just limited to the price of the software?
- m-i-l 2y ago> "50k windows-based endpoints or so. All down." I'm a dev rather than infra guy, but I'm pretty sure everywhere I've worked which has a large server estate has always done rolling patch updates, i.e. over multiple days (if critical) or multiple weekends (if routine), not blast every single machine everywhere all at once.
- EdwardDiego 2y agoThis feels like an auto-update functionality. For something that's running in kernel space (presumably, if it can BSOD you?) Which is fucking terrifying.
- fredoralive 2y agoIf this comment tree: https://news.ycombinator.com/item?id=41003390 https://news.ycombinator.com/item?id=41003390 is correct, someone at Crowdstrike looked at their documented update staging process, slammed their beer down, and said: "Fuck it, let's test it in production", and just pushed it to everyone.
- averageRoyalty 2y agoEven if true, the orgs whose machines they are have the responsibility to validate patches.
- sofixa 2y agoThis is not a patch per se, it was Crowdstrike updating their virus definition or whatever it's called internal database. Such things are usually enabled by default to auto-update, because otherwise you lose a big part of the interest (if there's any) of running an antivirus.
- fredoralive 2y agoSurely their should be at least some staging on update files as well, to avoid the "oops, we accidentally blacklisted explorer.exe" type things (or, indeed, this)?
- dschuetz 2y agoAre you rolling out CS updates as is everywhere? Are you not testing any published updates immediately at least with some N-1 staging involved?
- heraldgeezer 2y ago[flagged]
- averageRoyalty 2y agoI don't mean this to be rude or as an attack, but do you just auto update without validation? This appears to be a clear fault from the companies where the buck stops - those who _use_ CS and should be validating patches from them and other vendors.
- sunbum 2y agoI'm pretty sure crowdstrike autoupdates, with 0 option to disable or manually rollout updates. Even worse people running N-1 and N-2 channels also seem to have been impacted by this.
- averageRoyalty 2y agoMy point stands then. If you're applying kernel grade patches on machines which you knowingly cannot disable or test, that's just simple negligence.
- rjmunro 2y agoI think it's probably not a kernel patch per se. I think it's something like an update to a data file that Crowdstrike considers low risk, but it turns out that the already-deployed kernel module has a bug that means it crashes when it reads this file.
- everybodyknows 2y agoWhich suggests the question: What's the current state of "fuzz testing" within the Crowdstrike dev org?
- aenis 2y agoApparently, CS and ZScaler can apply updates on their own and thats by design, with 0day patches expected to be deployed the minute they are announced.
- heraldgeezer 2y agoCS, S1, Zscaler etc auto updates and they have to. Thats the point of the product. If they dont get definitions they cannot protect.
- purpleblue 2y agoDo you need to manually fix all your windows boxes? Or is there a way to update it remotely?
- 1oooqooq 2y agostop the pandering. you know very well crowdstrike doesn't offer good protection to begin with! everyone pay for legal protection. after it happens you can show you did everything, which means nothing (well now this show even worse than nothing), by showing you paid them. if they tell you to disable everything, what does it change? they're still your blame shield. which is the reason you have cs. ... the only real feature anybody care is inventory control.
- hello_moto 2y agoQuite a few people in this thread disagree with you though.
- 1oooqooq 2y agoyou mean their career depend on them claiming ignorance.
- hello_moto 2y agoYou said Crowdstrike doesn't offer protection but there are plenty in this thread that suggested they actually do and seemed to be highly regarded at the field. Not sure who the ignoramus here...
- 1oooqooq 2y agofacts speak more than words. if you cared about protection you would be securing your system, not installing yet more things, specially one that now require you open up several other attack vectors. but i will never managed to make you see it.
- hello_moto 2y agoWriting software in the safest programming language to develop mission critical product deployed on the most secure and stable OS that the world depends on would be developer's wet dream. But it's just that... developer's wet dream.