10 ms·
Ask HN: Can anyone from Crowdstrike explain the back story?
Can anyone from crowdstrike or someone who knows what lead to this incident, how it got past QA etc... care to share?
That other thread with more than 3000 msgs may or may not have this info, hard to read...
- pcunite 2y agoAn internet connected kernel level driver, with a bricking update, that passed QA, that could also be rolled out via a non-staged method, at global scale, under 24hrs? That is unthinkable.
- intrasight 2y agoThe core issue? Using Wintel.
- lofaszvanitt 2y agoThe script of the film is being sold at this very moment. :D
- tlogan 2y agoPutting on my tin foil hat for a moment… I suspect the US government might have pressured them to push this update because they’ve found out that Crowdstrike’s system has either already been breached or has a significant zero-day security vulnerability. Now, taking off my tin foil hat…
- yalogin 2y agoUsually things like this can be explained by basic human incompetence or a misstep. Tin foils are rarely ever needed
- tlogan 2y agoThat is even scarier. Which means that they might already got hacked due to some other incompetence or misstep. But I 100% agree. There is a good book on that topic: Normal Accidents: Living with High-Risk Technologies by Charles Perrow
- rowanG077 2y agoMaybe companies will rethink handing over critical security to a company where multiple highly security critical mistakes compounded each other.
- CoastalCoder 2y agoI think the good news, if it's incompetence, is that they can greatly reduce the risk of recurrence by improved processes and tooling. I doubt there's much they could do about forcible government intervention.
- pdimitar 2y ago> I think the good news, if it's incompetence, is that they can greatly reduce the risk of recurrence by improved processes and tooling. Now let me give you the realistic news: every executive everywhere will just hand-wave this away with "it will not happen again anytime soon, or ever, right?" and be back to business as usual. The extreme cost-cutting and allowing non-technical people to rule us is what is getting us into these messes every time. And humanity in its totality seems almost completely unable to learn.
- MuffinFlavored 2y ago> That other thread with more than 3000 msgs may or may not have this info, hard to read... Feed it to an LLM to summarize /s
- mikequinlan 2y agoGemini says this: "This is an article about a Reddit thread discussing why a diesel repair shop had a system-wide failure. The conversation centers around why industrial equipment, like lifts and cranes, would be running Windows and be connected to a network. People debate the pros and cons of having internet-connected machinery, with some arguing for the benefits of remote monitoring and updates and others expressing concerns about security vulnerabilities. Many users point out that some critical systems, like elevators, could be controlled with simpler and more reliable solutions."
- qingcharles 2y agoPack it up, boys! Our work here is done...
- lobsterthief 2y agoMission Accomplished(tm)
- ToucanLoucan 2y agoI'm extremely for integration of smart features into home, business and industrial applications. However, and this is a great, great big HOWEVER, the smart features must always be an added feature, and absent the customer desiring to use them, or the ability to use them via technical issues/obsolescence, the equipment in question should degrade to a usable state. Perfect example: I own a bunch of iHome smart outlets that I use with Apple HomeKit. Got a whole box of the things off ebay years ago, paid about $6 each. Along with (obviously) being a smart home accessory that runs via HomeKit, they have a little button on the side you can push to toggle the power on and off. So, in the odd event that my network is being uncooperative and I need to turn a light off, I can do that very easily. Not as easily as I could with my phone, but still quite easily. I don't understand why industry is so hostile to this. It just makes sense to me: build in all the smart features you like, but if they're not working, just have the bloody thing operate like a normal... whatever. Microwave, lamp, etc.
- Retr0id 2y agoBy the way, Crowdstrike has shared "technical details", discussed here: https://news.ycombinator.com/item?id=41013198 https://news.ycombinator.com/item?id=41013198 Unfortunately it's rather lacking in both technicality and detail.
- Havoc 2y agoCan’t say I’m surprised. Every major org will at least attempt to get compensation so playing entirely open cards is maybe not ideal
- Kudos 2y ago> We will update our findings in the root cause analysis as the investigation progresses. Give them a minute.
- devwastaken 2y agoNope. Tech is specially protected by both sides of the isle. Why do you think there's no real or legally defined "engineering" happening? No red tape. No filter of quality, no accountability. We have to legislate out big tech and start holding them accountable. Prison time for execs and significant fines is the correct and effective method.
- giantg2 2y agoAny org using a tech product should have a team reviewing contracts. Many contracts have financial penalties for not meeting SLAs. That's not going to be in the standard terms of service, but rather in an enterprise contract. Any large company just accepting standard ToS for business critical software is not doing their due diligence.
- mikkom 2y agoMost likely legal is highly involved in any postings about the incident
- giantg2 2y agoIt doesn't have to be that technical. What I'm reading in there is that they had a logic flaw and their test cases nd testing processes aren't robust enough to catch major logic flaws. Not that surprising the way most orgs treat their testing.
- wkat4242 2y agoI'm sure Crowdstrike will not take kindly to their employees spilling the beans here, just saying... Especially a very high-profile issue like this that is likely to end up in the courts.
- JCharante 2y agoAnd it’s a publicly traded company that went down 10% on Friday due to these events. I hope no one is dumb enough to spill the beans just for some karma. Especially when you don’t know what the official findings are yet.
- crngefest 2y agoJust do it, what do you have to lose, that company is toast anyway
- lambdaone 2y agoI'd be surprised if they were toast. Big companies can fuck up with impunity - just look at Boeing. They can take a bit of an insurance hit, chalk it up to the cost of doing business, and pass the cost back to their customers in the long run.
- deleted 2y ago[deleted]
- plonk 2y agoIt’s only market manipulation if you have the intention of making the stock go down, right? If answering the OP is illegal, then no employee of any listed company can ever say something bad about it, which sounds too restrictive?
- rvz 2y ago> It’s only market manipulation if you have the intention of making the stock go down, right? Sounds like a great swing trade to take a long position in CRWD for retail.
- user3939382 2y agoMy guess is it’s (phased rollout) like all safety precautions. You can flout it and you’ll be fine most of the time, possibly for a while. It works until it doesn’t. In this case it’s a bit ironic since their whole business is mitigating risk.
- gizajob 2y agoI’d also say “no, they can’t” given crowdstrike is at risk of being litigated into non-existence right now. Also NDAs because security.
- panarky 2y ago> litigated into non-existence There's a long list of incredibly damaging fuckups by software companies, but I can't think of a single example of an existence-ending judgement from litigation. The biggest reason why billion-dollar software companies continue to release shitty and dangerous products is because they face zero legal liability for their negligence.
- oaiey 2y agoIn an industry which is based on ticking check boxes for auditors, relying on Crowd strike will auto untick the checkbox. The only hope for them is rebrand and starting from scratch.
- nothercastle 2y agoLet me introduce you to Ernest young. Still in business https://en.m.wikipedia.org/wiki/Ernst_%26_Young https://en.m.wikipedia.org/wiki/Ernst_%26_Young
- itsdrewmiller 2y agoBut are you familiar with Arth & Andersen? https://en.m.wikipedia.org/wiki/Arthur_Andersen https://en.m.wikipedia.org/wiki/Arthur_Andersen
- nothercastle 2y agoCockroaches always come back. https://www.businessinsider.com/arthur-andersen-name-makes-a-comeback-2014-9?amp https://www.businessinsider.com/arthur-andersen-name-makes-a...
- 2y ago
- supportengineer 2y agoI saw a post somewhere that said configuration data was added after the QE cycle, but before the broad push to the world
- discreteevent 2y agoAny data that is interpreted is code. So the short answer is that they deployed code without testing it.
- brians 2y agoI like that framing! Alternately, they did not ever test the ability of their code to accept arbitrary configurations. That totally could have been done, allowing fast config pushes. But it wasn’t. So as a reliability engineer, I’d point at that change: configs were originally expected to be tested, so the parser was only tested to be sure it showed issues on pre-release testing. Later, they started shipping configs faster than their release cycle, but didn’t check to requalify the parser for this new requirement. Behind that, I’d look at the engineering and product culture that had that happen. Is there a list of what’s been tested for what purpose? Was the expectation that configs get tested like software written down someplace, and do the roles who scheduled that config release read that place? Who is organizationally set up to notice this, and why is it a director-level IC straight out of xkcd 2347?
- jzl 2y agoThis post has some info but seems more from a customer: https://news.ycombinator.com/item?id=41003390 https://news.ycombinator.com/item?id=41003390
- jenscow 2y ago> They have a staging system which is supposed to give clients control over this but they pissed over everyone's staging and rules and just pushed this to production.
- jordanpg 2y agoI expect we will see detailed forensics published by various third parties in the coming days and weeks. As to what CS itself will publish, that remains to be seen.
- markus_zhang 2y agoJust curious is there any RE done on CS itself? Like what it does and how it achieves the objectives.
- talkingtab 2y agoThis is a simple problem to understand and solve. Corporations need to buy a license from CrowdStrikeCrowdStrike.com. This allows large corporations that do not know crowdstrike is a security nightmare built on top of the microsoft security nightmare to assure themselves they are now secure.
- notaharvardmba 2y agoThey could bring in Xzibit for the commercial: “Yo dawg, we heard you liked Crowdstrike so we got you some Crowdstrike for your Crowdstrike!”
- breadwinner 2y agoTo people who say this is not a Microsoft issue... it absolutely is a Microsoft issue. Microsoft allowed third parties to muck with the Windows kernel in a way that makes the computer unbootable. How is that not a Microsoft issue? Apple has a vetting process before they will allow an app to be added to their app store. Why doesn't Microsoft have a vetting process before allowing a third party to mess with the Windows kernel? Does Crowdstrike have SOC2 or some other certification to make sure they are following secure practices, with third-party verification that they are following their documented practices? If not, why not? Why doesn't Microsoft require that? It is clear that the status quo can't continue. Think about the 911 calls that didn't get answered and the surgeries that had to be postponed. How many people lost their lives because of this? How does the industry make sure this doesn't happen again? Just rely on Crowdstrike to get their act together? Is it enough to trust them to do so?
- deleted 2y ago[deleted]
- efitz 2y agoThey absolutely do, long before Apple did: WHQL (pronounced “wickle”). https://blogs.windows.com/windowsexperience/2018/12/19/driver-quality-in-the-windows-ecosystem/ https://blogs.windows.com/windowsexperience/2018/12/19/drive...
- flowerlad 2y ago“Drivers are tested extensively through a process that involves many companies.” That didn’t happen. Apparently, the policy for drivers does not apply to kernel modules… WTF!!
- efitz 2y agothe driver in question was tested and passed WHQL. CrowdStrike included functionality in the driver to interpret a downloadable file (similar to an antivirus signature file). The file in the problematic update was malformed, and the CrowdStrike driver did not handle this case properly; Windows was unable to continue given the exception in question[1]. No operating system can guarantee that a driver will never cause the machine to crash. This wasn’t Microsoft’s fault. [1] https://youtu.be/wAzEJxOo1ts?si=WZC1fq2Vro8K9vqS https://youtu.be/wAzEJxOo1ts?si=WZC1fq2Vro8K9vqS
- ilaksh 2y agoI had never heard of CrowdStrike before. The name seems to imply that it's a military cybersecurity weapon, such as for a massive DDOS on command.
- remram 2y agoRight? From the name you'd assume it's either some sort of missile, or a famous OpenSSL vulnerability (heartbleed, specter, etc). I'd feel weird picking a protective app called "crowdstrike".
- 2-3-7-43-1807 2y agomicrosoft and apple enter the chat
- zdragnar 2y agoNames aren't really all that accurate in tech anyway. You don't go to Amazon for rainforest information. You don't go to Google for information about big numbers. You don't go to Uber for an Aryan vibe. You don't go to Apple for fruit. You don't open Windows to let in a cool breeze. You don't need a trap for your wireless mice. Monster isn't going to eat you. Jeeves isn't actually a person you can ask, and knows less about buttling than you'd expect. FireEye is competition to crowdstrike, not an STD. You get used to it after awhile.
- rvz 2y agoI don't think you will get the introductory "As a (so-called) senior software engineer at Crowdstrike" brags here at this time since they are internally screaming and swallowing their pride over that massive outage. Why would any of them come forward right now?
- dvfjsdhgfv 2y agoYeah even that guy who always told them they should always do canary deployments should stifle his "I toldja!" instinct and just keep quiet until everybody forgets.
- pdimitar 2y agoOh it was a NULL pointer access... again. I wonder if this is a preventable problem, hmmmm. No sir, not at all! It was inevitable! Let's just leave this behind us! Those who think it's preventable are part of a certain "evangelism strike force", I tell you! They make zero good arguments, never listen to them! Back to business as usual, boys! C/C++ are without a single flaw!
- binary132 2y agoThey’ll never admit the obvious truth, which is that it was caused by a combination of incompetent administration and incompetent engineering.
- lubujackson 2y agoSussing things out, they uploaded a "config" file that had a .sys extension that caused all the trouble. .sys files are supposed to be protected system files and require specual privileges to touch. I imagine Crowdstrike requires some special type of Windows "root access" to operate effectively (like many antivirus packages) in order to detect and block low level attacks. So where things likely went pear-shaped was Crowdstrike's QA process for config updates is possibly less stringent than core code updates. But because they were using .sys files it was actually given elevated privileges and gets installed during boot. As for the actual bug, I expect it was either something like the sys file referencing itself or some sort of stack overflow somewhere, both of which I would pin on Microsoft for not being able to detect and recover from during boot up. All of this is straight guesswork based solely on experience as a longtime Windows user.
- gruez 2y ago>.sys files are supposed to be protected system files and require specual privileges to touch. I imagine Crowdstrike requires some special type of Windows "root access" to operate effectively (like many antivirus packages) in order to detect and block low level attacks. Any file in C:\windows\ is protected by default and requires administrative privileges to modify. There isn't any extra protections for .sys files specifically. >As for the actual bug, I expect it was either something like the sys file referencing itself or some sort of stack overflow somewhere, both of which I would pin on Microsoft for not being able to detect and recover from during boot up. Since when is it the operating system's responsibility to recover from badly written kernel mode code?
- jen20 2y agoI’d say it’s a pretty reasonable course of action to stop loading a driver if crashes three times in a row, unless explicitly told otherwise.
- gruez 2y agoHow do you know which driver is responsible for the crash? It's all in kernel mode so everything is sharing the same address space. Even if you could determine that the crashed code belonged to a given driver, it's impossible to know whether that was responsible the crash, or some other driver called it with bad parameters. Even if you could get past all those problems, not loading a driver also comes with other problems. If the driver is required for some critical functionality, not loading the driver might cause other crashes. It also means you can disable an antivirus by getting to crash 3 times in a row.
- pentagrama 2y agoI think they don't share specific details because it will reveal how bad they handle QA and update rollouts processes for a product that expensive, maybe at some point law enforcement will reveal the embarrassing truth.
- TradingPlaces 2y agoI am not from Crowdstrike, but they were under pressure in 2022-2023 to show a profit. They did that by cutting expense growth across the board, and I am now guessing QA got hit particularly hard.
- markus_zhang 2y agoIf that's the case they probably also outsourced some cost center. IT and QA come to mind.
- 2-3-7-43-1807 2y agothe world deserves a detailed post mortem and an apology!!! considering the scale of this shit show there's no way this didn't affect a significant amount of people very seriously. i would make a bet that people died because of it or experienced any kind of calamity be it personally or medically.
- Bluestein 2y agoI think we have reached and inflection point. I mean we have to make an inflection point out of this.- This outage represents more than just a temporary disruption in service; it's a black swan célèbre of the perilous state of our current technological landscape. This incident must be seen as an inflection point, a moment where we collectively decide to no longer tolerate the erosion of craftsmanship, excellence, and accountability that I feel we've been seeing all over the place. All over critical places.- Who are we to make this demand? Most likely technologists, managers, specialists, and concerned citizens with the expertise and insight to recognize the dangers inherent in our increasingly careless approach to ... many things, but, particularly technology. Who is to uphold the standards that ensure the safety, reliability, and integrity of the systems that underpin modern life? Government? Historically, the call for accountability and excellence is not new. From Socrates to the industrial revolutions, humanity has periodically grappled with the balance between progress and prudence. People have seen - and complained about - life going to hell, downhill, fast, in a hand basket without brakes since at least Socrates.- Yet, today’s technological failures have unprecedented potential for harm. The Crowdsource outage killed, halted businesses, and posed serious risks to safety—consequences that were almost unthinkable in previous eras. This isn't merely a technical failure; it’s a societal one, revealing a disregard for foundational principles of quality and responsibility. Craftsmanship. Care and pride in one's work.- Part of the problem lies in the systemic undervaluation of excellence. In pursuit of speed and profit uber alles. Many companies have forsaken rigorous testing, comprehensive risk assessments, and robust security measures. The very basics of engineering discipline—redundancy, fault tolerance, and continuous improvement—are being sacrificed. This negligence is not just unprofessional; it’s dangerous. As this outage has shown, the repercussions are not confined to the digital realm but spill over into the physical world, affecting real lives. As it always has. But never before have the actions of so few "perennial interns" affected so many.- This is a clarion call for all of us with the knowledge and passion to stand up and insist on change. Holding companies accountable, beginning with those directly responsible for the most recent failures.- Yet, it must go beyond punitive measures. We need a cultural shift that re-emphasizes the value of craftsmanship in technology. Educational institutions, professional organizations, and regulatory bodies must collaborate to instill and enforce higher standards. Otherwise, lacking that, we must enforce them ourselves. Even if we only reach ourselves in that commitment.- Perhaps we need more interdisciplinary dialogue. Technological excellence does not exist in a vacuum. It requires input from ethical philosophers, sociologists, legal experts. Anybody willing and able to think these things through.- The ramifications of neglecting these responsibilities are clear and severe. The fallout from technological failures can be catastrophic, extending well beyond financial losses to endanger lives and societal stability. We must therefore approach our work with the gravity it deserves, understanding that excellence is not an optional extra but an essential quality sine qua non in certain fields.- We really need to make this be an actual tuning point, and not just another Wikipedia page.-
- craxdevil7 2y ago[dead]
- tnvmadhav 2y agoI'd guess they were missing a if err != nil { return err }