10 ms·
This should not have passed a competent C/I pipeline for a system in the critical path. I’m not even particularly stringent when it comes to automated test acr
by AndrewKemendo 2y ago
This should not have passed a competent C/I pipeline for a system in the critical path.
I’m not even particularly stringent when it comes to automated test across-the-board but for this level of criticality of system, you need exceptionally good state management
To the point where you should not roll to production without an integration test on every environment that you claim to support
Like it’s insane to me that this size and criticality of a company doesn’t have a staging or even a development test server that tests all of the possible target images that they claim to support.
Who is running stuff over there - total incompetence
- martinky24 2y agoA lot of assumptions here that probably aren't worth making without more info -- For example it could certainly be the case that there was a "real" file that worked and the bug was in the "upload verified artifact to CDN code" or something, at which point it passes a lot of things before the failure. We don't have the answers, but I'm not in a rush to assume that they don't test anything they put out at all on Windows.
- EvanAnderson 2y agoI haven't seen the file, but surely each build artifact should be signed and verified when it's loaded by the client. The failure mode of bit rot / malice in the CDN should be handled.
- gjsman-1000 2y agoPerhaps - but if I made a list of all of the things your company should be doing and didn't, or even things that your side project should be doing and didn't, or even things in your personal life that you should be doing and haven't, I'm sure it would be very long.
- EvanAnderson 2y agoA company deploying kernel-mode code that can render huge numbers of machines unusable should have done better. It's one of those "you had one job" kind of situations. They would be a gigantic target for malware. Imagine pwning a CDN to pwn millions of client computers. The CDN being malicious would be a major threat.
- soraminazuki 2y agoOh, they have one job for sure. Selling compliance. All else isn't their job, including actual security. Antiviruses are security cosplay that works by using a combination of bug-riddled custom kernel drivers and unsandboxed C++ parsers running with the highest level of privileges to tamper with every bit of data it can get its hands on. They violate every security common sense. They also won't even hesitate to disable or delay rollouts of actual security mechanisms built into browsers and OSes if it gets in the way. The software industry needs to call out this scam and put them out of business sooner than later. This has been the case for at least a decade or two and it's sad that nothing has changed. https://ia801200.us.archive.org/1/items/SyScanArchiveInfocon/SyScan%202014%20Singapore/SyScan%202014%20presentations/SyScan2014_JoxeanKoret_BreakingAntivirusSoftware.pdf https://ia801200.us.archive.org/1/items/SyScanArchiveInfocon... https://robert.ocallahan.org/2017/01/disable-your-antivirus-software-except.html https://robert.ocallahan.org/2017/01/disable-your-antivirus-...
- heraldgeezer 2y agoNope, I have seen software like Crowdstrike, S1, Huntress and Defender E5 stop active ransomware attacks.
- soraminazuki 2y agoThat anecdote doesn't justify installing gaping security holes into the kernel with those tools. Actual security requires knowledge, good practice, and good engineering. Antiviruses can never be a substitute.
- xyst 2y agoHindsight is 20/20 This is a public company after all. In this market, you don’t become a “Top-Tier Cybersecurity Company At A Premium Valuation” with amazing engineering practices. Priority is sales, increasing ARR, and shareholders.
- fsloth 2y agoThis is the market. Good engineering practices don’t hurt but they are not mandatory. If Boeing can wing it so can everybody.
- StressedDev 2y agoBoeing has been losing market share to AirBus for decades. That is what happens when you cannot fix your problems, sell a safe product, keep costs in line, etc.
- bsaul 2y agoi wonder how far from the edge a company driven by business people can go before they start to put the focus back on good engineering. Probably much too late in general. Business bonus are yearly, and good/bad engineering practices take years to really make a difference.
- moomin 2y agoThe question then becomes: if the market is producing near-monopolies of stuff that is barely fit for purpose, how do we fix the market?
- StressedDev 2y agoNot caring about the actual product will eventually kill a company. All companies have to constantly work to maintain and grow their customer base. Customers will eventually figure out if a company is selling snake oil, or a shoddy product. Also, the tech industry is extremely competitive. Leaders frequently become laggards or go out of business. Here are some companies who failed or shrank because their products could not complete: IBM, Digital Equipment, Sun, Borland, Yahoo, Control Data, Lotus (later IBM), Evernote, etc. Note all of these companies were at some point at the top of their industry. They aren't anymore.
- AdamJacobMuller 2y agoThe file was just full of null bytes. It's very possible the signature validation and verification happens after the bug was triggered.
- wk_end 2y ago"Load a kernel module and then verify it" is not the way any remotely competent engineer would do things. (...which doesn't rule out the possibility that CS was doing it.)
- justinclift 2y agoThe ClownStrike Falcon software that runs on both Linux and macOS was incredibly flaky and a constant source of kernel problems at my previous work place. We had to push back on it regardless of the security team's (strongly stated) wishes, just to keep some of the more critical servers functional. Pretty sure "competence" wasn't part of the job description of the ClownStrike developers, at least for those pieces. :( :( :(
- soraminazuki 2y agoClownStrike left kernel panics unfixed for a year until macOS deprecated kernel extensions altogether. It was scary because crash logs indicated that memory was corrupted while processing network packets. It might've been exploitable.
- usr1106 2y agoHaven't used Windows for close to 15 years, but I read the file is (or rather supposed to be) a NT kernel driver. Are those drivers signed? Who can sign them? Only Microsoft? If it's true the file contained nothing but zeros that seems to be also kernel vulnerability. Even if signing were not mandatory, shouldn't the kernel check for some structure, symbol tables or the the like before proceeding?
- dagaci 2y ago
- chatmasta 2y agoThe actual bug is not that they pushed out a data file with all nulls. It’s that their kernel module crashes when it reads this file. I’m not surprised that there is no test pipeline for new data files. Those aren’t even really “build artifacts.” The software assumes they’re just data. But I am surprised that the kernel module was deployed with a bug that crashed on a data file with all nulls. (In fact, it’s so surprising, that I wonder if there is a known failing test in the codebase that somebody marked “skip” and then someone else decided to prove a point…) Btw: is that bug in the kernel module even fixed? Or did they just delete the data file filled with nulls?
- hansvm 2y ago> Btw: is that bug in the kernel module even fixed? Or did they just delete the data file filled with nulls? Is that a real question? They definitely didn't do anything more than delete the file, perhaps just rename it.
- chatmasta 2y agoYeah they have been very obfuscatory in calling this a “fix.” I watched the CEO on Cramer and he kind of danced around this point.
- SoftTalker 2y agoThe instructions that my employer emailed were: 1. Start Windows in Safe Mode or the Windows Recovery Environment (Windows 11 option). 2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory. 3. Locate the file matching C-00000291*.sys and delete it. 4. Restart your device normally.
- chrisjj 2y ago> it could certainly be the case that there was a "real" file that worked and the bug was in the "upload verified artifact to CDN code" or something I.e. only one link in the chain wasn't tested. Sorry, but that will not do. > We don't have the answers, but I'm not in a rush to assume that they don't test anything they put out at all on Windows. The parent post did not suggest they don't test anything. It suggested they did not test the whole chain.
- martinky24 2y agoFrom the parent comment: > it’s insane to me that this size and criticality of a company doesn’t have a staging or even a development test server that tests all of the possible target images that they claim to support I know nothing about Crowdstrike, but I can guarantee that "they need to test target images that they claim to support" isn't what went wrong here. The implication that they don't test against Windows is so incredulous, it's hard to take the poster of that comment seriously.
- StressedDev 2y agoThank you for pointing this out. Whenever I read articles about security, or reliability failures, it seems like the majority of the commenters assume that the person or organization which made the mistake is a bunch of bozos. The fact is mistakes happen (even huge ones), and the best thing to do is learn from the mistakes. The other thing people seem to forget is they are probably doing a lot of the same things which got CrowdStrike into trouble. If I had to guess, one problem may be that CrowdStrike's Windows code did not validate the data it received from the update process. Unfortunately, this is very common. The lesson is to validate any data received from the network, from an update process, received as user input, etc. If the data is not valid, reject it. Note I bet at least 50% of the software engineers commenting in this thread do not regularly validate untrusted data.
- maxquesar 2y agoNot validating an update signature is a huge security compliance issue. When you get certified, and I assume CroudStrike had many certifications, you provide proof of your compliance to many scenarios. Proving your updates are signed and verified is absolutely one of those.
- 0xcafecafe 2y agoThey could even have done slow rollouts. Roll it out to a geographical region and wait an hour or so before deploying elsewhere.
- xyst 2y agoOr test in local environments first. Slow rollouts like this tend to make deployments very very painful.
- koliber 2y agoSlow rollouts can be quite quick. We used to do 3-day rollouts. Day one was a tiny fraction. Day two was about 20%. Day three was a full rollout. It was ages ago, but from what I remember, the first day rollout did occasionally catch issues. It only affected a small number of users and the risk was within the tolerance window. We also tested locally before the first rollout.
- rplnt 2y agoI don't know about this particular update, but when I used to work for an AV vendor we did like 4 "data" updates a day. It is/was about being quick a lot of the time, you can't stage those over 3 days. Program updates are different, drivers of this level were very different (Microsoft had to sign those, among many things). Not thay it exuces anything, just that this probably wasn't treated as an update at all.
- daseiner1 2y agoYou say even (emphasis mine). Is this not industry standard?
- saati 2y agoIn theory CrowdStrike protects you from threats, leaving regions unprotected for an hour would be an issue.
- Thaxll 2y ago
- notabee 2y agoWithout delving into any kind of specific conspiratorial thinking, I think people should also include the possibility that this was malicious. It's much more likely to be incompetence and hubris, but ever since I found out that this is basically an authorized rootkit, I've been concerned about what happens if another Solarwinds incident occurs with Crowdstrike or another such tool. And either way, we have the answer to that question now: it has extreme consequences. We really need to end this blind checkbox compliance culture and start doing real security.
- sonotathrowaway 2y agoThat’s not even getting into the fuckups that must have happened to allow a bad patch to get rolled out everywhere all at once.
- carterschonwald 2y agoThe strange thing is that when I interviewed there years ago with the team that owns the language that runs in the kernel, they said their ci has 20k or 40k machine os combinations/configurations. Surely some of them were vanilla windows!
- dboreham 2y agoThey used synthetic test data in CI that doesn't consist of zeros.
- hnlmorg 2y agoIt seems unlikely that a file entirely full of null characters was the output of any automated build pipeline. So I’d wager something got built, passed the CI tests, then the system broke at some point after that when the file was copied ready for deployment. But at this stage, all we are doing is speculating.
- russdill 2y agoYou can have all the CI, staging, test, etc. If some bug after that process nulls the file, the rest doesn't matter
- Jtsummers 2y agoIf a garbage file is pushed out, the program could have handled it by ignoring it. In this case, it did not and now we're (the collective IT industry) dealing with the consequences of one company that can't be bothered to validate its input (they aren't the only ones, but this is a particularly catastrophic demonstration of the importance of input validation).
- russdill 2y agoI'll agree that this appears to have been preventable. Whatever goes through CI should have a hash, deployment should validate that hash, and the deployment system itself should be rigorously tested to insure it breaks properly if the hash mismatches at some point in the process
- fabian2k 2y agoThose signature files should have a checksum, or even a digital signature. I mean even if it doesn't crash the entire computer, a flipped bit in there could still turn the entire thing against a harmless component of the system and lead to the same result.
- HL33tibCe7 2y agoWhat happens when your mechanism for checksumming doesn't work? What happens when your mechanism for installing after the checksum is validated doesn't work? It's just too early to tell what happened here. The likelihood is that it _was_ negligence. But we need a proper post-mortem to be able to determine one way or another.
- LorenPechtel 2y agoYup. I had quite a battle with some sort of system bug (never fully traced) where I wrote valid data but what ended up on disk was all zero. It appeared to involve corrupted packets being accepted as valid. It doesn't matter how much you test if something down the line zeroes out your stuff.
- dagaci 2y ago/* Acceptance criteria #1: do not allow machine to boot if invalid data signatures are present, this could indicate a compromised system. Booting could cause presidents diary to transmit to rival 'Country' of the week */ if(dataFileIsNotValid) { throw FatalKernelException("All your base are compromised"); } EDIT+ Explanation: With hindsight not booting may be exactly the right thing to do since a bad datafile would indicate a compromised distribution/ network. The machines should not fully boot until file with valid signature is downloaded.*
- arp242 2y ago> Like it’s insane to me that this size and criticality of a company doesn’t have a staging or even a development test server that tests all of the possible target images that they claim to support. Who is saying they don't have that? Who is saying it didn't pass all of that? You're making tons of assumptions here.
- martinky24 2y agoYeah... the comment above reads like someone who has read a lot of books on CI deployment, but has zero experience in a real world environment actually doing it. Quick to throw stones with absolutely no understanding of any of the nuances involved.
- chrisjj 2y agoSo let's hear the "nuances" that excuse this.
- cweld510 2y agoIt’s not a matter of excusing or not excusing it. Incidents like this one happen for a reason, though, and the real solution is almost never “just do better.” Presumably crowdstrike employs some smart engineers. I think it’s reasonable to assume that those engineers know what CI/CD is, they understand its utility, and they’ve used it in the past, hopefully even at Crowdstrike. Assuming that this is the case, then how does a bug like this make it into production? Why aren’t they doing the things that would have prevented this? If they cut corners, why? It’s not useful or productive to throw around accusations or demands for specific improvements without answering questions like these.
- arp242 2y agoI am not defending of excusing anything. I am saying there is not enough information to make a judgement one way or the other. Right now, we have almost zero technical details. Call me old-fashioned and boring, but I'd like to have some basic facts about the situation first. After this I decide who does and doesn't deserve a bollocking.
- dheera 2y agoI don't know if people on Microsoft ecosystems even know what CI pipelines are. Linux and Unix ecosystems in general work by people thoroughly testing and taking responsibility for their work. Windows ecosystems work by blame passing. Blame Ron, the IT guy. Blame Windows Update. Blame Microsoft. That's how stuff works. It has always worked this way. But also, all the good devs got offered 3X the salary at Google, Meta, and Apple. Have you ever applied for a job at CrowdStrike? No? That's why they suck. * A disproportionately large number of Windows IT guys are named Ron, in my experience.
- miki123211 2y agoKeep in mind that this was probably a data file, not necessarily a code file. It's possible that they run tests on new commits, but not when some other, external, non-git system pushes out new data. Team A thinks that "obviously the driver developers are going to write it defensively and protect it against malformed data", team B thinks "obviously all this data comes from us, so we never have to worry about it being malformed" I don't have any non-public info about what actually happened, but something along these lines seems to be the most likely hypothesis to me. Edit: Now what would have helped here is a "staged rollout" process with some telemetry. Push the update to 0.01% of your users and solicit acknowledgments after 15 minutes. If the vast majority of systems are still alive and haven't been restarted, keep increasing the threshold. If, at any point, too many of the updated systems stop responding or indicate a failure, immediately stop the rollout, page your on-call engineers and give them a one-click process to completely roll the update back, even for already-updated clients. This is exactly the kind of issue that non-invasive, completely anonymous, opt-out telemetry would have solved.
- adzm 2y agoThis was a .dll in all but name fwiw.
- ar_lan 2y ago> tests all of the possible target images that they claim to support. Or even at the very least the most popular OS that they support. I'm genuinely imagining right now that for this component, the entirety of the company does not have a single Windows machine they run tests on.
- tinytime 2y agoIt's wild that I'm out here boosting existing unit testing practices with mutation testing https://github.com/codeintegrity-ai/mutahunter https://github.com/codeintegrity-ai/mutahunter and there are folks out there that don't even do the basic testing.