8 ms·
There's at least five different things that went wrong simultaneously. 1. Poorly written code in the kernel module crashed the whole OS, and kept trying to par
by mtlmtlmtlmtl 2y ago
There's at least five different things that went wrong simultaneously.
1. Poorly written code in the kernel module crashed the whole OS, and kept trying to parse the corrupted files, causing a boot loop. Instead of handling the error gracefully and deleting/marking the files as corrupt.
2. Either the corrupted files slipped through internal testing, or there is no internal testing.
3. Individual settings for when to apply such updates were apparently ignored. It's unclear whether this was a glitch or standard practice. Either way I consider it a bug(it's just a matter of whether it's a software bug or a bug in their procedures).
4. This was pushed out everywhere simultaneously instead of staggered to limit any potential damage.
5. Whatever caused the corruption in the first place, which is anyone's guess.
- simonh 2y agoThere is a story out that the problem was introduced in a post processing step after testing. That makes more sense than that there was no testing. If true it means they thought they’d tested the update, but actually hadn’t.
- rwmj 2y agoZero effort to fuzz test the parser too. I mean, we know how to harden parsers against bugs and attacks, and any semi-competent fuzzer would have caught such a trivial bug.
- chrisjj 2y agoThe triggering file was all zeros. It is not possible that only this pattern caused the crash, and fuzzing omitted to try this unfuzzy pattern?
- monsieurbanana 2y agoIn my limited experience, I thought any serious fuzzing program does test for all "standard" patters like only null bytes, empty strings, etc...
- gliptic 2y agoNo, it wasn't. Crowdstrike denied it had to do with zeros in the files.
- jojobas 2y agoAt this point I wouldn't be paying too much attention to what Crowdstrike is saying.
- hello_moto 2y agoHave to speak the truth albeit at minimum, in case legal...
- kchr 2y agoWhich also explains why they, only if needed to cover their back legally, confirm or deny details being shared on social and mass media.
- watwut 2y agoPossible? Yes. Likely? No.
- formerly_proven 2y agoInstrumented fuzzing (like AFL and friends) tweaks the input to traverse unseen code paths in the target, so they're super quick to find stuff like "heyyyyy, nobody is actually checking if this offset is in bounds before loading from that address".
- omeid2 2y agoThe files in question has a magic number is "0xAAAAAAAA" so it is not possible that the file was all zeros.
- Retr0id 2y agoCompetent fuzzers don't just use random bytes, they systematically explore the state-space of the target program. If there's a crash state to be found by feeding in a file full of null bytes, it's probably going to be found quickly. A fun example is that if you point AFL at a JPEG parser, it will eventually "learn" to produce valid JPEG files as test cases, without ever having been told what JPEG file is supposed to look like. https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-th...
- rwmj 2y agoAFL is really "magical". It finds bugs very quickly and with little effort on our part except to leave it running and look at the results occasionally. We use it to fuzz test a variety of file formats and network interfaces, including QEMU image parsing, nbdkit, libnbd, hivex. We also use clang's libfuzzer with QEMU which is another good fuzzing solution. There's really no excuse for CrowdStrike not to have been using fuzzing.
- layer8 2y agoNo, it wasn’t all zeros: https://x.com/patrickwardle/status/1814782404583936170 https://x.com/patrickwardle/status/1814782404583936170
- mavhc 2y agoAV software is a great target for malware, badly written, probably runs too much stuff in the kernel, tries to parse everything
- Comfy-Tinwork 2y agoAnd at the very least straight to system level access if not more.
- MyFedora 2y agoAnti-cheats also whitelist legit AV drivers, even though cheaters exploit them to no end.
- londons_explore 2y agoAV software needs kernel privilidges to have access to everything it needs to inspect, but the actual inspection of that data should be done with no privilidges. I think most AV companies now have a helper process to do that. If you successfully exploit the helper process, the worst damage you ought to be able to do is falsely find files to be clean.
- bornfreddy 2y ago> ...the worst damage you ought to be able to do is... Ought. But it depends on the way the communication with the main process is done. I wouldn't be surprised if the main process trusts the output from the parser just a tiny bit too much.
- jatins 2y agoYou are seriously overestimating the engineering practises at these companies. I have worked in "enterprise security" previously though not at this scale. In a previous life I worked with of the engineering leaders currently at Crowdstrike. I'll bet you this company has some arbitrary unit test coverage requirements for PRs which developers game be mocking the heck out of dependencies. I am sure they have some vanity sonarqube integration to ensure great "code quality". This likely also went through manual QA. However I am sure the topic of fuzz testing would not have come up once. These companies sell checkbox compliance, and they themselves develop their software the same way. Checking all the "quality engineering" boxes with very little regards for long term engineering initiatives that would provide real value. And I am not trying to kick Crowdstrike when they are down. It's the state of any software company run by suits with myopic vision. Their engineering blogs and their codebases are poles apart.
- hulitu 2y ago6. No development process, no testing.
- krisoft 2y agoHow is that different from point 2?
- ratorx 2y agoI’d also maybe add another one on the Windows end: 6) some form of sandboxing/error handling/api changes to make it possible to write safer kernel modules (not sure if it already exists and was just not used). It seems like the design could be better if a bad kernel module can cause a boot loop in the OS…
- leosarev 2y agoThere is sandboxing API in Windows. It's called running programs in userspace.
- hello_moto 2y agoRun what a userspace?
- layer8 2y agoIt’s a tough problem, because you also don’t want the system to start without the CrowdStrike protection. Or more generally, a kernel driver is supposedly installed for a reason, and presumably you don’t want to keep the system running if it doesn’t work. So the alternative would be to shut down the system upon detection of the faulty driver without rebooting, which wouldn’t be much of an improvement in the present case.
- ratorx 2y agoI can imagine better defaults. Assuming the threat vector is malicious programs running in userspace (probably malicious programs in kernel space is game over anyway right?), then you could simply boot into safe mode or something instead of crashlooping. One of the problems with this outage was that you couldn’t even boot into safe mode without having the bit locker recovery key.
- layer8 2y agoYou don’t want to boot into safe mode with networking enabled if the software that is supposed to detect attacks from the network isn’t running. Safe mode doesn’t protect you from malicious code in userspace, it only “protects” you from faulty drivers. Safe mode is for troubleshooting system components, not for increasing security. I don’t know the exact reasoning why safe mode requires the BitLocker recovery key, but presumably not doing so would open up an attack vector defeating the BitLocker protection.
- dartos 2y agoBugs happen. Not staggering the updates is what blew my mind.
- londons_explore 2y agoSince the issue manifested at 04:09 UTC, which is 11pm where Crowdstrikes HQ is, I would guess someone was working late at night and skipped the proper process so they could get the update done and go to bed. They probably considered it low risk, had done similar things of times hundreds of times before, etc.
- dartos 2y ago> They probably considered it low risk Wild that anyone would consider anything in the “critical path” low risk. I would bet that they just don’t do rolling releases normally since it never caused issues before.
- hello_moto 2y agoCompanies these days are global btw. Not everyone is working on the same timezone.
- londons_explore 2y agoThey don't appear to have engineering jobs in any location where that would be considered regular office hours...
- hello_moto 2y agohttps://crowdstrike.wd5.myworkdayjobs.com/crowdstrikecareers https://crowdstrike.wd5.myworkdayjobs.com/crowdstrikecareers I see remote, Israel, Canada. https://crowdstrike.wd5.myworkdayjobs.com/en-US/crowdstrikecareers/job/Machine-Learning-Operations-and-Response-Manager--Data-Science--Remote-_R18932 https://crowdstrike.wd5.myworkdayjobs.com/en-US/crowdstrikec... This one specifically Spain and Romania I know they bought companies all over the globe from Denmark to other locations.
- rco8786 2y agoNumber 4 continues to be the most surprising bit to me. I could not fathom having a process that involves deploying to 8.5 million remote machines simultaneously. Bugs in code I can almost always understand and forgive, even the ones that seem like they’d be obvious with hindsight. But this is just an egregious lack of the most basic rollout standards.
- gitfan86 2y agoThey probably don't get to claim agile story points until the ticket is in finished state. And they probably have a culture where vanity Metrics like "velocity" are prioritized
- nmg 2y agoThis would answer the question that i've not heard anyone asking: what incentivized the bad decisions that led to this catastrophic failure?
- phs318u 2y agoMy understanding is that the culture (as reported by some customers) is quite aggressive and pushy. They are quite vocal when customers don’t turn in automatic updates. It makes sense in a way - given their fast growth strategy (from nowhere to top 3) and desire to “do things differently” - the iconoclast upstarts that redefine the industry. Or to summarise - hubris.
- hello_moto 2y agoTo catch 0day quickly, EDR needs to know "how". The "how" here is AV definition or a way to identify the attack. In CS-speak: content. Catching 0day quickly results in good reputation that your EDR works well. If people turn off their AV definition auto-update, they are at-risk. Why use EDR if folks don't want to stop attack quickly?
- 2y ago
- rainsford 2y ago> 2. Either the corrupted files slipped through internal testing, or there is no internal testing. This is the most interesting question to me because it doesn't seem like there is an obviously guessable answer. It seem very unlikely to me that a company like CrowdStrike pushes out updates of any kind without doing some sort of testing, but the widespread nature of the outage would also seem to suggest any sort of testing setup should have caught the issue. Unless it's somehow possible for CrowdStrike to test an update that was different than what was deployed, it's not obvious what went wrong here.
- bloopernova 2y agoI had read somewhere that the definition file was corrupted after testing, during the final CI/CD pipeline.
- dboreham 2y agoBecause nowadays nobody knows you test the actual bits you're going to ship, not whatever random crap comes out of someone's build script run in a different place and time that's supposed to be the same as what you'll ship.
- shrimp_emoji 2y agoWell, Microsoft led by example with #2: https://news.ycombinator.com/item?id=20557488 https://news.ycombinator.com/item?id=20557488
- pclmulqdq 2y agoNumber 4 is what everyone will fixate on, but I have the biggest problem with number 1. Anything like this sort of file should have (1) validation on all its pointers and (2) probably >2 layers of checksumming/signing. They should generally expect these files to get corrupted in transit once in a while, but they didn't seem to plan for anything other than exactly perfect communication between their intent and their kernel driver.
- HDThoreaun 2y agoFiles dont get corrupted in transit with TCP.
- pclmulqdq 2y agoI think you mean: In theory, files don't get corrupted in transit with TCP. In theory, they also don't get corrupted when stored in memory or on disks either. The only reason any of these things don't cause issues in practice is checksums and error correcting codes.
- HDThoreaun 2y agoNo, I mean that TCP checks for corruption in transit and has the packets resent in that case. I guess you could be running a buggy implementation, but that seems unlikely with how ubiquitous TCP is.
- pclmulqdq 2y agoI would suggest that you read this: https://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf https://web.mit.edu/Saltzer/www/publications/endtoend/endtoe... Errors can show up any time, and usually show up between the parts that checksums correct. On the wire, TCP protects you with a (weak) checksum. Off the wire, your computer and filesystem can still screw things up. Even CPU bugs can do this.
- dcuthbertson 2y agoI wonder if it was pushed anywhere that didn't crash, as an extension of "It works on my machine. Ship it!" I've built a couple of kernel drivers over the years and what I know is that ".sys" files are to the kernel as ".dll" files are to user-space programs in that the ones with code in them run only after they are loaded and a desired function is run (assuming boilerplate initialization code is good). I've never made a data-only .sys file, but I don't see why someone couldn't. In that case, I'd guess that no one ever checked it was correct, and the service/program that loads it didn't do any verification either -- why would it, the developers of said service/program would tend to trust their own data .sys file would be valid, never thinking they'd release a broken file or consider that files sometimes get corrupted -- another failure mode waiting to happen on some unfortunate soul's computer.
- kchr 2y agoThe file extension is `sys` by convention, it's nothing magical to it and it's not handled in any special way by the OS. In the case of CrowdStrike, there seems to be some confusion as to why they use this file extension since it's only supposed to be a config/data file to be used by the real kernel driver.
- dcuthbertson 2y agoThanks. I understand that '.sys' is a naming convention. I'd guess that they used it because those config/data files are used by their kernel driver, and so makes kernel vs user-space files easier to distinguish.
- cynicalsecurity 2y agoI'm betting on them having no internal testing.
- fhub 2y ago#1 could be slit into two parts I think. Microsoft kernel side and CloudStrike module side.
- LtWorf 2y ago> 4. This was pushed out everywhere simultaneously instead of staggered to limit any potential damage. Most importantly it was never tested at all :D
- dboreham 2y agoTBF it was extensively tested. On airline and hospital computers.
- spike021 2y ago6. Companies using CS have no testing to verify that new updates won't break anything. Any SWE job I've worked over my entire career, nothing is deployed with new versions of dependencies without testing them against a staging environment first.
- strunz 2y agoCrowdstrike doesn't give that option. Updates happen without a choice to "keep you safe".
- bryant 2y agoOf all of these, I think #3 has crowdstrike the most exposed, legally. Companies with robust update and config management protocols got burned by this as well, including places like hospitals and others with mission critical systems where config management is more strictly enforced. If the crowdstrike selloff continues, I'm betting this will be why. (There's a chance I'll make trading decisions based on this rationale in the next 72 hours, though I'm not certain yet)
- cratermoon 2y ago> Individual settings for when to apply such updates were apparently ignored. I've heard that said elsewhere, but I haven't found a source for it at all. Are you able to point to one for me?
- WWLink 2y ago6. The software architecture where a kernel mode driver is required is faulty and poor design.
- dboreham 2y ago7. The operating system vendor that allows buggy kernel drivers to be loaded and run and panic the system.
- denton-scratch 2y ago> or there is no internal testing Thing is, as far as I can see, deploying this database update to a Windows machine will result promptly and unconditionally in a BSOD. That implies that this update was tried on exactly zero machines before it was shipped. The bug can't have "slipped through internal testing"; it would have failed immediately on any machine it was loaded on.