11 ms·
The CrowdStrike file that broke everything was full of null characters?
- fourteenfour 2y agoAt least it compressed well, which must have saved network resources during the update. :)
- sitkack 2y agoResources like wall clock time.
- TillE 2y agoThat probably explains how it got past internal testing. Something went wrong after that, during deployment.
- behnamoh 2y agoIt could be as simple as cosmic radiation that flipped a bit (it has happened before: https://www.independent.co.uk/news/science/subatomic-particles-cosmic-rays-computers-change-elections-planes-autopilot-a7584616.html https://www.independent.co.uk/news/science/subatomic-particl...), or as sophisticated as an adversarial hacking.
- rvnx 2y agoThe same cosmic radiation that flips the bits to make some specific political party win.
- zimpenfish 2y agoSomeone on the Fediverse conjectured that it might have been down to the Azure glitch earlier in the day. An empty file would fit that if they weren't doing proper error checking on their downloads, etc.
- homero 2y agoIt's crazy if they weren't signing and verifying downloads
- janice1999 2y agoIt's still crazy that a security tool does not validate content files it loads from disk that get regularly updated. Clearly fuzzing was not a priority either.
- Zigurd 2y agoHow many years has this Crowdstrike code been running without issues? You have put your finger on it: Fuzzing should have been part of a test plan. Even TDD isn't a bastard test engineer writing tests that probe edge cases. Even observing that your unit tests have good code coverage isn't a substitute for fuzzing. There is even a counter-argument that something that been reliable in the field should not be fixed for reasons like failing a test case never seen in real deployments, so why go making trouble.
- password4321 2y agohttps://news.ycombinator.com/item?id=41006104#41006555 https://news.ycombinator.com/item?id=41006104#41006555 the flawed data was added in a post-processing step of the configuration update, which is after it's been tested internally but before it's copied to their update servers
- gbin 2y agoI don't understand, how the signature even worked? Please please tell me those drivers are signed... Right? ...
- Retr0id 2y agoWhile I won't discount it entirely, I think the people acting like this (alone) implies malice are being very silly.
- JumpCrisscross 2y agoIt demonstrates terrible QC.
- loloquwowndueo 2y agoDon’t mess with Quebec :D
- Retr0id 2y agoClearly something failed catastrophically, but it could well be post-QC
- midtake 2y agoI disagree. In the current day the stakes are too highly to naively attribute software flaws to incompetence. We should assume malice until it is ruled out, otherwise it will become a vector for software implants. These are matters of national security at this point.
- Retr0id 2y agoYou can (and should) want to identify the root cause, without assuming malice.
- bloopernova 2y agoNote to self: on Monday, add a null character check to pre-commit hooks, and add the same check to pipelines.
- Retr0id 2y agoIt's perfectly normal for binary artifacts to contain null bytes, even long runs of them.
- bloopernova 2y agoYeah, I'd need to figure it out properly, but for unicode text files it should be OK. Good point about the binaries though, thank you!
- hawski 2y agoYou say Unicode, but you mean UTF-8. Now for 16 bit Unicode the story is different :)
- MilStdJunkie 2y agoI mentioned it in a separate parent, but null purge is - for the stuff I work with - completely non-negotiable. Nulls seem to break virtually everything, just by existing. Furthermore, old-timey PDFs are chock full of the things, for God knows what reason, and a huge amount of data I work with are old-timey PDF.
- toast0 2y ago> Furthermore, old-timey PDFs are chock full of the things, for God knows what reason, and a huge amount of data I work with are old-timey PDF. Probably UCS-2/UTF-16 encoding with ascii data.
- MilStdJunkie 2y agoIt's hard to localize. Early Postscript - PDF software was the wild west, particularly when it comes to the text streams. Something I've noticed is that they're used a LOT in things like randlists (bullet lists), tab leaders, other "space that isn't a space". I'm reminded of how you have to use `{empty}` character refs in lightweight markup like Asciidoc to "hold your place" in a list, in case you need the first element in the list to be an admonition or block. Like so: . {empty} + WARNING: Don't catch yourself on fire! + Pour the gasoline. And the equivalent XML which would be something like <procedure> <step> <warning> Don't catch yourself on fire!</warning> <para>Pour the gasoline.</para> </step> </procedure> This is one of those rare cases where the XML is more elegant than the lightweight markup. That hack with `{empty}` bugs me. Anyways, I'm spitballing that these old-timey nulls I'm seeing are being employed in an equivalent way, some sort of internal bespoke workaround to a format restriction.
- thrill 2y agoSo ... the checksum of all the Seinfeld episodes?
- geor9e 2y agoExplain joke please
- themagicteeth 2y agoSeinfeld is a show about nothing http://seinfeldscripts.com/ThePitch.htm http://seinfeldscripts.com/ThePitch.htm
- WhyCause 2y agoSeinfeld was "a show about nothing."
- deleted 2y ago[deleted]
- kristjansson 2y agoSomething like zero_output_file(fh, len(file)) flush() fill_output_file(fh, data) with an oops in line 3?
- AndrewKemendo 2y agoThis should not have passed a competent C/I pipeline for a system in the critical path. I’m not even particularly stringent when it comes to automated test across-the-board but for this level of criticality of system, you need exceptionally good state management To the point where you should not roll to production without an integration test on every environment that you claim to support Like it’s insane to me that this size and criticality of a company doesn’t have a staging or even a development test server that tests all of the possible target images that they claim to support. Who is running stuff over there - total incompetence
- martinky24 2y agoA lot of assumptions here that probably aren't worth making without more info -- For example it could certainly be the case that there was a "real" file that worked and the bug was in the "upload verified artifact to CDN code" or something, at which point it passes a lot of things before the failure. We don't have the answers, but I'm not in a rush to assume that they don't test anything they put out at all on Windows.
- EvanAnderson 2y agoI haven't seen the file, but surely each build artifact should be signed and verified when it's loaded by the client. The failure mode of bit rot / malice in the CDN should be handled.
- gjsman-1000 2y agoPerhaps - but if I made a list of all of the things your company should be doing and didn't, or even things that your side project should be doing and didn't, or even things in your personal life that you should be doing and haven't, I'm sure it would be very long.
- EvanAnderson 2y agoA company deploying kernel-mode code that can render huge numbers of machines unusable should have done better. It's one of those "you had one job" kind of situations. They would be a gigantic target for malware. Imagine pwning a CDN to pwn millions of client computers. The CDN being malicious would be a major threat.
- deleted 2y ago[deleted]
- bryanlarsen 2y agoSegue: What the heck is about Windows files and null characters? I've been almost exclusively dealing with POSIX file systems for the last 30 years, but I'm currently shipping a cross-platform app and a lot of my Windows users are encountering corrupted files which exhibit a bunch of NULs in random places in the file. I've added ugly hacks to deal with them but it'd be nice to get down to root causes. Is there a handy list of things that are safe on POSIX but not on Windows so I can figure out what I'm doing wrong? I'm at the stage where I'm thinking "%$#@ this, I'm never going to write to the Windows file system again, I'm just going to create an SQLite DB and write to that instead". Will that fix my problems?
- nradov 2y agoThe Windows NTFS is safe and reliable. It doesn't corrupt files. You have probably misunderstood the problem.
- bryanlarsen 2y agoHowever I'm using it it certainly isn't. It's certainly quite likely the problem is me, not Windows. Most of the files that are getting corrupted are being written to in an append-only fashion, which is generally one of the mechanisms for writing to files to avoid corruption, at least on POSIX.
- omoikane 2y agoI have observed a file filled NULs that was caused by a power loss in the middle of a write -- my UPS alerted me that utility power is gone, I tried to shutdown cleanly, but the battery died before a clean shutdown completed. This was NTFS on a HDD and not a SSD. I am not saying it happens often, but it does happen once in a while.
- bryanlarsen 2y agoYes, corruption does appear to be correlated with power cycles.
- 2y ago
- neffy 2y agoThere's a claim over on Mastodon from Kevin Beaumont that the file is different on every customer he´s received the file from. https://cyberplace.social/@GossiTheDog/112812454405913406 https://cyberplace.social/@GossiTheDog/112812454405913406 (scroll down a little)
- drewg123 2y agoI thought windows required all kernel modules to be signed..? If there are multiple corrupt copies, rather than just some test escape, how could they have passed the signature verification and been loaded by the kernel?
- dist-epoch 2y agoThis is not even a valid executable. Most likely is not loaded as a driver binary, but instead is some data file used by the CrowdStrike driver.
- Izkata 2y agoSeems like a data file: https://news.ycombinator.com/item?id=41004103 https://news.ycombinator.com/item?id=41004103
- millero 2y agoYes, this fits in with what I heard on the grapevine about this bug from a friend who knows someone working for Crowdstrike. The bug had been sitting there in the kernel driver for years before being triggered by this flawed data, which actually was added in a post-processing step of the configuration update - after it had been tested but before being copied to their update servers for clients to obtain. Apparently, Crowdstrike's test setup was fine for this configuration data itself, but they didn't catch it before it was sent out in production, as they were testing the wrong thing. Hopefully they own up to this, and explain what they're going to do to prevent another global-impact process failure, in whatever post-mortem writeup they may release.
- oldpersonintx 2y ago[dead]
- deleted 2y ago[deleted]
- MilStdJunkie 2y agoHoly smokes. I'm no programmer, but I've built out bazillions of publishing/conversion/analysis systems, and null purge is pretty much the first thing that happens, every time. x00 breaks virtually everything just by existing - like, seriously, markup with one of these damn things will make the rest of the system choke and die as soon as it looks at it. Numpy? Pytorch? XSL? Doesn't matter. cough cough cough GACK And my systems are all halfass, and I don't really know what I'm doing. I can't imagine actual real professionals letting that moulder its way downstream. Maybe their stuff is just way more complex and amazing than I can possibly imagine.
- wormlord 2y agoNot a C programmer, why is 0x00 so bad? It's the string terminator character right?
- bagful 2y agoIndeed, '\0' is the sentinel character for uncounted strings in C, and even if your own counted string implementation is “null-byte clean”, aspects of the underlying system may not be (Windows and Unix filesystems forbid embedded null characters, for example).
- tedunangst 2y agoIt's a byte like any other. You're more likely to see big files full of 0x0 than 0x1, but it's really not so different.
- hawski 2y agoBinary files are full of null bytes it is one of the main criteria of binary file recognition. Also large swaths of null bytes are also common, common enough we have sparse files - files with holes in them. Those holes are all zeroes, but are not allocated in the file system. For an easy example think about a disk image.
- MilStdJunkie 2y agoAh, I'm a dummy. Of course these would be binaries getting chucked around.
- j-wags 2y agoIt's possible that these aren't the original file contents, but rather the result of a manual attempt to stop the bleeding. Someone may have hoped that overwriting the bad file with an all-0 file of the correct size would make the update benign. Or following the "QA was bypassed because there was a critical vulnerability" hypothesis, stopping distribution of the real patch may be an attempt to reduce access to the real data and slow reverse-engineering of the vulnerability.
- 0cf8612b2e1e 2y agoOn the plus side of this disaster, I am holding out some pico-sized hope that maybe organizations will rethink kernel level access. No, random gaming company, you are not good enough to write kernel level anti cheat software.
- majormajor 2y agoI can't imagine gaming software being affected at all, unless MS does a ton of cracking down (and would still probably give hooks for gaming since they have gaming companies in their umbrella). No corporate org is gonna bat an eye at Riot's anti-cheat practices, because they aren't installing LoL on their line of business machines anyway.
- InitialLastName 2y agoRight, MS just paid $75e9 for a company whose main products are competitive multiplayer games. They are never going to be incentivized to compromise that sector by limiting what anti-cheat they can do.
- Y_Y 2y agoThat's 7.5e10 USD in SI.
- InitialLastName 2y agoEngineering notation used for prefix convenience.
- minetest2048 2y agoUntil the malware bring their own compromised signed anti cheat driver on their own, like what happened with Genshin Impact anti cheat mhyprot2
- tgsovlerkhgsel 2y ago> because they aren't installing LoL on their line of business machines anyway But if their business is incompatible with strict software whitelisting, their employees might...
- hn_throwaway_99 2y agoOn a related note, I don't think that it's a coincidence that 2 of the largest tech meltdowns in history (this one and the SolarWinds hack from a few years ago) were both the result of "security software" run amok. (Also sad that both of these companies are based in Austin, which certainly gives Austin's tech scene a black eye). IMO, I think a root cause issue is that the "hacker types" who are most likely to want to start security software companies are also the least likely to want to implement the "boring" pieces of a process-oriented culture. For example, I can't speak so much for CrowdStrike, but it came out the SolarWinds had an egregiously bad security culture at their company. When the root cause comes out about this issue dollars-to-donuts it was just a fast and loose deployment process.
- koliber 2y agoDon't forget heartbleed, a vulnerability in OpenSSL, the software that secures pretty much everything.
- Lennie 2y agoI don't see that as a business problem directly. But I see it as the XKCD 2347 problem.
- NegativeK 2y agoAlternate hypothesis that's not necessarily mutually exclusive: security software tends to need significant and widespread access. That means that fuckups and compromises tend to be more impactful.
- hn_throwaway_99 2y ago100% agree with that. The thing that baffles me a bit, then, is that if you are writing software that is so critical and can have such a catastrophic impact when things go wrong, that you double and triple check everything you do - what you DON'T do is use the same level of care you may use with some social media CRUD app (move fast and break things and all that...) To emphasize, I'm really just thinking about the bad practices that were reported after the SolarWinds hack (password of "solarwinds123" and a bunch of other insider reports), so I can't say that totally applies to CrowdStrike, but in general I don't feel like these companies that can have such a catastrophic impact take appropriate care of their responsibilities.
- OutOfHere 2y agoThis looks like a test file that got deployed. Perhaps a QA test was newly added which ran and overwrote the build. This is all I can think of.
- OutOfHere 2y agoThis could happen if there was a test for the above QA test. Such a meta test would necessarily have to write a test file to the same path as the build path. Unfortunately, the meta tests could have got incorporated into the same suite as the regular tests. Someone then could have decided to activate a test order randomizer for fun. The random order finally hit the anti-jackpot.
- reify 2y ago[flagged]
- markus_zhang 2y agoI'm starting to think that the timing (Friday) and the scale as well as other things (like this finding) might -- just might point to a bad actor. We will probably have to wait for CS' own report.
- breadwinner 2y agoI blame Microsoft. Why? Because they rely on third parties to fill in their gaps. When I buy a Mac it already has drivers for my printers, but not if I buy a Windows PC. Some of these printers drivers are 250 MB, which is a crazy size for a driver. If it is more than a few 100 KB it means the manufacturer does not know how to make a driver software. Microsoft should make it unnecessary to rely on crappy third party software so much.
- luuurker 2y agoCrowdStrike's mess up is CrowdStrike's fault, not Microsoft's. We might not like the way Windows works, but it usually works fine and more restrictive systems also have downsides. In any case, it was CrowdStrike who dropped the ball and created this mess. I don't like what Microsoft is doing with Windows and only use it for gaming (I'm glad Linux is becoming a good option for that), so I'm far from being a "Microsoft fan", but Windows is very good at installing the software needed. Plug a GPU, mouse, etc, from any well known brand and it should work without you doing much. I didn't have to install anything on my Windows PC (or my MBP) last time I bought a new printer (Epson). The option to let Windows install the drivers needed is enabled though... some people disable that.
- breadwinner 2y ago> CrowdStrike's mess up is CrowdStrike's fault, not Microsoft's. Disagree. It is everyone's fault. It is CrowdStrike's fault for not testing their product. It is Microsoft's fault for allowing CrowdStrike to mess with kernel and not vetting such critical third parties. It is the end customers' fault for installing crapware and not vetting the vendor.
- yoavm 2y agoso now we're vouching for more restrictive operating systems? the last thing I want is an operating system that can only install vetted apps, and that these apps are restricted even if I provide my root password.
- cmrdporcupine 2y agoIs there not responsibility at some level as well to Microsoft for having a kernel which even loaded this? Not just because of the apparent corruption, but also ... it was, I heard.. signed and given a bit of an MS blessing. This crap shouldn't be run in kernel space. But putting that aside, we need kernels that will be resilient to and reject this stuff.
- ale42 2y agoThe thing is that, despite the file has a confusing .sys extension, it's not the driver, but rather a data file loaded by the Crowdstrike driver.
- ThinkBeat 2y agoMaybe Crowdstrike has adopted the modern ethos Move fast and break things With continuous integration we ship a thousand times a day. fuck QA.
- motohagiography 2y agoconspiracy prediction: I don't think CS will give a complete public RCA on it, but I do think the impact and crisis will be a pretext for granting new internet governance powers, maybe via EO, or a co-ordinated international response via the UN/ITU and the EU.
- cherryteastain 2y agoEU recently passed a law in this domain: https://www.eiopa.europa.eu/digital-operational-resilience-act-dora_en https://www.eiopa.europa.eu/digital-operational-resilience-a...
- motohagiography 2y agoDORA is coincidental, but worth watching as a potential source of authority under some partisan disinformation initiatives. security in general is governance, where previously it had a high bar for competence to practice, and you can see it getting stacked in institutions of people without practitioner competence who can be relied on to defer to narrative. as soon as security can come fully uncoupled from physical competence and the factual constraints of engineering, it will get subsumed into the managerialist hivemind as a rhetorical lever on narrative actuated cogs, and become as meaningless as everything else they do in service to administrative power for its own sake. I have a real problem with these people. :)
- kragen 2y agothis seems like the second or third test file any qa person would have tried, after an empty file and maybe a minimal valid file. the level of pervasive incompetence implied here is staggering in a market where companies compete by impressing nontechnical upper management with presentations, it should be no surprise that technically competent companies have no advantage over incompetent ones i recently read through the craig wright decision https://www.judiciary.uk/judgments/copa-v-wright/ https://www.judiciary.uk/judgments/copa-v-wright/ (the guy who fraudulently claimed to be satoshi nakamoto) and he lacked even the most basic technical competence in the fields where he was supposedly a world-class specialist (decompiling malware to c); he didn't know what 'unsigned' meant when questioned on the witness stand. he'd been doing infosec work for big companies going back to the 90s. he'd apparently been tricking people with technobabble and rigged demos and forged documents for his entire career george kurtz, ceo and founder of crowdstrike, was the cto of mcafee when they did the exact same thing 14 years ago: https://old.reddit.com/r/sysadmin/comments/1e78l0g/can_crowdstrike_survive_this_impact/ldz4n6h/?share_id=rafKW0OThY1VGL08GtbWb https://old.reddit.com/r/sysadmin/comments/1e78l0g/can_crowd... https://en.wikipedia.org/wiki/George_Kurtz https://en.wikipedia.org/wiki/George_Kurtz crowdstrike itself caused the same problem on debian stable three months ago: https://old.reddit.com/r/debian/comments/1c8db7l/linuximage61020_killed_all_my_debian_vms/ https://old.reddit.com/r/debian/comments/1c8db7l/linuximage6... it's horrifying that pci compliance regulations have injected crowdstrike (and antivirus) into virtually every aspect of today's it infrastructure
- GardenLetter27 2y agoAlso ironic that the compliance ended up introducing the biggest vulnerability as a massive single point of failure. But that's government regulation for you.
- kragen 2y agopci-dss is not a government agency but it might as well be; it's a collusion between visa, mastercard, american express, discover, and jcb to prevent them from having different data security standards (and therefore being able to compete on security)
- olliej 2y agoWe can argue all we want about CI infrastructure, manual testing, test nets/deployment, staged deployment. All of that is secondary: they wrote and shipped code that blindly loaded and tried to parse content from the network, and crashed when that failed. In kernel mode. Honestly it’s probably good that this happened, because presumably someone malicious could use this level of broken logic to compromise kernel space. Certainly the trust they put in the safety of parsing content downloaded from the internet makes me wonder about the correctness of their code for reading data from userspace.
- bb88 2y agoWe've had security software in the past break software compilation in this method by replacing entire files with zeros. I'm not saying this is the case, but it wouldn't surprise me if it were. Basically the linker couldn't open the file on windows (because it was locked by another process scanning it), and didn't error. Just replaced the object code to be linked with zeros. People couldn't figure out what was wrong until they opened a debugger and saw large chunks of object code replaced with zeros.
- heywire 2y agoThe number of times I encounter issues with Visual Studio not being able to write to the file I just compiled, ugh... We've even got an mt.exe wrapper that adds retries because AV likes to grab hold of newly created binaries right around the time mt.exe needs to write to it. Even our CI build process ends up failing randomly due to this.
- slashdave 2y agoI don't get it. Shouldn't the file have a standard format, with a required header and checksum (among other things), that the driver checks before executing?
- nikau 2y agoyou would think that...
- aaron695 2y ago[dead]
- fhub 2y agoAnytime critical infrastructure goes down I always have a fleeting thought back to “Spy Game” movie where the CIA cut power to part of a Chinese city to help with a prison escape.
- Thaxll 2y agoI'm not versed enough into windows loading dll / driver, but isn't the caller able to handle that situation? Or windows itself? Does loading an empty file driver can be handled in a way that it does not make the OS crash?
- yamumsahoe 2y agothats a lot of prod to test in.
- jmspring 2y agoPoor testing. But we also need to stop CISOs, etc doing "checkbox compliance" and installing every 3rd party thing on employee laptops. My prior employer, there were literally 13 things installed for "laptop security" - 1/2 of them overlapped. Developers had the same policy as an AE and as a Sales Engineer as well as an HR person. Crowdstrike was one of the worst. Updating third party packages in go was 30-40% faster in an emulated arm64 VM (qemu) - virtualized disk / disk just a large file - on an Intel MBP compared to doing the same operation on the native system in OSX.
- snowwrestler 2y agoI have flagged this because it is wrong, and there is no other way to kick it off the front page. Finding a file full of zeroes on a broken computer does not mean it was shipped as all zeroes! https://x.com/craiu/status/1814339965347610863 https://x.com/craiu/status/1814339965347610863 https://x.com/cyb3rops/status/1814329155833516492 https://x.com/cyb3rops/status/1814329155833516492
- snowwrestler 2y agoFinally confirmed by CrowdStrike themselves: https://www.crowdstrike.com/blog/tech-analysis-channel-file-may-contain-null-bytes/ https://www.crowdstrike.com/blog/tech-analysis-channel-file-...
- execveat 2y agoCrowdStrike does this trick where it replaces the file (being transferred over a network socket) with zeroes if it matches the malware signature. Assuming that these are the malware signature files themselves, a match wouldn't be surprising.
- bombcar 2y agoCrowStrike foot gunning itself would be amusing, if expected.
- kristjansson 2y agoFar and away the most entertaining of the possible root causes.
- tsavo 2y agoThis actually makes the most sense, and would help explain how the error didn't occur during testing (in good faith, I assume it was tested). In testing, the dev may have worked from their primary to deploy the update to a series of secondary drives, then sequential performed a test boot from each secondary drive configured for each supported OS version. A shortcut/quick way to test that would've bypassed how their product updates in customer environments, also bypassing checks their software may have performed (in this case, overwriting their own file's contents).
- deleted 2y ago[deleted]
- keketi 2y agoFound this post on 4chan's technology board: What actually happened was two-folded: Step 1: CSAgent.sys is a certified durgasoft kernel driver that parses clownstrike virus definition files Step 2: Clownstrike never fixed cases for malformed virus definition files that could trigger an invalid memory access in CSAgent.sys Step 3: Clownstrike ships the buggy CSAgent.sys and it works for months without problems on customer computers with good virus definition files Step 4: For some reason the webserver serving virus definition files to all endpoints started serving malformed virus definition files, sometimes they were all blank or filled with random bytes Step 5: All clownstrike updaters now download malformed LE NEXT GEN APT PROTECTION CLOUD AI LIVE UPDATES into C:\Windows\System32\drivers\clownstrike Step 6: CSAgent.sys reloads virus definition files Step 7: CSAgent.sys parses them and crashes with PAGE_FAULT_IN_NONPAGED_AREA (in kernel that means memory at an oopsie address was accessed) Step 8: Computer BSOD and reboots Step 9: CSAgent.sys loads virus definition files Step 10: Goto Step 7 The kernel driver was a ticking timebomb just waiting for a bug in the CDN. I think it was some funny caching bug with le cloud http loadbalancer. Users reported that their C-00000291-00000000-00000032.sys contained random shit and actual real files that were a completely different part of the software, like US localization files. You can see it in the diff between CSAgent.sys_18511.sys and CSAgent.sys_18513.sys, they changed size checks and increased sizes of buffers so that future malformed virus definition files wouldnt crash.
- denisdamico 2y agoStep 4. If this has occurred so many times, it is a hacker attack, targeting the poorly written Kernel driver. (But hey, right now at this election moment, with Trump's beloved company. It could be a show of strength)
- justinclift 2y ago> For some reason the webserver serving virus definition files to all endpoints started serving malformed virus definition files, sometimes they were all blank or filled with random bytes That sounds like some virus protection on the server was stopping the reads from disk, and instead of throwing an error it was providing 0's for the output data instead. It'll be funny if there's actually some antivirus package that caused it. ;)
- denisdamico 2y agoI assume the Kernel driver itself has been signed, tested and verified by MS. The channel update files would be the responsibility of the CS and contain most of the information, including urgent (untested) information. This CS kernel driver is there to inspect, collect everything, receive CS settings and commands, and act in privileged mode. It's not a so complex system, the intelligence is in CS, that's why MS partners with them. But the code in the driver could at least do a perfect input check. Shits happens. Blaming MS: I want to believe they do extensive sandbox testing, but in this case they could do better canary deployments, staged updates and not affect worldwide.
- commandersaki 2y agoI wonder if this is linked to the Azure outage that happened just prior to the Crowdstrike incident.
- havkom 2y agoLots of very strong opinions here what they should have done in terms of code/pipelines/etx from people who have no idea how CrownStrikes stack or build systems works and people who have no clue how to write that type of software themselves (including Windows kerneln modules).
- ExoticPearTree 2y agoSo basically they don't do input validation. Any file signed (maybe) by CS is treated as valid and the kernel driver loads it. Wow. I'm curious how this scenario, malformed definitions file signed by CS itself is not verified in any way on the endpoint. Like a try {} catch something... I mean, the failure is spectacular in its simplicity.
- kmxr 2y agoI am going to take a guess on how the CrowdStrike event might have happened. This is based on two (2) observations: (1) CS said they have always followed the same process in the past without issues (2) (personal observation) When certain NAS drives get full and you modify a file, it ends up with all NULL bytes. Here's my guess on how their process was: Typically, when their content file is ready, they push it to their farm of QA test machines for testing and also to this NAS drive that's ready for deployment. And when all of their test deployments are successful, they press this BIG RED button that picks up this content file from the NAS drive for global deployment. During their prior deployments, this NAS drive had plenty of free space and deployments went well. Now, when the NAS was full, upon copy, it produces a file with all NULL bytes. In summary, the content file they tested with was not the file that got globally deployed.