6 ms·
It seems baffling that Microsoft is getting heat for this. They didn't cause the issue, a third party vendor's software did. Even if you were trying to make an
by gruez 2y ago
It seems baffling that Microsoft is getting heat for this. They didn't cause the issue, a third party vendor's software did. Even if you were trying to make an argument of "if we had more diversity it wouldn't be as bad", shouldn't you be focusing on the EDR vendors rather than the OS vendor?
- dlahoda 2y agois not it the case that kernel drivers are signed and approved by microsoft?
- bilekas 2y agoI'm not aware of Microsoft signing off on vendor kernal drivers code changes. Driver signing is not a Quality Assurance measure. It's an identify measure.
- AstralStorm 2y agoMore importantly, the code signing is a security measure. You can in theory vet the signature database by hand preventing the execution of unapproved drivers. Microsoft normally puts it in revocation mode only, with bad drivers with unfixed holes being blocked.
- gruez 2y agoWould the disaster have been prevented if there was no code signing? Or are you arguing that Microsoft should scrutinize code even harder?
- halfcat 2y agoMicrosoft isn’t scrutinizing this code, at all, right? It’s not like they’re doing code reviews of every vendor that does code signing in Windows.
- gruez 2y agoNot code reviews but the driver is tested https://en.wikipedia.org/wiki/WHQL_Testing https://en.wikipedia.org/wiki/WHQL_Testing
- dlahoda 2y agohttps://chatgpt.com/share/8eca1a08-062f-4474-9310-95444bab670f https://chatgpt.com/share/8eca1a08-062f-4474-9310-95444bab67... said ms tests drivers
- thaithrowawayy 2y agoReally? Is this the future of developers to just blindly trust a statistical text generator? It's cool that you know how to ask a question on ChatGPT. But don't share it as if it's the truth on an obscure, halluncinable factoid.
- k8sToGo 2y agoYou are confusing notarized (what Apple does) and signed (what both do). The latter just allows to confirm legitimacy.
- FireBeyond 2y agoI guarantee you Apple will take zero responsibility for a failing or issue with third party software, notwithstanding the purported difference between notarized and signed.
- rty32 2y ago[flagged]
- scopeh 2y agoI think it was essentially a data/config file that was corrupt not a new driver kernel. nothing really to be signed and approved by mircosoft.
- slowmovintarget 2y agoDrivers yes, virus signature files or configuration files pushed by third parties, no.
- kermatt 2y agoRegulators who may not understand the difference between an OS and a driver.
- teekert 2y agoSo normies in governments are finally seeing the issues, it’s for the wrong reason indeed, but is it though? It could have been an MS trip up and then it may have been worse even…
- slowmovintarget 2y agoMicrosoft personnel are usually more careful with existential risks... usually.
- guidedlight 2y agoMicrosoft are signing kernel drivers for over-the-air updates pushed by vendors. Whilst telling their corporate customers that using Microsoft InTune will allow them to be in full control of their configuration management.
- gruez 2y ago>Microsoft are signing kernel drivers for over-the-air updates pushed by vendors. The crash was caused by a configuration update pushed by crowdstrike, not a new driver. >Whilst telling their corporate customers that using Microsoft InTune will allow them to be in full control of their configuration management. How is this relevant? This wasn't caused by some sysadmin that goofed a config change using intune, it's something pushed by crowdstrike itself.
- guidedlight 2y ago> The crash was caused by a configuration update pushed by crowdstrike, not a new driver. Microsoft didn’t do enough due diligence on the behaviour of CrowdStrike updates. It shouldn’t allow out-of-band updates. Third-parties should not be signing code that allows third-parties to reach into corporate services and push files. Microsoft InTune is the mechanism that all configuration updates should use. It appears to me that Microsoft makes it harder for third-parties to update an Xbox game, than the configuration of a kernel driver.
- gruez 2y ago>Microsoft didn’t do enough due diligence on the behaviour of CrowdStrike updates. It shouldn’t allow out-of-band updates. You want Microsoft to be doing code reviews of third party software? That might have prevented this disaster but would get them in hot waters for other reasons (eg. anti-competition accusations). Not even Apple gatekeeps that hard. >Third-parties should not be signing code that allows third-parties to reach into corporate services and push files. So dropbox should be banned as well? It's also a third party service that pushes files onto computers. >Microsoft InTune is the mechanism that all configuration updates should use. Okay, so crowdstrike pushes its virus definition updates via intune instead. The bad file still lands on computers, the kernel mode driver still crashes before the computer can boot and the computer is bricked. How is this any better?
- jeroenhd 2y agoPart of Microsoft's problem is that right before the CrowdStrike outage, there was an outage of some large Microsoft services, making the two look connected. It didn't help that a lot of Microsoft support consultants also ran CrowdStrike laptops, getting affected right when customers were. But this is quite typical. Microsoft generally doesn't direct blame to anyone (anyone non-malicious at least) but in this case I feel like they really should.
- thayne 2y agoIs it baffling? Only windows computers were affected, so this provides ammunition for people who are already concerned about the prevalence of windows. I think this is more politically motivated than actually trying to address the real problem. However I do, think Microsoft deserves some blame, since if Windows was more secure by default there would be less need for 3rd party anti-malware software that can fail so catastrophically. But, Crowdstrike certainly is more to blame for thos particular disaster.
- Rinzler89 2y ago>Only windows computers were affected Because only Crowdstrike's Windows release was broken by them and they didn't fuck it up on the other OS. How is this Window's fault? It's not like that tool was binary cross platform compatible for all operating systems, like Electron VS Code, in order to put the blame on the OS. It's basically a complete different tool tailored to each OS kernel, under the same brand name. > if Windows was more secure by default there would be less need for 3rd party anti-malware software that can fail so catastrophicall Which Windows security vulnerability needed the use of Crowsdstrike's tool? How do you measure this security threshold at which you say an OS is so secure that you don't need a third party EDR vendor anymore? You see, here's the problem. That threshold does not exist, since security it's a continuous arms race, and EDR vendors are incentivize to scaremonger you into thinking Windows or whatever OS you use is so insecure and your employees so incompetent and untrustworthy with what they do on their computers that your multi billion dollar won't be able to survive if you don't buy their tool. And that's partly true to some extents. You can have the most secure OS in the world, but if Bob from accounting runs a script he found online as root then all that OS security was for nothing since the OS can't magically detect that Bob is an idiot and should not run that command as sudo even though Bob has the right credentials for it. That's why companies hire these digital watchmen, to watch over their systems for suspicios behavior since neither Windows, nor MacOS, nor Linux can do this out of the box. The question remains who watches the watchmen? You're now trusting an EDR vendor with the keys to your kingdom instead of Bob.
- thayne 2y ago> Because only Crowdstrike's Windows release was broken by them and they didn't fuck it up on the other OS. How is this Window's fault? First of all, whether it is actually Window's fault or not isn't the point. What matters is the perception by the general public and policy makers. And maybe it could have just as easily happened on another OS, but the reason why is somewhat technical, so people who have an agenda can spin it as being MS's fault. Secondly, it is possible, although I don't think terribly probable, that the bug was influenced by some quirk of the windows platform, although it is impossible to know without more details of the exact nature of the bug.
- ehsankia 2y agoI'm curious what people think, but while obviously CrowdStrike caused the breakage, does the Operating System not have some responsibility in not allowing such outages to happen? Especially if it's an enterprise product? Ideas: 1. Microsoft themselves could potentially enforce a gradual rollout on updates (did the update go through windows updates?) 2. Have better automatic recovery options, could windows have detected the bad driver and reverted it automatically? 3. just generally be more resilient, why should one bad driver take everything down?
- gruez 2y ago>1. Microsoft themselves could potentially enforce a gradual rollout on updates (did the update go through windows updates?) No. The crash was caused by a "configuration update", not through code pushed through windows update. https://www.crowdstrike.com/blog/falcon-update-for-windows-hosts-technical-details/ https://www.crowdstrike.com/blog/falcon-update-for-windows-h... >2. Have better automatic recovery options, could windows have detected the bad driver and reverted it automatically? see: https://news.ycombinator.com/item?id=41019743 https://news.ycombinator.com/item?id=41019743 >3. just generally be more resilient, why should one bad driver take everything down? It's kernel mode. With great power comes great responsibility. Blaming the OS is like blaming linux that you can sudo rm -rf /.
- xboxnolifes 2y agoI'd expect the HN crowd to not give Microsoft beef here or want them to lock down the OS more. At least, that would be consistent with usual comments about letting users control their OS. Companies chose (with some regulatory pressure) to install CrowdStrike at the kernel level after all.
- ehsankia 2y agoI didn't talk about locking down, but just generally building more resilience/recovery into the OS. Also, from the little I saw, CrowdStrike got around certifying every driver update by pushing updates through "update" files that are read and executed by the driver. That seems like a huge hack to bypass certification and likely shouldn't be allowed?
- rewgs 2y agoThere is a frustrating amount of nuance being lost in this discussion, and as usual it's devolving into tribalism. However, I'll say that, while this clearly is not Microsoft's fault, the realization of just how much critical infrastructure is running on Windows -- let alone Windows that's connected to the internet and has automatic updates enabled -- was sobering. Are kernel mode drives maybe a bad idea? Yes! Should Windows be able to automatically roll back if a kernel mode driver fails? Yes! Should CrowdStrike have tested before pushing out the update? Yes! Should hospitals, police stations, airlines, etc be testing any and all updates that come their way prior to releasing them onto the rest of their fleets? Yes! And so on and so on. The failures here are legion. I can at least say for myself that all of that (and more) is what I'm grappling with today, rather than the direct, root cause and content of the issue itself. I say this entirely with as little tribalism or bias as I can muster: Windows should not be the foundation upon which critical infrastructure is built. It is a bad OS. It is a rickety, insecure mess; this observation (note that I did not say opinion) has only ever increased with time. The NT kernel, I hear, is great. I generally trust those who say this, as they tend to be demonstrably much smarter than I am, especially with regards to kernel design, which is something I know next to nothing about. But Windows? Nah man. Maybe there's a good kernel running the show, but the OS as a whole is a shitshow. Yes, this exact same problem could have (and has) happened on Linux; the OS is indeed not the culprit. But the understanding that this one, single, no good very bad OS is responsible for this much critical infrastructure going from an abstract worry to a real, concrete horror show has, I think, hit a lot of people, leading to a poorly-pivoted change of subject.
- gruez 2y ago>There is a frustrating amount of nuance being lost in this discussion, and as usual it's devolving into tribalism. It's ironic that you're saying this given the points you're making below. Let's go through them: >Are kernel mode drives maybe a bad idea? Yes! You can't have a EDR product that isn't kernel mode. Otherwise it's trivial for malware to evade (eg. by being kernel mode themselves). >Should Windows be able to automatically roll back if a kernel mode driver fails? Yes! see: https://news.ycombinator.com/item?id=41019743 https://news.ycombinator.com/item?id=41019743 >Should hospitals, police stations, airlines, etc be testing any and all updates that come their way prior to releasing them onto the rest of their fleets? Yes! Getting companies to test updates is already like pulling teeth. Besides, crowdstrike said the update they pushed was "designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks". Is this something you really want to sit on for testing, which might take weeks or months? >Windows should not be the foundation upon which critical infrastructure is built. It is a bad OS. It is a rickety, insecure mess; this observation (note that I did not say opinion) has only ever increased with time. >The NT kernel, I hear, is great. I generally trust those who say this, as they tend to be demonstrably much smarter than I am, especially with regards to kernel design, which is something I know next to nothing about. >But Windows? Nah man. Maybe there's a good kernel running the show, but the OS as a whole is a shitshow. What specific security issues do you think windows/NT kernel has? Moreover, how is windows being a "shitshow" relevant to the question of resiliency or dependence? Don't get me wrong, windows spying on you or using dark patterns to get you to use Edge or whatever isn't great, but it's a weird thing to bring up in a discussion about how airports run on windows, and reeks of "don't let a disaster go to waste" on the part of the anti-windows tribe.
- mixmastamyk 2y agoBesides cloud outages, there's also been quite a few security disasters recently, one with the Russians all up in Uncle Sam's business. There's no reason for MS to have such a lock on governments, when their security and reliability is worst in the industry among BigTech. Even "Blue Hat" could do better.
- dissent 2y agoI go the other way, it's absolutely their fault. Their historical business model of monopolizing the enterprise while outsourcing security is at the root of all this. The same thing could happen with Crowdstrike on other platforms, but it's only on Windows where these suites are considered essential.
- gruez 2y ago>but it's only on Windows where these suites are considered essential. They're "considered essential" for all corporate machines for compliance and CYA reasons. If you're the CISO/CTO and a hack occurred on your watch, do you really want to be the one telling the board that you didn't need EDR on your linux machines because linux is secure?
- dissent 2y agoIt's a culture that has grown up around a dominant platform that had little or no security built in. Where I work, Windows is heavily locked down with all sorts of security junk on it, but iOS gets by with minimal MDM.