12 ms·
> We plan to work with the anti-malware ecosystem to take advantage of these integrated features to modernize their approach, helping to support and even increa
by rdtsc 2y ago
> We plan to work with the anti-malware ecosystem to take advantage of these integrated features to modernize their approach, helping to support and even increase security along with reliability.
> Providing safe rollout guidance, best practices, and technologies to make it safer to perform updates to security products.
> Reducing the need for kernel drivers to access important security data.
They are being as diplomatic as they can, but it's definitely a slap to CS. Read as "they don't know how to roll things out, they need guidance on basic QA practices, we'll happily teach them...". Then, they list a set of facilities running in user-mode to avoid needing to run as many things in kernel mode.
I would be interested what the water cooler discussion about CS was like inside Microsoft. Especially in teams needed to respond to customers about "Your windows OS is broken, our hospital patients are suffering...".
- notepad0x90 2y agoI must disagree with that take, your last quoted sentence is in response to all the supposed self-proclaimed experts asking "why does it need kernel access", the ones before that is to limit their own liability. What I've heard from people in the industry is not this silly "oh no, crowdstrike is so incompetent" b.s. that is being spread on sites like HN and reddit but more of an empathic "it could have been us" sentiment. In this write up as well, Microsoft knows they have caused their share of outages, it is a technical write-up but in part, it is to cover their bases for government investigations and lawsuits that will arise from this incident. And in part, they are also responsible for recovering from third-party driver errors and repeated boot failures caused by faulty drivers.
- retrochameleon 2y agoCrowdStrike blamed their test software, but in the same breath revealed that they haven't been using any canary deployments. The bug that caused all this was present in their kernel driver for a long time. For being such a large cybersecurity player and deploying updates to 8.5 million devices, their quality control practices are embarrasingly lacking.
- rvnx 2y agoClearly incompetence to deploy from 0 to 8 million devices without any gradual rollout. That goes even further, because apparently they were fully blind and didn't have crash metrics. "Ok we push the update, and pray".
- galangalalgol 2y agoI think it is past incompetence, and on into negligence. Given the stories we have heard here about emergency service failures it is likely that people died. When people die due to negligence isn't that usually criminal?
- rvnx 2y agoCan't agree more, you found the right words.
- binkHN 2y agoAnd this is how the lawsuits will start.
- SoftTalker 2y agoWho is negligent though? Crowdstrike, or the emergency services that are using an OS that requires third party endpoint security right out of the box in order to be safely used, or the company that makes and sells that OS?
- crazygringo 2y agoWhy not both? Crowdstrike, for negligently not rolling out updates gradually. And emergency services, if they don't have robust fallback procedures/systems for when their IT system goes down. I mean it's totally fine if regular doctor's visits get postponed, but 911 should never go down just because their computers down. Just like aircraft have redundant systems, so too should 911. (The company that makes and sells the OS -- I don't see any negligence there, in this case. If security software fundamentally requires running at the kernel level and Microsoft allows that, I don't see how Microsoft can be at fault.)
- gjsman-1000 2y agoMicrosoft should be sued, for literally having blood on their hands. There was an easily mitigated design flaw in Windows that would have greatly blunted the impact. https://news.ycombinator.com/item?id=41095788 https://news.ycombinator.com/item?id=41095788
- deleted 2y ago[deleted]
- freehorse 2y agoIf "it could have been them", then I would like to read such professionals write exactly about how to avoid having a global outage like this again, rather than "showing empathy" with a corporation. Or do we just leave it up to luck, and if "it happens to them too" in a month or year, oopsies? What about which practices could be improved?
- michaelt 2y agoAnyone in the industry could have a bug get through testing. Some companies could have a severe and readily reproducible bug get through testing. A few of those companies have a hand-rolled update mechanism, and can accidentally break their ability to roll back a bad release. A few of those companies are in a position to push a release that breaks not only their own software, but the entire OS. Very few companies in that position would roll out to 100% of client machines in a single worldwide deployment.
- gnfargbl 2y agoIt didn't read as particularly diplomatic to me. In particular, this paragraph.. > It is possible today for security tools to balance security and reliability. For example, security vendors can use minimal sensors that run in kernel mode for data collection and enforcement limiting exposure to availability issues. The remainder of the key product functionality includes managing updates, parsing content, and other operations can occur isolated within user mode where recoverability is possible. ...was about as close to tetchy as a post like this would ever get. Basically they are saying "there was no good reason at all why CrowdStrike had to put so much code inside the actual kernel." And with the benefit of hindsight, it's a strong point.
- ffhhj 2y ago> there was no good reason at all why CrowdStrike Their business is corporate spyware to surveil employees, ofcourse they'll use any tactic to make it work, that's the why. And their EULA states there is no liability for the company: https://www.crowdstrike.com/terms-conditions/ https://www.crowdstrike.com/terms-conditions/ Dirty policies on top of dirty practices.
- Rinzler89 2y ago>Their business is corporate spyware to surveil employees What?! Anything you do on your corporate provided laptop is always gonna be logged by IT for security in every large company everywhere, that's news to nobody, but your company doesn't care that you use your corpo laptop to book your vacation, IT has better things to do than narc on you for that. If your boss wants to actually spy on you they don't need Crowdstrike, there's other SW dedicated for that depending on the laws in your jurisdiction but that' not what Crowdstrike is for. If you want complete privacy from your employer, just use your personal machine for your private activities instead of your work laptop, why is this so hard?
- deleted 2y ago[deleted]
- userbinator 2y ago
- holsta 2y ago> they need guidance on basic QA practices Microsoft has a loooong history of botched (security) updates, so I'm not hopeful they can teach Crowdstrike much.
- SoftTalker 2y agoYes, quite the epitome of throwing stones from a glass house.
- Rinzler89 2y agoDo you happen to have a list of that "loooong history" of botched (security) updates? I can only find a couple of examples after googling, which a bit smaller than a "loooong history" you're talking about, so unless Microsoft is paying Google to delete results, maybe you're mistaken.
- SoftTalker 2y agoThis is a company whose OS could not even be installed on a live network without getting rooted within a few minutes. Anybody who was paying attention knew that you didn't use any new Windows release until at least the first service pack had come out. Granted that was a while back but painful memories die hard.
- Rinzler89 2y ago>This is a company whose OS could not even be installed on a live network without getting rooted within a few minutes. That was WIndows XP 20 years ago. Please bring arguments about modern Window 11 security which is the current up to date product they're selling and supporting not scenarios that haven't happened in 20 years.
- clwg 2y agoFirst thing that comes to mind is that Recall stuff from a month ago, they also release updates[0] that crash machines. [0] https://www.tomsguide.com/news/windows-11-update-causing-blue-screen-of-death-how-to-fix-it https://www.tomsguide.com/news/windows-11-update-causing-blu...
- f001 2y agoI can tell you they’re quite unhappy about it. Have a friend working there who frustratedly says it wasn’t their fault every-time it comes up. Which is quite often and at every social occasion since.
- fishywang 2y agobut it's kind of their fault? they designed the api that way, they decided what can be done in userland and what must be done via kernel. they at least _allowed_ it to happen every time.
- lozenge 2y agoYou can't just let people do anything from userland, the performance would tank. As for restricting kernelland, EU competition regulators would not be happy if MS was the only one able to write anti virus software that runs in kernelland.
- deleted 2y ago[deleted]
- justinclift 2y ago[flagged]
- throwaway237289 2y ago[flagged]
- justinclift 2y ago[flagged]
- pixl97 2y agoThere is literally a ton of existing software out there that is keeping MS from doing exactly that. When it comes to avoiding breaking legacy applications MS scores far higher than any other operating systems out there.
- lupusreal 2y agoWhy are they being diplomatic, instead of plainly stating their contempt and revoking CS's driver/etc signing keys? Doing so would help to repair the reputational harm that CrowdStrike inflicted on Windows. Are their lawyers telling them they can't impede CrowdStrike even though CrowdStrike is breaking Microsoft's product? They should do it anyway and dare CS to take it to court so they can publicly humiliate CS by dragging all the dirty details of their incompetence out.
- Aeolun 2y agoPeople are free to install kernel modules. It shouldn’t be up to microsoft to stop them from doing so.
- nimbius 2y agothis isnt even the first time its happened. Crowdstrike has killed an OS every month for the past four months. At this point they are a threat actor. if you havent kicked their amateur-hour software out of your infrastructure by now, chances are good senior management and engineering have at least considered it formally. https://en.wikipedia.org/wiki/CrowdStrike#Severe_outage_incidents https://en.wikipedia.org/wiki/CrowdStrike#Severe_outage_inci...
- metadat 2y agoThat incident list is damning. Is senior leadership asleep at the wheel, or how can this many incidents possibly happen every 30 days for months on end? If leadership really cared, they'd make sure post-mortems and other best practices are in place to reduce the frequency. Unfortunately, the executive disconnect isn't new. It's actually uncommon that they care about the reality for end users and customers (which is antithical to my entire ethos, hence why I get paid the medium bucks). Why bother waking up and going to work everyday unless you are contributing in some way to sustaining a better future for everyone? It's actually great for marketing and it's already going to be a tough 100+ years from today for our children, even with our collective care. P.s. People can be so selfish, it kind of breaks my brain but not really. Have you seen the CO2 emissions visualization from NASA this week? It was a wakeup call for me. 'Tremendous' NASA Video Shows CO2 Spewing from US into Earth's Atmosphere https://www.newsweek.com/nasa-video-carbon-dioxide-co2-emissions-1929622 https://www.newsweek.com/nasa-video-carbon-dioxide-co2-emiss... It's concerning.. and caught no traction.. http://news.ycombinator.com/item?id=41064029 http://news.ycombinator.com/item?id=41064029
- swasheck 2y agohere’s a fun connection: https://x.com/anshelsag/status/1814426186933776846 https://x.com/anshelsag/status/1814426186933776846 “ For those who don't remember, in 2010, McAfee had a colossal glitch with Windows XP that took down a good part of the internet. The man who was McAfee's CTO at that time is now the CEO of Crowdstrike. The McAfee incident cost the company so much they ended up selling to Intel.” so yeah, “leadership” (and that’s a loose term) doesn’t seem supremely concerned about much more than earnings
- blackoil 2y agoMS should have something like Project Zero for Windows applications and drivers. Any app on more than 1-5% PC should be tested and fuzzed and ... And the vendor than pressured into fixing the issues. Even if it is not technically their fault, it is definitely optics problem for MS, half of the world refers it as Windows blue screen issue.
- fragmede 2y agoRaymond Chen: That Time We Bought EVERYTHING at Egghead. https://youtu.be/6m_Im7J9Iaw?si=q8jLBefEdgm-PrrZ https://youtu.be/6m_Im7J9Iaw?si=q8jLBefEdgm-PrrZ
- pimlottc 2y agoBlogpost version: https://devblogs.microsoft.com/oldnewthing/20050824-11/?p=34463 https://devblogs.microsoft.com/oldnewthing/20050824-11/?p=34...
- MBCook 2y ago> And the vendor than pressured into fixing the issues How would Microsoft apply pressure? Short of publicly shaming them what power do they have?
- blackoil 2y agoumm. Give a x days deadline and make after it public like Project 0 works, threaten to take away "Verified by MS" badge or create a WhatsApp group of Fortune 500 CIOs and badmouth in it.
- 9dev 2y agoBoth of these have legal percussions: Microsoft could very well be called a competitor of CS, so they cannot force them to do something without getting accused of abusing their market position; and a publicly traded company badmouthing another publicly traded company with an awfully complex web of mutual investments is a very bad idea in general. It’s not that easy.
- cratermoon 2y agoMicrosoft tried to push back on vendors wanting kernel access in 2006 <https://arstechnica.com/information-technology/2006/10/7998/ https://arstechnica.com/information-technology/2006/10/7998/> Microsoft has (somewhat correctly IMNSHO) pointed at the EU agreement that forced them to open the kernel up to third parties as being a factor in the CrowdStrike catastrophe. <https://www.theregister.com/2024/07/22/windows_crowdstrike_kernel_eu/ https://www.theregister.com/2024/07/22/windows_crowdstrike_k...>
- oneeyedpigeon 2y agoFrom the latter: > However, nothing in that undertaking would have prevented Microsoft from creating an out-of-kernel API for it and other security vendors to use. Instead, CrowdStrike and its ilk run at a low enough level in the kernel to maximize visibility for anti-malware purposes. The flip side is this can cause mayhem should something go wrong. > The Register asked Microsoft if the position reported by the Wall Street Journal was still the IT titan's stance on why a CrowdStrike update for Windows could cause the chaos it did. Redmond has yet to respond.
- naasking 2y agoPeople wouldn't need CS if Windows was better designed to begin with...
- rty32 2y agoCare to elaborate? How would a better designed Windows eliminate the business & compliance need for installing software like CS? And why hasn't that already happened? I would think Microsoft and CS' customers have an incentive to not have such third party software on their system if possible.
- naasking 2y agoNo commodity operating systems are designed for least privilege. It hasn't happened because the market incentives for existing, widely deployed software are too great. Just patch and release is cheaper than starting from an actually secure foundation.
- thebytefairy 2y agoIt's a little ironic they are taking the high ground on safe rollout practices when they had an Azure/365 outage caused by a bad config at the same time as the CS incident. Though to be fair, it only affected US central.
- mrguyorama 2y ago>Especially in teams needed to respond to customers about "Your windows OS is broken, our hospital patients are suffering...". The reality is that this experience has been built into working for Microsoft since at least the 90s. The entire point of the compatibility effort they put into Windows 95 and later came from people blaming a Windows upgrade for breaking software that barely worked in the first place. Windows Vista was known for lots of BSODs, but at least 60% of them were entirely due to nVidia GPU drivers crashing.