4 ms·
Microsoft has serious questions to answer after the biggest IT outage in history
- duxup 2y agoThis is a pretty empty article that just seems like spin on the current issues going on.
- johnnyo 2y agoI don’t see how this is Microsoft’s fault or issue. MS can’t prevent a software vendor from breaking the machine.
- williamstein 2y agoApple
- duxup 2y agoDoes Apple have a better method for preventing something like that? I love my Mac, but I've had crashes that I suspect were caused by an application.
- rawgabbit 2y agohttps://support.apple.com/guide/security/operating-system-integrity-sec8b776536b/1/web/1#sec41bf3cd61 https://support.apple.com/guide/security/operating-system-in...
- duxup 2y agoTY
- jnwatson 2y agoThey have sunset third party kext files. That means if the kern crashes, it is Apple's fault. I once did a little MacOS driver development and had a kext signing key. It was an unforgiving, poorly-documented environment. Good riddance.
- deleted 2y ago[deleted]
- conscion 2y agoLarge corporations buy Windows _because_ they can have this level of control over their machines. The CTOs and auditors want to be able to say they've personally secured their systems using "top of the line" security software.
- MattGaiser 2y agoThey could basically eliminate vendors and make everyone use a Microsoft tool. But I suspect many would object to that solution also.
- diego_sandoval 2y agoThe they'd get sued by the EU for bundling these tools into the OS and eliminating the market for third party vendors.
- MattGaiser 2y agoExactly. If you are going to demand a ton of different companies build different parts of a product, integration issues are unavoidable.
- drpossum 2y agoI'm going to follow the guidelines and not be snarky: Microsoft is not some weak company at the mercy of the market. They choose their vendors (they also actively throw their weight around with vendors) and MS on top of that is capable of doing anything in-house (or buying it to bring it in house). The stance of "let's not hold companies accountable for cutting corners" is one reason everything is getting worse. It's because we collectively let it get worse.
- johnnyo 2y agoI’m not following your line of reasoning, can you clarify? Is the argument that it was MS responsibility to bake something like this in at the OS level? And if they did it would be more robust? I’m not sure I agree. MS has already gotten in trouble for monopolistic practices before, so from a legal standpoint, I’m not sure that’s the best course of action.
- MattGaiser 2y agoExcept that doing those things could be viewed as monopolistic or anti-competitive behaviour. If Microsoft is responsible, then they need to also need greater control. If Microsoft isn't supposed to have that kind of power, they cannot be blamed.
- alwa 2y agoIs it possible that you’re talking about a different incident? In the incident at hand, didn’t enterprises in question choose the EDR vendor, not Microsoft? Is the implication that Microsoft should be compelled to develop its own EDR product at a level of sophistication comparable to what CrowdStrike offered, and compete with them on that basis? It feels strange to me to hold Microsoft accountable for the poor design decisions of firms who just develop third-party software on their platform.
- georgeplusplus 2y ago>>>It's because we collectively let it get worse What would you suggest we do to make sure it doesn't get worse? A large majority of the population probably has no idea the implications of this outage and what to do about it because most are tech deaf.
- jgeada 2y agoWTF? Isn't that exactly the one of the main jobs of the OS, to not crash regardless of what user-space software is doing?
- johnnyo 2y agoI think the idea is that CrowdStrike doesn’t run in user space. If an Nvidia driver had bricked the machines, would that be MS fault or Nvidia fault?
- diffeomorphism 2y agoWhy not both? I am perfectly happy to blame multiple parties, not just one.
- flohofwoe 2y agoIME a graphics driver crash recovers just fine on Windows. The screen goes black for half a second and you're back in business without losing progress.
- velcrovan 2y agoI've had NVidia drivers bluescreen Windows 10 and 11 machines within the past six months.
- zdragnar 2y agoEndpoint protection is hardly user-space software. It gets deeply privileged access to the entire system.
- fifteen1506 2y agoit was a kernel mode driver.
- theblazehen 2y agoIt wasn't user space, it installs a kernel mode driver
- 2y ago
- dzonga 2y agobecause the underlying os they provide allows kernel access. if they had windows fence of the kernel and maybe provide a security api ? then this whole thing wouldn't be an issue
- deleted 2y ago[deleted]
- traveler1 2y agoDrivers have the right to crash the system in my books - software doesn't. They need to take a stronger stance on antiviruses and kernel based software in general and push defender as the defacto antivirus for Windows.
- commandlinefan 2y agoI predict that the people who were actually responsible (the "deadline above all else" crowd) will not be the ones who are actually blamed.
- velcrovan 2y ago“The Entire Culture Around So-Called ‘Software Engineering’ And Our Collective Failure to Build Strong Legal Institutions Around That Culture has serious questions to answer after the biggest IT outage in history” there fixed it for you
- zdragnar 2y agoThis is an odd take. My electrical power goes out more often than Windows crashes.
- gjsman-1000 2y agoAt a minimum, at a minimum, we should absolutely demand that Microsoft abandon requiring Microsoft Accounts to set up Windows. They have lost all rights to do that with a straight face.
- bebop 2y agoYou can still use local users, you just have to work a little harder to do so.
- rolph 2y agofor me that translates as opt-out friction, vs it just opts right in.
- blackoil 2y agoA third party s/w crashed because users failed to validate deployments. So from MS perspective users can't be trusted and need to be further locked down.
- velcrovan 2y agoMaybe that is relevant to home users but there's no relevance to today's outage. And if you're in a business setting I'm not sure why you wouldn't be using a domain controller and/or Entra ID and Intune to manage your Windows endpoints.
- geodel 2y agoThe answer is Cloud based Windows OS which MS is working towards for many years.
- MattGaiser 2y agoI'd be curious whether people would want the cost effective fix for this, which is basically to eliminate vendors for anything important. Near complete vertical integration of security, like with Apple.
- jamescun 2y agoNot sure what questions Microsoft have to answer. A third-party vendor shipped defective software. I guess the only question they could answer is why they don't provide a framework like Apple do with Endpoint Security for third-party vendors to use.
- drpossum 2y agoYou're confusing the Crowdstrike issue with Azure being down. Microsoft is ultimately responsible for anything regarding Azure even if it was a vendor that did something wrong because they choose their vendors
- danbruc 2y agoThe article is about CrowdStrike incident and not the Azure configuration issue.
- Daviey 2y agoBecause an essential enterprise security application was /able/ to bring down an entire OS like this. The issue is that Microsoft doesn't provide an interface for an application to operate in user-space to have the functionality it requires. Linux has eBPF which can provide most of the capability that Crowdstrike needs, by using an "in-kernel verifier which performs static code analysis and rejects programs which crash, hang or otherwise interfere with the kernel negatively". If MS had this functionality, it is likely this incident would not have happened. That said, from personal experience on Linux it's been an extremely long time since a bad kernel module has rendered a system entirely FUBAR'd. (To Microsoft credit, they have begun copying the eBPF methodoloy to Windows, but it is still in it's infancy https://github.com/Microsoft/ebpf-for-windows/ https://github.com/Microsoft/ebpf-for-windows/ ).
- netdevnet 2y ago> Linux has eBPF which can provide most of the capability that Crowdstrike needs, by using an "in-kernel verifier which performs static code analysis and rejects programs which crash, hang or otherwise interfere with the kernel negatively". If MS had this functionality, it is likely this incident would not have happened. It didn't stop Linux machines from being down so it is clearly not as easy as you put it. The reality is that writing software is hard yet devs often trivialise it to their own detriment
- Lx1oG-AWb6h_ZG0 2y agoApparently Crowdstrike also brought down Linux hosts in the same way in April but it didn’t get widely reported: https://news.ycombinator.com/item?id=41005936 https://news.ycombinator.com/item?id=41005936
- lloydatkinson 2y agoWhat an infuriatingly poor article, Tom Clarke should be ashamed. > A software update from cybersecurity company CrowdStrike has now taken a large number of those machines offline. So Tom opens the article with the admission that it is CrowdStrike, not Microsoft. > Thankfully, the update that caused the Microsoft meltdown did not affect these other software families - if it had, the impacts could have been catastrophic. This is such a strawman (like the rest of the article honestly) I don't know where to begin. Inflammatory language. A fucking "meltdown"? A meltdown of Microsoft, no less? Putting aside the fact that Microsoft and Windows are not the same thing, it is again nothing "meltdown" like that Microsoft did or could do. > There are serious questions of course for CrowdStrike. As a leading provider of security software for large companies like Microsoft. Tom, was you paid by CrowdStrike or what? What do you mean "of course"? It is literally the only party that should be answering questions here. I suspect that even if Tom were to "question" Microsoft their answer about kernels, drivers, privileges, and how shipping seemingly untested code into the core of an operating system is a bad idea wouldn't even be comprehendible for him. > The situation may also lead to calls from Microsoft users about what more the company could do to ensure products made for their software aren't going to cause major outages like this one. This is getting absurd now, and I just can't give more energy to this. OK, it could now insist only memory safe languages such as Rust are allowed for drivers. Or outright permanently blacklisting drivers from certain vendors. The bitching and moaning from manufacturers would then, of course, have people like Tom writing articles like "Microsoft is making manufacturers lives harder, think of the poor IT professionals!". > Any engineer will tell you over-reliance on one system leaves you open to a "single point of failure". Critical digital infrastructure has to have redundancy - back up systems - built in to ensure it is resilient. Please Tom, tell us more on your thoughts about memory safe languages, failure recovery modes, the unikernel vs microkernel debate, and how it's just a simple matter of overnight making operating systems "not a single point of failure". This entire article is some kind of exercise in trying to get everything wrong while meeting a minimum word count, and I bet with some ChatGPT thrown in there too. I flagged this post because I think it's far below even the minimum quality level for HN. It's outright clickbait drivel.
- luma 2y agoI'm not sure what to make of this but I'm noting something odd this morning: coverage of this event out of the UK near-unanimously is laying this outage on Microsoft. BBC ran a story this morning that didn't mention Crowdstrike until the 4th paragraph, and headline after headline is repeating the message that Microsoft caused a global outage. Reporting from the US and elsewhere seems to be a bit more on point. Is it just because the Brits went to press earlier in the day before the problem was understood?
- plesner 2y agoThere seemed to me to be a clear shift in focus from CrowdStrike to Microsoft somewhere along the way, maybe a little while after George Kurz' message. I was wondering if it was either spin or the media collectively deciding that people understand what MS is better than CS.
- kkfx 2y agoWell... The only question should be architectural: - why automatic, silent upgrades - why no boot environment/generations at boot to reboot into a previous snapshot of the system (since nfts do have snapshots indeed), meaning why no integration between the storage and the system management - why massive rollout instead of partitioned testing rollout slowly propagating For the rest is a third party tool, not mandated by the vendor so... It's a user choice.
- bradford 2y agoI see discussion about who's at fault: Microsoft or Crowdstrike. But one thing I don't get about this: what was the role of the enterprise admins? Most administrators at large companies are cautious about rolling out new software versions to their employees. They (normally?) test before broad deployment. Seems like one of three things would have had to have happened for this to be missed: 1. Admins ignored testing this update prior to enterprise rollout. 2. Crowdstrike forced the update on unwilling users. 3. Crowdstrike does not provide a framework for such pre-rollout testing, and enterprises chose to use it anyway. Can anyone offer insight? [Disclosure: I'm a Microsoft employee, but not an enterprise admin]
- swozey 2y agoAs a previous IT Manager (SRE last decade), honestly, most IT Admins I know literally do nothing beyond click Auto Update checkboxes and let things churn until they break. I hate to put that career down but I worked at all levels of it starting from tech support in a call center. It's a very easy job to get either get complacent with your skill set, comfortable, and really just half ass things. I have a lot of friends that are on IT teams and most of them don't have any interest in what I do, like learning to write golang, rust, python, learn kubernetes, docker, etc.. I tell them all the time about how much money they can make if you really buckle down and learn to program or just learn a cloud and terraform. They all bitch about how much they hate their jobs because they're doing basically line tech support but are fine where they're at. I had horrible IT jobs so I'm super sympathetic to it and always try to hire them when I can. I hired one last year, not a nepotism hire I didn't know him, but he was an IT guy wanting to move into SRE. They just use Windows and let it do its thing. Its their day job and they don't work on improving so their skillset is super subpar. To them it's all they need to do their job, I don't have that personality, I'm obsessively min-maxing things. I even worked at MSPs (Managed Service Providers) so I did IT/Network admin work for tons of companies around different cities and every single MSP just either puts everything on auto update or has a strict rule of NOTHING on auto update which just means nothing ever gets updated until a customer calls in for $$ hourly support. You throw the updating in because you get more hours. Or you have a scheduled update ticket every month/etc. I also saw a comment or a meme about Crowdstrike being able to update whenever wherever, no idea if that's true or not. I wrote this post from my point of view which is that of being a Windows SysEng at 4-5 big webhosts over a decade. I had to write a TON of my own security and backup and whatever services because Windows was so barebones and at the fleet (tens of thousands) server level I was managing, with the revenue webhosting makes, we definitely couldn't buy expensive software. Most of my customers paid $10/mo and we were cramming thousands of them on one server the licensing was a huge pain for any software. https://news.ycombinator.com/item?id=41007824 https://news.ycombinator.com/item?id=41007824
- multimoon 2y agoI can’t explain enough how much I dislike Microsoft the corporation, but this wasn’t their fault - a 3rd party kernel driver crashed the system.
- 1vuio0pswjnm7 2y agoI have been 100% Microsoft-free in all computers and networks I control for decades. I know I should miss Excel, etc. but strangely I feel like I have sacrificed nothing. There is more I can do without Microsoft than with it.
- arshiiita 2y agoThe problematic driver was dowloaded from Microsoft managed infrastructure, even though it was a third-party module. Microsoft needs to do a better job at running integration tests between windows kernel and driver updates for sure. They can’t publish security updates without running integration tests, this is basics of software engineering. Windows is their product not Crowdstrike. 100% Microsoft failure here.