7 ms·
Major Microsoft 365 outage caused by Azure configuration change
- vb-8448 2y agoI guess we can call it a "black Thursday" for microsoft.
- bluedino 2y agoWhy is MS getting any blame for the Crowdstrike thing?
- yelnatz 2y agoThey allow a 3rd party to change Kernel stuff with an update. Apple banned this a while ago.
- tambourine_man 2y ago*banned Made it a lot harder for everyone involved, but still possible, as it’s a very useful technique. PS: Since I'm being downvoted, here is the link showing that it is still possible using Reduced Security: https://support.apple.com/en-gb/guide/mac-help/mchl768f7291/mac https://support.apple.com/en-gb/guide/mac-help/mchl768f7291/... I doubt Apple will completely disable kext in the near future. Making it hard enough to be impractical has most of the benefits already.
- kykqx 2y agoIt is supposed to be like that. Antivirus software always works as a driver in the kernel, no other way. You'll get the same in Linux, for example. In MacOS it may be slightly better (if I remember right Darwin is a micro-kernel), but in fact a broken driver still can crash the system there.
- philistine 2y ago> Antivirus software always works as a driver in the kernel, no other way. You're confidently wrong: https://developer.apple.com/support/kernel-extensions/ https://developer.apple.com/support/kernel-extensions/
- kykqx 2y agoThis page is only about using some APIs, that are now supposed to be called through wrappers. I would say it significantly limits the developers, and also may introduce additional flaws.
- philistine 2y agoYet it is how antivirus works on Mac now.
- AmVess 2y agoLetting an app take down your OS, perhaps.
- DebtDeflation 2y agoNot just take down the OS, but render the machine completely unbootable until a config file gets manually removed from the HD by starting in Safe Mode from the console.
- TillE 2y agoIt's not an "app", it's a kernel driver. A serious driver bug is catastrophic in nearly any OS.
- darby_nine 2y agoPutting aside the semantic swamp of figuring out what an app actually refers to, this seems to be exactly the problem. They should have their signing keys revoked.
- vb-8448 2y agoWell, if you look on all news sites, they put highlight that is a windows issue. I personally think that IT managers should be blamed for the disaster, but in the collective imagination this will e a microsoft/windows problem.
- mingus88 2y agoBlaming IT managers is akin to blaming a bus driver when the bridge collapses. IT managers are required to implement AV for compliance. They'd lose their jobs if they refused to do it and either failed an audit or got hit by common malware. It's ironic to see so much praise this morning for the macOS model for kernel extensions. The fact of the matter is Microsoft allows 3rd party code to do exactly what happened today, and the industry leader in endpoint security completely failed to QA their updates.
- vb-8448 2y agoIT managers allowed an external entity to do a massive update on all their devices and servers. I'm not saying to don't install AV or not update it, just first update system-test envs and then roll out patches gradually, not all together.
- mingus88 2y agoBus drivers allowed themselves to be assigned routes over bridges that were not engineered or built well enough. It's ridiculous to put the blame on IT managers for this. The fact that Microsoft Azure itself has been affected by this shows that this is not a problem that any middle manager at a small firm should have be held responsible for.
- vb-8448 2y agoAre you serious? If I push a code change in production before going through the test environment and this cause trouble, I'll be fired. We already have rules and procedures that work for "normal" software, why they cannot be applied to AV software?
- bguebert 2y agoSeems like microsoft is getting put into the headlines because the Crowdstrike versions for mac and linux aren't affected. Plus Crowdstrike has a history of pushing people onto microsoft tech support when their software causes problems.
- phatfish 2y agoMicrosoft should have a certification program for software that messes around with kernel modules that could cause a BSOD. I guess they already do this for hardware drivers. Once software is effecting a certain number of endpoints the vendor has to prove they are testing sufficiently.
- ensignavenger 2y agoI would strongly oppose such restrictions. Of course I use Linux so I can install whatever kernel.modules I want to and no one can stop me :)
- philistine 2y agoHot take, but I think you should build from source to be able to modify the kernel. Kernel extensions don't have their place in our modern world.
- whamlastxmas 2y agoHN user in other thread claimed same issue happened last week with either Mac or gnu/Linux, I don’t recall which
- ensignavenger 2y agoCrowdstrike has caused kernel panics on Linux systems in the past too. I push FOSS hard and will hold that this is an example where using FOSS for all critical business software would have saved companies, but I don't particularly blame Microsoft directly for this outage. Incompetent IT managers buying software from their golf buddies is the real heart of the issue.
- skywhopper 2y agoBecause their OS allowed a third-party tool to crash it, worldwide, with no automated way of recovery?
- deleted 2y ago[deleted]
- bongodongobob 2y ago[dead]
- jameskilton 2y agoAnd CloudStrike hitting them at the same time must have been just unbelievable.
- patmorgan23 2y agoAlso Microsoft using Crowd strike instead of their own defender for endpoint? Guess it could be part of a risk mitigation strategy to not but all their eggs in that product.
- joelvalleroy 2y agoThe Crowdstrike issue is causing the Azure issue as I understand, lots of machines using that software that all updated to the blue screen feature around 6 PM apparently.
- rcarmo 2y agoNope. Azure does not run CrowdStrike. It doesn't even run Windows as you know it.
- joelvalleroy 2y agoah you are right, I misread the Microsoft notification
- hypeatei 2y agoThe Central US outage really exposed a lot of shortcomings both at my company and at MSFT apparently. Even having regional fail over available wouldn't help because the control plane was unreliable meaning it couldn't be triggered by anyone.
- macote 2y agoThat's an interesting recommendation... > We've received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage. https://azure.status.microsoft/en-ca/status https://azure.status.microsoft/en-ca/status
- underdeserver 2y agoWhy would multiple reboots make sense? I can accept three reboots triggering some condition that tries three times before it stops trying, but fifteen?
- easton 2y agoIt sounds like a race condition; you want the CrowdStrike updater to start and pull down the fix before the affected virus definition file is loaded and kills the box. If you keep rebooting, you eventually may get those to load in the right order.
- Tade0 2y agoSounds a lot like my work laptop's antivirus software, which gives me a total of two minutes of internet from within WSL2 before blocking it entirely.
- szszrk 2y agoI'm curious if you are sure it's antivirus? WSL networking is weird and can be a pain in the ass. Especially in VPNs are used. I'm wondering if you know wsl-vpnkit?
- Tade0 2y agoNot familiar with this tool, sorry. Anyway, had that confirmed by whoever is in charge of security and whatnot, as we have an internal StackOverflow clone and someone asking the same question was pointed to a PowerPoint presentation of dos and don'ts.
- terom 2y agohttps://azure.status.microsoft/en-us/status/history/ https://azure.status.microsoft/en-us/status/history/ doesn't seem to have links to the individual incidents. Some reports claim the Azure / Microsoft 365 outages were related to crowdstrike, but this sounds like an entirely separate incident. AFAIK the broken crowdstrike channel update happened at 2024-07-19 06:05 UTC and was "fixed" (rolled back) at 06:47 UTC, but I don't have a proper source for that timeline? EDIT: https://azure.status.microsoft/en-gb/status https://azure.status.microsoft/en-gb/status claims 2024-07-18 19:00 UTC as the approximate start of impact for the crowdstrike update. It would be nice to find a proper source for the start and mitigation timelines... EDIT: reddit threads reporting symptoms start at approx 2024-07-19 05:00 UTC. That would mean the crowdstrike impact started soon after the azure recovery. --- What happened? Between 21:56 UTC on 18 July 2024 and 12:15 UTC on 19 July 2024, customers may have experienced issues with multiple Azure services in the Central US region including failures with service management operations and connectivity or availability of services. A storage incident impacted the availability of Virtual Machines which may have also restarted unexpectedly. Services with dependencies on the impacted virtual machines and storage resources would have experienced impact. What do we know so far? We determined that a backend cluster management workflow deployed a configuration change causing backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks hosted on impacted storage resources. How did we respond? 21:56 UTC on 18 July 2024 – Customer impact began 22:13 UTC on 18 July 2024 – Storage team started investigating 22:41 UTC on 18 July 2024 – Additional Teams engaged to assist investigations 23:27 UTC on 18 July 2024 – All deployments in Central US stopped 23:35 UTC on 18 July 2024 – All deployments paused for all regions 00:45 UTC on 18 July 2024 – A configuration change as the underlying cause was confirmed 01:10 UTC on 19 July 2024 – Mitigation started 01:30 UTC on 19 July 2024 – Customers started seeing signs of recovery 02:51 UTC on 19 July 2024 – 99% of all impacted compute resources recovered 03:23 UTC on 19 July 2024 – All Azure Storage clusters confirmed recovery 03:41 UTC on 19 July 2024 – Mitigation confirmed for compute resources Between 03:41 and 12:15 UTC on 19 July 2024 – Services which were impacted by this outage recovered progressively and engineers from the respective teams intervened where further manual recovery was needed. Following an extended monitoring period, we determined that impacted services had returned to their expected availability levels.
- gjsman-1000 2y agoI'm not sure if this has anything to do with it; but I got an email this morning at 2 AM saying that my Microsoft account password was changed. The email was authentic - it came from Microsoft's servers, and had no buttons for me to click. It said that the IP Address of the password change was my own ISP in my area... and that the reset came from a phone number I didn't recognize on my security info, that even now does not show on my account dashboard. I ran in a panic to reset my password... only to discover, the password was "changed" to the exact same password? And how would they even get in the account without 2FA? My "sign in history" also showed no trace of anything unusual. At this point, and reading these headlines, I feel more confident something's broke.
- isaacremuant 2y agoThis entire thing is not surprising but it's fascinating to see unfold. In many dimensions, even listening to the wild speculation or weird reporting around it.
- swayvil 2y ago[flagged]
- Jabrov 2y agoWhat's so interesting about that?
- swayvil 2y ago2 calamities so close together in space and time. It makes you wonder if there's a connection. They are both universally distracting. That's one special thing that they share. A conspiracist might suggest that our attention is being handled.
- CWuestefeld 2y agoOK, it's a pedantic nit, but doesn't anybody have editors anymore? > This massive outage started around 6:00 PM EST We're currently in daylight savings time, so the time should read "EDT".
- jftuga 2y agoFurthermore, the article states that it was "affecting customers across the Central US region". Doesn't it seem like they should have used "5:00 PM CDT" in the article?
- boringg 2y agoHonestly is anyone surprised that Microsoft is having these issues? They still run the same sloppy software as they always have.
- ilrwbwrkhv 2y agoWith some of the worst engineers. I had the misfortunate of working with a couple of Microsoft principal engineers and I still shudder at how bad they were and how many layer of abstractions they could write.
- prerok 2y agoNot at MS, but I had the misfortune of working with such "engineers" as well. I belive the term is "architecture astronauts".