9 ms·
Major Azure Outage
- Analemma_ 7y agoAzure SQL is totally down for us, Storage (tables/blobs/queues) is mostly down. Seems to be a DNS issue, and this wouldn’t be the first time Microsoft has been brought down by DNS.
- steveadoo 7y agoSame here. East US2, West US and West Europe are all affected for me.
- ljoshua 7y agoSame. Name resolution is failing but intermittently, changing from second to second whether it resolves or not.
- odorousrex 7y agoSame here. On a positive note, all our apps are still working - apparently their internal DNS is still working, at least in the same datacenters. I can't access anything from here though.
- steveadoo 7y agoWhen I tell my clients Azure had another outage they're going to demand we move to another cloud service. Looks like I'm in for a looong couple weeks.
- mattbillenstein 7y agoHow did you get on Azure to begin with?
- ecaron 7y agoI'm guessing he inherited it. I'm guessing most people on this thread are in that boat.
- deleted 7y ago[deleted]
- GFischer 7y agoIn my case, we actually chose it, as we´re a Microsoft shop. Hasn´t been that bad, but we did experience 2 major outages in 1 year. Not sure how many outages are there on AWS.
- tracker1 7y agoAWS has one or two a year from what I've seen... IIRC gcp has had outages too... Unless you're designing with several safeguards in place across multiple regions and cloud providers, there's no getting around it. Everyone has down time, it's just often coordinated to a lot of people when it does happen. It's still generally less than when you try to self host on a cloud provider, it's just not your mistake that did it.
- outworlder 7y ago> AWS has one or two a year from what I've seen Worldwide outage? That's really not the case. Over thousands of machines in 3 years I haven't seen a single failure that spanned more than one region. Or even a region going entirely offline. At most you'll see one service affected.
- goobynight 7y ago"an" outage is different from a global outage though. I'm not sure what the SLA is on a single region, but going down 0-2 times per year is a reasonable expectation, depending on the length of each one. If you want more, you have to have regional failover. Azure is burning error budget in every region today and you would need to failover to a different cloud provider or your own datacenter. If I'm interpreting this correctly, there was no plan you could implement solely in Azure that could have helped you today.
- deleted 7y ago[deleted]
- lghh 7y agoWhat are you going to tell them when the other cloud services have their inevitable outages?
- PlanetLotus 7y agoI'm not the parent, but I maintain services on both AWS and Azure, and in the last few years, I can definitely say the outages on Azure have been more frequent and more severe. The only AWS outages I recall are S3 and the Dyn DNS issue that brought many other providers down too.
- user5994461 7y agoThe Dyn DNS outage was easier to explain. Half of the internet is down, it's not just AWS.
- PlanetLotus 7y agoExactly. As an AWS customer, that outage didn't really bother me because it was obvious it impacted many other unrelated services too.
- cheeze 7y agoI'd tell them "other providers have inevitable outages, however..." 1. The other providers have a better track record 2. The other providers don't go down globally multiple times a year
- ecaron 7y agohttps://azure.microsoft.com/en-us/status/ https://azure.microsoft.com/en-us/status/ was just updated, showing "Network Infrastructure" is down across the board.
- ZanyProgrammer 7y agoThings have been back up for several minutes.
- zeven7 7y agoThings are not back up for me. And looking at Twitter, it looks like things are still broken for a lot of people.
- PlanetLotus 7y agoSame. West US here.
- augustl 7y agoAzure SQL is totally down for us as well. Unable to resolve the DNS for it, both from within azure (kubernetes pods) and from outside (my laptop). EDIT (23:37 Oslo/Norway): connectivity is restored for us now
- plasma 7y agoI'm seeing some recovery of services in West US
- norad73 7y agoRecovery started in Europe (AMS)
- sbr464 7y agoOn a more serious note, how would your entire network, worldwide go down? Are there really no independent zones (that are unaware of each other)? That can’t be good.
- popotamonga 7y agoPaying thousands for replications and redundancies, multizone etc and bam, all down. Nice.
- cheeze 7y agoGlobal dependencies. Something in DNS had a dependency on something central within Azure, and when that breaks, you're done for. I'm not in the cloud provider game, but it seems like it would be important to audit and ensure that there are no critical cross-region dependencies. I assume GCP and AWS do this regularly? It seems like there are some things that have to be somewhat global (IAM comes to mind), but minimization of that seems important. IMO this is the most embarrassing non-security thing that can happen to you as a cloud provider.
- cheeze 7y agoMan... Azure seems to be an order of magnitude worse than AWS and GCP when it comes to reliability. Seems like they have tons of global dependencies within their services which cause these cascading failures rather often... Seems like only a few months ago we were reading about a global outage that affected auth? Regardless: Godspeed to the engineers working to fix this.
- 6ak74rfy 7y agoPublic clouds will have outages - that's not the point. What's most concerning about this outage is that it is across all regions. That violates the fundamental assumption, of developing for the cloud, that failures in every region are independent. If regions fail independently and a failure in 1-2 regions brought down my system, that's my fault. But if region failures aren't independent and a global outage such as this is possible - well, that's pretty bad.
- cheeze 7y agoYup, fully agreed. Nobody can keep sql instances or hosts up forever. They will go down. Further, humans work on this stuff. Humans make mistakes. Bad config push, bad code push, lightning hitting a data center, human vandalism, etc. will happen. What shouldn't happen ever is that your entire cloud goes down because somebody pushed a bad config change to a service that serves literally your entire cloud. Microsoft clearly hasn't architected everything to be region-independent. There are things that will always be somewhat global, but Azure seems pretty bad at this. This isn't the first time even in the past 12 months that they've had a global outage.
- deleted 7y ago[deleted]