11 ms·
MS Azure down: An emerging issue is being investigated
- deathanatos 6y agoEdit: it seems like it might be over. So, the outage appears to be that DNS for `azure.com` and maybe also `windows.net` (blob storage for us, but I'm not sure) is not resolving. So, the OP's link here is broken. Tweets indicate that it might be intermittently resolving. https://twitter.com/AzureSupport/status/1377737333307437059 https://twitter.com/AzureSupport/status/1377737333307437059 > Warning sign We are aware of an issue affecting the Azure Portal and Azure services, please visit our alternate Status Page here https://status2.azure.com https://status2.azure.com for more information and updates. Which is the link above, and is also down for me & many others. Edit: seems hit or miss. A coworker got a successful resolution of status2.azure.com of 104.84.77.137 , so manually, I can get there now. They're directing customers to an all green status page, except for the "An emerging issue is being investigate." bit at the top… (I know of at least two services that are not happy…) It had to have been less than a month ago that AAD caused a cross-service global outage. Now it's DNS. It's always DNS.
- bartread 6y agoIt was less than a month ago and, with us starting our migration to Azure which is due to complete by mid-June, I'm getting decidedly jumpy about the amount of downtime. Our current provider, with whom we have a number of dedicated servers, might be piss poor in some ways (takes ages to get changes made, lack of pricing transparency, kind of expensive for what they are), but I don't remember the last time they had an outage. There's definitely been one in the last year, but two in the space of a month is ridiculous. Is this normal for Azure? Is AWS any better?
- cmpb 6y agoWe’ve been with Azure for two years. This scale of an issue is definitely abnormal, but other minor outages (typically geo-specific) are more frequent, which is why we do have some things using their “paired regions” geo-redundancy mechanisms.
- abarringer 6y agoWhat I truly don't get is everyone compares piss poor on-prem servers with premium cloud offerings. Sure, you can screw up on prem. But if you are only marginally competent spinning up services/servers/clusters takes but an instant, it's dirt cheap, and vastly more reliable. Our only significant outages in the past 10 years have been our services dependent on big clouds. Including the Pre-Thanksgiving AWS one last year. AWS outage history here https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Services#Amazon_Web_Services_outages https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Service...
- paulryanrogers 6y ago> Sure, you can screw up on prem. But if you are only marginally competent spinning up services/servers/clusters takes but an instant, it's dirt cheap, and vastly more reliable. Do you mean on premise you can spin up virtual servers and virtual clusters instantly? Because I imagine you'd have to order, physically rack, and setup the bare metal if one doesn't have the capacity already.
- iJohnDoe 6y agoExactly. Sure, my cousin Earl can spin up a container and some VMs. Will they be secure? Probably not. Will they be backed up and offsite failover? Probably not. Will they get ransomware at some point? Probably. There is no need to make blanket statements about on-prem vs. cloud. You have to weigh the pros and cons as any other business decision. It’s about controlling you’re own destiny with either decision.
- paulryanrogers 6y agoNot trying to make blanket statements. I'm actually a fan of on prem when load and reliability needs fit.
- petre 6y agoYou can use Proxmox. We have an on prem cluster. It works reliably as long as your hardware works. It does LXC, VMs, migrations to another node, snasphots, ZFS. We hand an outage when a router died. Surprisingly a bare metal FreeBSD server directly connected to the public network was still up when the VMs were unacessible.
- nonbirithm 6y agoThis is not even the first time that Azure has had single-point-of-failure DNS issues. https://news.ycombinator.com/item?id=19812919 https://news.ycombinator.com/item?id=19812919 https://news.ycombinator.com/item?id=12505478 https://news.ycombinator.com/item?id=12505478
- king_magic 6y agobing.com, status.azure.com, status2.azure.com - all down. Can't sign into portal.azure.com, can't hit Azure File Shares, etc. The last outage a few days ago was enough for my company to up and move most of our stuff to AWS. This new outage is enough for us to fully migrate away from Azure. What a cluster.
- snoshy 6y agoCortana's revenge for Microsoft shutting her down.
- barkingcat 6y agoCortana's Revenge would be a great name for a System Shock style thriller.
- eurasiantiger 6y agoSiri Skynet Cortana sounds like a female italian mob lawyer from an anime.
- mnordhoff 6y agoInstead of going from relying on a single provider to relying on a single provider, you could use both AWS and Azure.
- reasonabl_human 6y agoThis is the right answer from a proper dev ops and opsec POV
- Elgolfin 6y agoThe solution may have a serious cost depending on your architecture. You will have to make sure it is worth the investment. Replicating and keeping a whole infrastructure stand-by is not an easy job for an one-hour outage. Sometimes, it is not the right solution either. It really depends on the business you are in.
- certifiedloud 6y agoLooks like L3, Azure and Google are all being affected by a DNS DDoS attack.
- geitir 6y agohow can you tell?
- feyman_r 6y agoAWS: https://downdetector.com/status/aws-amazon-web-services/ https://downdetector.com/status/aws-amazon-web-services/ Cloudflare: https://downdetector.com/status/cloudflare/ https://downdetector.com/status/cloudflare/
- feyman_r 6y agoAnd Google too: https://downdetector.com/status/google/ https://downdetector.com/status/google/
- ceejayoz 6y agoDownDetector is worthless. There’s no reason to put any stock in it. Look at the comments on any entry and it’s clear people do stuff like report “outages” for Google because a random website won’t work in Chrome.
- lima 6y agoThe data is still useful in aggregate.
- ceejayoz 6y agoMarginally, if anything. Non-technical users misattribute and misunderstand causes of outages. There’s a comment on the AWS page complaining that iTunes gift cards are being slow to arrive. The Google page has people who think the comments are the place to talk to Google support. A major outage will make itself known in far clearer ways.
- e0m 6y agoFrom status2.azure.com: ---------------------- InformationAzure DNS - Investigating We are currently investigating reports of an issue affecting Azure DNS. More information will be provided as it is known. This message was last updated at 21:59 UTC on 01 April 2021 ----------------- WarningDNS issues - Investigating Engineering is investigating an issue with DNS that is impacting several downstream Azure services. This message was last updated at 22:07 UTC on 01 April 2021
- sufyanadam 6y agoTeams is down too
- ceejayoz 6y agoI can’t play Flight Simulator, even, as apparently setting up offline mode requires being online.
- basilgohar 6y agoThis demonstrates one of my least favorite dark patterns of recent software.
- filomeno 6y agoIt doesn't really make a difference, does it?
- yuliyp 6y agoSeems like it's coming back up? Sites that were giving DNS errors are now resolving for me.
- mnehring 6y agoYup, it does seem to be coming back alive. A 3rd party API that I use is in an Azure data center. My customers were reporting outages, but I just got a text from a customer that things are working in real life. So, coming back up!
- dustymcp 6y agoJust me or azure been really unstable lately atleast here in europe?
- bicx 6y agoIt's not just you, and not just in Europe
- semicolon_storm 6y agoMajor worldwide outages for AAD, CosmosDB, and now DNS in just the past few months so it’s definitely not just you. Maybe Microsoft should invest less in their shiny new AI/ML platforms and more into stability of their core services.
- beebob 6y agoI'd like to second this. From where I'm sitting it feels like Microsoft is prioritizing marketing more than engineering. It seems like most larger customers that use Azure do so because management got shiny presentations from Microsoft and now it's their "strategical partner". A lot of overselling with huge discounts gets them their in. Already seen this at multiple companies. Azure is a nice platform for your Windows administrators to shift some load to the cloud. But to build large applications on? Edit: And I kinda feel bad for saying this, since I assume that there are indeed pretty competent engineers working on Azure. But somewhere something isn't right.
- deleted 6y ago[deleted]
- ridethebike 6y agoMove to cloud they said. It will be more reliable they said. But seriously, as inflexible, painful and tedious to support our on-prem infra last time we had critical outage was ~3 years ago, for like an hour. We've almost finished migration to azure. I hope this outage and AD outage earlier this month are outliers.
- 6y ago
- e0m 6y agoWe offer an API behind Azure DNS. As a hacky, but functional, workaround for our customers they can manually add our server's IP address to their /etc/hosts file. It'd be important for them to revert that once DNS returns; however as a quick fix this might help some people.
- Nelkins 6y agoI don't think I get this April Fool's joke.
- rvz 6y agoI thought this was another April Fools joke from Microsoft. Then I looked at the status page and here we are. Then GitHub Actions and some services stopped working and went on holiday today: https://news.ycombinator.com/item?id=26666782 https://news.ycombinator.com/item?id=26666782
- jftuga 6y agoAll Microsoft properties are being featured on https://downdetector.com/ https://downdetector.com/ today!
- SketchySeaBeast 6y agoNothing better than starting a long weekend with a client calling to say their site is down and you can't even access Azure portal to see what's going on.
- NicoJuicy 6y agoEverything seems to be running again.
- Scene_Cast2 6y agoI found out about this because I almost lost a password with bitwarden just now. Their "add new password" prompt is failing silently.
- qbasic_forever 6y agoYikes! That's not a good failure mode at all.
- Jouvence 6y agoThe same thing happens (or at least, did happen last year) if you don't have the correct app permissions set when using Bitwarden on a mobile device.
- smitop 6y agoLooks like this took down the national emergency alert system in Canada. I'm registered as an alert LMD (last mile distributor), and Pelmorex (corporation running the system) just emailed me to say "Please note that currently there is an unexpected significant outage on Microsoft Azure that is affecting the availability of the NAADS system and other clients globally. The NAAD System feeds are currently not accessible. We are following up on this and we will update you as soon as the issue is resolved.". I sure hope there aren't any emergencies in Canada until this is resolved...
- PretzelPirate 6y agoWhy would Canada deploy this to a single provider? A national alert system should have a better DR plan than that.
- gnabgib 6y agoBig cloud systems, the ones we all regularly use, get much closer scrutiny than government systems normally (because we all know when Azure, GCP, AWS, Linode, DO, CloudFlare or OVH have a major outage, misconfiguration, or A/C fire). As a "single" provider I'm sure Microsoft is much better at this than the (likely shoestring budget) government group of admins was before they moved. There's more burst capacity (if needed) without paying for redundant equipment most of the year, and ultimately it's probably at a lower cost. Meanwhile Azure is the only cloud with two Canada regions (Canada "Central", Canada East)[0], AWS has one (Canada "Central")[1], GCP has one (Montreal same city as AWS)[2].. there's really only one player if you want to use managed cloud services. [0]: https://azure.microsoft.com/en-us/global-infrastructure/geographies/ https://azure.microsoft.com/en-us/global-infrastructure/geog... [1]: https://aws.amazon.com/about-aws/global-infrastructure/regions_az/ https://aws.amazon.com/about-aws/global-infrastructure/regio... [2]: https://cloud.google.com/about/locations/ https://cloud.google.com/about/locations/
- jkaplowitz 6y agoI think the point was that a national emergency system should redundantly use two cloud providers, not that they made the wrong choice of single cloud provider.
- Narkov 6y agoIt's always DNS!
- aksss 6y agoDid Hololens go down? :^)
- michaelmcmillan 6y agoIronically Azure tweeted a guide on how to protect your DNS against unwanted changes only 3 hours before everything blew up: https://twitter.com/AzureSupport/status/1377697274378260489 https://twitter.com/AzureSupport/status/1377697274378260489 You can't make this stuff up.
- emodendroket 6y agoIf they would have published it a bit earlier this whole problem could have been avoided.
- abledon 6y agoIMO thats a home-run april fools joke
- orf 6y ago> Microsoft rerouted traffic to our resilient DNS capabilities and are seeing improvement in service availability At least use well formed English when posting your update...
- jasonvorhe 6y agoYou're not on call very often, are you?
- sokoloff 6y agoI suspect that the update started as “We” (which was grammatically correct) and got hastily changed along the way to Microsoft.
- orf 6y agoWould I forget English if I was? To be clear: I’m not ridiculing anyone in particular (native or non-native). I’m pointing out that a multi-billion dollar company experiencing a major outage could probably proof read the statements they put out about that incident before publishing. Else why not just go “if ur dns is bad its probs us soz, were on it”
- deleted 6y ago[deleted]
- nonfamous 6y agoIt may be British English, where companies are collective nouns and pluralised.
- wjd2030 6y agoso. fucking. tired. of. this.
- cecida 6y agoThis is starting to look serious. Is there a new zero-day DNS vulnerability?
- tyingq 6y agoPurely guessing, but the Ubiquiti breach got some press coverage lately. Buying that list of ids and passwords would let you launch a pretty good DNS attack.
- lmeyerov 6y agoCurious: For low traffic systems (b2b, not consumer), how do folks do resiliency here, esp in a simple operable manner? ex: hidden primary in cloudflare, with some sort of automated secondaries in az + aws (and how does replication get automated?)
- jbirer 6y agoI'm curious now, what benefits do Azure users see over AWS or any other provider?
- joenathanone 6y agoIntegration with Microsofts Stack, 365 offers lots of services, vertical integration can bring lots of benefits.
- atlgator 6y agoI really wish AWS would come out with an Azure App Service competitor. And no, it's not the same as a container.
- TedShiller 6y agoMicrosoft. Is anyone surprised?
- FAKEDETECTOR 6y agoPeople are still using that? Think about it - what kind of work does involve "reliable infra" today? Instead of knowing one Linux system well and maintaining multiple servers in different data centers with different providers you now have the stupid overhead of multiple cloud infra providers with all their lockin-pitfalls and incompatible specialities. The promises of cloud have not been delivered. It is all fake.
- tkzed49 6y agoa lot of pretty big business (including the cloud providers themselves) run very successful services on cloud servers. I don't see how it's fake.