9 ms·
Tell HN: Azure is having a major outage
One hour into it, no status page update. Looks like Active Directory is down.
I guess they felt left out with all the AWS issues lately?
- LinuxBender 5y agoI think you are right. Quite a few Microsoft services on DownDetector spiking [1] [1] - https://downdetector.com/ https://downdetector.com/
- KiranRao0 5y agoOne thing I've wondered is how much of Amazon/Google/Microsoft's services are running on the public versions of AWS/GCP/Azure?
- dontreact 5y agoVery little at google runs on cloud. It runs on borg. Cloud runs on borg.
- robertlagrant 5y agoThis is interesting. I've not read about this before. Any blog posts/articles you'd recommend?
- dontreact 5y agoI know this from working there, but it’s kind of a shameful “secret” as far as I can tell?
- RKearney 5y agoConsidering almost everything public facing at Google supports IPv6 and almost no GCP services support it outside of a few regions (1 in the US), I’d guess none of it.
- abrookewood 5y agoAWS outage and the world is in a panic. Azure outage and ... crickets?
- echelon 5y agoA few interpretations: - Folks on Azure are active-active in multiple availability zones - Most of those using Azure are not impacted by this particular outage - Azure isn't widely used ...
- jhugo 5y ago> Folks on Azure are active-active in multiple availability zones Why would people on Azure be more likely than people on AWS to have redundancy? Anyway, in the recent AWS outages, multiple AZs didn't help much. Needed to be in multiple regions.
- macksd 5y agoIn my experience, a higher portion of Azure customers are larger enterprise customers that engage professional services from Microsoft to architect things, etc. AWS is far more popular in general, but especially with smaller teams that are more self-directed, start with prototypes and iterate from there. It wouldn't surprise my if the latter often only built in redundancy after learning lessons the hard way, even though AWS provides the ability to do so from the get go. Purely conjecture, but that skew in the customer type is definitely there.
- bfung 5y agoBefore “the cloud”, it was already common to run Active Directory installs as active-active in on- prem scenarios, since those installs would’ve hit outages pretty early after installing AD. Maybe not across data center regions, but AD admins learn real quick to have redundancy and backups. Porting over to Azure AD probably wasn’t such an issue then, as most firms would lift&shift their on-prem to the cloud, keeping the same configurations.
- dylan604 5y ago
- _nickwhite 5y agoAt least they acknowledge it: https://portal.office.com/servicestatus https://portal.office.com/servicestatus
- Godel_unicode 5y agoOffice largely runs on top of, but is distinct from, Azure. Edit: looks like it's AAD that's having a bad day. https://status.azure.com/en-us/status https://status.azure.com/en-us/status
- KarlKemp 5y agoThey kinda need to. It'd be too embarrassing to have an outage that nobody notices.
- aoetalks 5y agoIt looks like Azure Active Directory, so O365 would be hit harder than Azure I imagine because most services in Azure are still using storage access keys and whatnot given that AAD auth for data plane is fairly recent in most offerings. I.e my VM running and having network connectivity doesn’t necessarily depend on AAD.
- whalesalad 5y agoIt’s the time of year where everyone is about to go home for the holidays and doesn’t give a shit anymore. Fuck it, ship it!
- geoffbp 5y agoHaha so true
- xwdv 5y agoNot a good look when your cloud services go down and the world isn’t brought to its knees.
- pookeh 5y agoDo these outages have anything to do with the log4js vulnerability?
- jhugo 5y agoSeems like a stretch.
- cmckn 5y agoWhile MS has Java in more than a few places, it’s a microscopic share of the codebase. Very unlikely those vulns could take an Azure service out.
- aiddun 5y agoYeah. I don't know the specific percentage, but I'd guess the vast vast majority of Azure code is C#. Source: I'm a former Azure intern
- ashtonkem 5y agoIf I had to guess on something external to MS, I'd blame the same storm system that caused a us-west-2 outage today.
- leros 5y agoAha! I knew Azure was built on top of AWS.
- rossdavidh 5y agoUnfortunately, in an effort to cut costs, Amazon will outsource the actual hosting of AWS to Google Cloud, who will outsource it to Microsoft...
- charcircuit 5y agoIt would be cheaper for Amazaon to just build their own datacenters since they are so big.
- jmartens 5y agoTook about 80 minutes for them to update their status page.
- jrs235 5y agoHow soon until "the cloud" and centralized services "vaporize" and on-premise becomes all the rage? The cycle will continue. What was old will become new again.
- harpratap 5y agoThe main idea with Google creating K8s was to commoditize it's complement. Now they are pushing for Anthos with is basically GCP with BYOD on-prem (or other vendor) hardware. So what you will see is people using Hetzner like services with cloud software provided by these companies. It is kind of already happening with Teleco world. Companies like Dish and Vodafone partner with AWS & GCP. These telecos buy their own OpenRAN hardware from Nokia or Ericsson and manage it using AWS, GCP, Azure software.
- shubb 5y agoAzure has a lot of features around this model.For instance, you can put a kind of azure agent on your on prem servers and then provision azure virtual machines and managed databases onto them, or cloud active directory for on-prem desktops. I hadn't thought of them in the context of "hire cheaper non-cloud servers for your baseline load and storage".
- barbazoo 5y agoToo many organizations I can think of that wouldn't exist without cloud services so I don't think what you said will happen.
- stevejb 5y agoSame as when horse and buggies will come back in response to a disruption of the gasoline supply chain.
- paradite 5y agoOne of my favourite quotes is from "Romance of the Three Kingdoms". It notes an observation that I find generally applicable to many things in life: "Those long divided shall be united; those long united shall be divided: such is the way of the universe." Original quote in Chinese: ＂话说天下大势，分久必合，合久必分。＂
- bandyaboot 5y agoTurns out that clouds are chaotic and difficult to control.
- twodave 5y agoOur production infrastructure is on Azure and we're seeing basically no impact from this event. I'm not sure what constitutes a major outage but we've had a few others this year that were much worse than this in terms of business impact.
- jmartens 5y agoFair point. Lots of stuff kept working, just the things that required auth were failing. So broad, but not deep.
- topspin 5y agoI spent two hours attempting to detect any impact to my services from the AWS US-West-2 outage that happened earlier today. I found nothing. Strange.
- callmeal 5y agoThe us-west outage did not affect connectivity to the west coast. Our west coast customers had no problems with that connectivity, but our east coast and overseas customers had problems for a couple of hours.
- 0xFFFE 5y agoImagine if these incidents happened in an on-prem infra with such regularity. Heads would have rolled.
- Godel_unicode 5y agoWasn't the last meaningful AAD outage in February sometime? This was an outage that lasted on the order of an hour, outside of normal US business hours. I think the crazy uptime we normally get from the cloud has seriously skewed people's perceptions of how things used to be on-prem. I personally remember when my Fortune 50 former employer had multiple multi-day email outages in one year.
- kemals 5y agoThe outage is clearly visible on the Internet Outages Map provided by ThousandEyes: https://www.thousandeyes.com/outages/ https://www.thousandeyes.com/outages/
- a-dub 5y agofor some reason, the idea of intentional outages as a form of pr for growth hacking is kinda cool to me. i know it's extremely unlikely, but that kind of cleverness would be pretty cool. "our new sexy feature x went down for an hour today." "wait, what? they offer x as a service now?"
- tmaly 5y agoSo what is the solution to all of these cloud outages?
- johndevor 5y agoDecentralization of some sort?
- AstroDogCatcher 5y agoPerhaps we should put computers inside each businesses location, to minimize the blast radius of any specific outage and make people take responsibility for their own stack.
- belfalas 5y agoIs there a status page specifically for instances of MongoDB hosted on Azure?
- jd_mongodb 5y agoIf MongoDB Atlas cloud services are affected it will be reported here: https://status.cloud.mongodb.com/ https://status.cloud.mongodb.com/
- NicoJuicy 5y agoI wouldn't call this a major outage. We had almost no issues with our applications ( it's resolved by now) and customers are in US, Canada and EU. The update that caused an issue, was at a pretty good time ( 20:42 EST ). So that probably explains it. > A shared component of the Microsoft account (MSA) and Azure AD sign-in services stopped responding when a combination of a configuration error and a routine update caused multiple redundant endpoints to become unreachable. This caused sign-in failures in Microsoft services for both Microsoft personal accounts and Azure AD accounts. https://portal.azure.com/#blade/Microsoft_Azure_Health/AzureHealthBrowseBlade/healthHistory/navigateTo/healthHistory/selectTrackingId/S_3M-FZZ https://portal.azure.com/#blade/Microsoft_Azure_Health/Azure...
- limoce 5y agoBing is down in some regions. Is it also related to this outage?
- have_faith 5y agoAre these threads necessary? especially when they're so common now. Don't want HN to turn into is-it-down.
- aorth 5y agoHosted PostgreSQL on `postgres.database.azure.com` is down for me...