7 ms·
Postmortem: Azure DevOps (VSTS) Outage of 4 Sep 2018
- byte1918 8y ago> VSTS (now called Azure DevOps) Not again.
- oceanswave 8y agoI know you just got used to and are productive with the old UX and naming... but hey now we call it something else and have a new fancy UX featuring more whitespace that you get to learn. Fire in motion!
- herbderb 8y agoWhat's the point of even mentioning that when they just use the old name for the entirety of the article anyway
- skrebbel 8y agoThe point is that the headline should've been "Azure DevOps Outage…" but they're afraid that other outlets will take that over as "Azure Outage..." and they don't want a headline like that making the rounds. So they use the old name for bad news and the new name for good news. TBH I'd do the same.
- freeone3000 8y agoBut is is an Azure outage. It took out a DC. VSTS is one of the services affected, but other services were also affected.
- a2tech 8y agoLong story short they give no indication as to why their data center cooling systems were unable to handle voltage changes caused by the storm and their systems are not designed for speedy restoration into another region.
- chrisbolt 8y agoSeems like there's more information in the preliminary RCA on https://azure.microsoft.com/en-us/status/history/ https://azure.microsoft.com/en-us/status/history/
- bpicolo 8y ago> Initially, the datacenter was able to maintain its operational temperatures through a load dependent thermal buffer that was designed within the cooling system. However, once this thermal buffer was depleted the datacenter temperature exceeded safe operational thresholds, and an automated shutdown of devices was initiated
- souterrain 8y agoTotal failure of a data center's cooling apparatus seems to be a very rare occurrence to me, perhaps limited to simultaneous failure of utility and genset power (example: electrical switchgear and fuel pumps underwater due to flooding). Anyone have any data around how frequently such a failure occurs?
- beh9540 8y agoI had the same thought. The only thing I could come up with is that it wasn't a failure of power supply, but that a surge took down enough cooling systems that they couldn't maintain temperature. A lot of DC's I've seen are N+1 with cooling (or even 2N), but they all run at the same time and are the same units. Or the control system went down, and they weren't able to get it back up and running, although I would think they would have redundancy in that case.
- outworlder 8y ago> Long story short they give no indication as to why their data center cooling systems were unable to handle voltage changes caused by the storm This shouldn't matter, only to them and whoever operates the datacenter. > and their systems are not designed for speedy restoration into another region. This is what matters. You have a cloud service that is not really designed for the cloud.
- indemnity 8y agoIs this what we have to look forward to as GitHub will be forced onto Azure?
- rhencke 8y agoDo you have a source for this?
- WorkLifeBalance 8y agoYou say that as if github never has outages when the reality is, like any large service, it has frequent service outages. Just check the github status pages, there are orange level issues almost weekly and you don't need to go back too far for full outages.
- thejosh 8y agoBecause GitHub is so stable and never goes down?
- manigandham 8y agoNo service is perfect and github has had plenty of outages. They will only become more reliable with the resources of MS/Azure at their disposal.
- tumetab1 8y agoHaving worked in big Azure customer I wouldn't say resources equals stability. The reality is more resources + quality engineering + failure testing. As this case tells, Azure, isn't spending a lot on failure testing. Also, having experience with being an big Azure customer, I can tell you that things look better than they are.
- eropple 8y agoSmall Azure customer (five figures a month), but can co-sign all of this. Azure looks shiny from the outside but we've had way, way more problems, from uptime to bad APIs to awful language SDKs to bad user interfaces to licensing hell, than I've ever had on AWS or GCP. It's so bad that I am currently weighing whether or not to advocate for a migration off of it, at nontrivial expense, because I cannot pretend to provide reliable services for our customers.
- em0ney 8y agoReally not the worst post mortem I've ever seen
- romaniv 8y agoAt the end of the day, Azure and AWS are monocultures with considerable amount of centralization and interdependency within their services. Their scale undermines the original purpose behind the Internet. It bothers me that increasing number of large companies dump their own data centers to jump into The Cloud. Thia means future outages (which will undoubtedly happen) will have wider and wider impact on end users. For example, if your email is hosted on AWS and it goes down, you loose access to your email. No big deal. However, if your email, VOIP and IM/chat go down at the same time, you may loose all ability to communicate electronically. This can be a very big deal in certain situations.
- ransom1538 8y agoMy favorite is aws slinging “make sure you have copies in other regions”. Look. Every time an entire region has an issue on aws all the regions are fucked. APIs etc just drop or hang. If a region has dropped things are really bad. So keep paying to double up your data, but it doesn’t do any good. If Virginia East has some type of nuclear attack - I have worse problems than running my crappy CRUD app.
- outworlder 8y ago> Look. Every time an entire region has an issue on aws all the regions are fucked. This is not true. I've watched some full region outages going on while our systems were chugging along just fine. I have also seen temporary network issues in one region(that caused our monitoring to go haywire) where others were completely unaffected. What tends to happen is that AWS locates a bunch of services in us-east-1 (Route53 being one example). This may knock out the control plane for a bunch of services, which shouldn't be a thing.
- ransom1538 8y ago" I've watched some full region outages going on while our systems were chugging along just fine." Yeah! It has a green dot!
- segmondy 8y ago
- sungju1203 8y agojust use AWS. simple.
- intern4tional 8y agoAWS has had these same issues when it has had region outages. Simply google AWS outage and see the results. Thankfully an entire datacenter outage is rare and Azure is taking steps to mitigate that risk.
- Bhilai 8y agoComments like this are counter productive for the discussion. Competition is always good and some of us like that AWS has competition in the form of Azure and GCP. AWS has had its own share of outages so its not perfect either.
- curiousDog 8y agoWhat will you do the day AWS decides to jack up their prices 50% and there are no competitors?
- outworlder 8y agoOk, it's understandable, freak events happen. > The primary solution we are pursuing to improve handling datacenter failures is Availability Zones, and we are exploring the feasibility of asynchronous replication. This I do not understand. I was also amazed when I saw that Azure AZs are not available on all regions. In AWS, the bare minimum is 2 AZs (except for one odd region). Same thing for Google Cloud.
- scarface74 8y agoFrom what I understand and I can't find the reference anywhere, each region has at least three availability zones. Some regions only have two user selectable AZ's. For instance, S3 promises that it is replicated between 3 AZ's in a region. That guaranteed is available in regions that only have two publicly available AZ's.
- swebs 8y agoThat's an extremely roundabout way of saying there was a lightning strike and they had inadequate surge protection.