5 ms·
Azure DevOps is down globally
https://status.dev.azure.com/
US Central Azure cloud is down also, must be related: https://azure.status.microsoft/en-us/status
- MrLeap 2y agoBeen down hours. Those poor 9's
- wsdookadr 2y ago99.999999999999999999999999% high availability fault tolerant and resilient infrastructure
- gmuslera 2y agoThose 9's are still there. Just that the decimal places were misinterpreted,
- MrLeap 2y agoThe decimal is going door to door, menacing the 9's.
- marginalia_nu 2y ago89.99999999999 also has a lot of 9s.
- akokanka 2y agoEntire Azure been offloaded to WITCH beware.
- WantonQuantum 2y ago[flagged]
- imadethis 2y agoPlease don't post ChatGPT spam here. If you're curious, the meaning of WITCH is Wipro Infosys TCS Cognizant HCL Commonly included is Accenture (India)
- dheerkt 2y agoWITCH is the acronym for Indian tech consulting companies for WiPro, InfoSys, Tata Consultancy Services, C something, H something. They're stereotyped as being cheap and low-quality. Not agreeing/disagreeing here, just stating author's intent.
- Rapzid 2y agoI have noticed something along these lines in the GitHub repos. Tons of bugs piling up, issues closed out as "by design" where clearly is was not. Etc. I honestly get the impression Azure is in decline, particularly around the .Net integrations which is their flagship ecosystem..
- TeaBrain 2y agoI don't necessarily share this impression, but the teams that simply close items after being unable to sufficiently develop a solution, leading to the original issue raiser giving up after realizing that those who are supporting the issues are useless, are not helping with the public's impression of azure.
- Rapzid 2y agoIt's a culture issue. The culture of the teams they are.. Offshoring ownership of these projects to. A lot goes into company and team culture, but I'm sure a lot here are familiar with the dynamics of outsourced development; offshore or not. Blame seeking, blame shifting, CYA, etc.. "by design".. It's all on full display in Microsofts Azure GitHub repos.
- xtracto 2y agoIt's a matter of culture in outsourcing. I'm a CTO in Mexico and outsource to companies in Mexico (insource modality) and it is evident outsourcing people don't give a danm about the product . Why should they? They get paid by the hour, and if the client goes bust after they are paid, another will bite.
- skipjackau 2y ago[flagged]
- leosanchez 2y agoDo you have a source for that? I couldn't find anything.
- sebazzz 2y agoWhat is WITCH?
- pbsladek 2y agoOoooof.
- planewave 2y agoMy web interface and pipelines appear to be down, but it seems like git operations are still working.
- cgijoe 2y agoGuessing this is related: Minecraft is down, and possibly other games that use XBox Live Login: https://support.xbox.com/en-US/xbox-live-status https://support.xbox.com/en-US/xbox-live-status "You may not be able to sign-in to your Xbox profile, may be disconnected while signed in, or have other related problems. Features that require sign-in like most games, apps and social activity won't be available."
- deleted 2y ago[deleted]
- candiddevmike 2y agoGitHub Actions is also degraded, coincidence? https://www.githubstatus.com/ https://www.githubstatus.com/ For me, pipelines are queuing and not even scheduling jobs on self hosted runners.
- planewave 2y agoNo, from the documentation it seems they share a lot of infrastructure between GitHub Actions and Azure DevOps pipelines.
- jacobwg 2y agoNot a coincidence, no. From the "self-hosted" perspectively, interestingly what we're seeing at Depot (https://status.depot.dev/clyrvud6i57402igofm6jtb7id https://status.depot.dev/clyrvud6i57402igofm6jtb7id) is API requests to provision new runners or receive runner jobs are receiving rate limit errors, but without the regular rate limit status HTTP headers[1]. I imagine the GitHub API / Actions control plane is rather overloaded at the moment with the outage. [1] https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api#checking-the-status-of-your-rate-limit https://docs.github.com/en/rest/using-the-rest-api/rate-limi...
- blondin 2y agovery surprised by all the infrastructure hosted in us central. the page claims codespaces is functional but it clearly is not. it fails at loading vscode after bootstrapping the environment.
- MrLeap 2y agoIs it surprising? Dallas, STL, KC, Chicago all count as central. More or less equidistant for east/west coast markets. Seems like natural edge node coordinates along the scale ladder.
- RajT88 2y ago"Central US" in this case, means the Des Moines, IA set of data centers. https://datacenters.microsoft.com/globe/explore/ https://datacenters.microsoft.com/globe/explore/
- hypeatei 2y agoThey're claiming only Central US is having issues on the cloud side. But, the portal itself is also having trouble loading pages and "global" services like identity management.
- lukeschlather 2y agoIt seems like Azure DevOps migrated to some new login mechanism recently which is visibly slower than the old one, and I was seeing random failures prior to today.
- vimwizard 2y agoI have also noticed strange behavior that was not present before this perceived update, e.g. basic auth challenges and pop-up windows if I'm waking up my computer and still have azdo pages open.
- raxxorraxor 2y agoI believe that was a change to auth from a few weeks ago and is affecting every MS service. Not sure though and it does work, but it doesn't work gracefully.
- eBombzor 2y agoADO vs Teams, who takes the crown for worst software of all time?
- leosanchez 2y agoTeams easily, IMO
- barryrandall 2y agoTeams, and it's not even close. Azure DevOps usually fails in understandable ways. Teams usually fails in ways that make me doubt my understanding of technology, society, and the nature of reality.
- red-iron-pine 2y agoteams, for sure
- hsfzxjy 2y agoVSCode extension marketplace is also unaccessible
- dkarras 2y agospent a lot of time trying to troubleshoot why vscode can't access extension marketplace as I just installed a new computer and am trying to bring in my profile but getting XHR errors, and devtools showing CORS errors... after an hour+ of trying to get it to connect, finally figured the services are down.
- tzs 2y agoI suppose this explains why OneDrive is down. Attempting to save to my OneDrive folder on my Mac gets an error in Excel, and suggests that I save a copy to somewhere else. Attempting to view my OneDrive files via the web interface shows no files and then eventually says something went wrong. Retrying cycles among different errors includes GeneralError, ServerTooBusy, and telling me to make sure I don't have any firewall or plugins that could be blocking access to api.onedrive.com.
- Rapzid 2y agoWider than DevOps. My entire company has been dead in the water for over 2 hours.
- MuffinFlavored 2y agoWhat could possibly cause this kind of outage? Loss of a datacenter? Physical problem? How can a "DNS" problem (or something similar) be this widescale?
- BandButcher 2y agoI remember something with facebook? where their internal DNS was misconfigured. edit: "This change caused a complete disconnection of our server connections between our data centers and the internet. And that total loss of connection caused a second issue that made things worse. " https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/ https://engineering.fb.com/2021/10/05/networking-traffic/out... https://blog.cloudflare.com/october-2021-facebook-outage/ https://blog.cloudflare.com/october-2021-facebook-outage/ https://www.thousandeyes.com/blog/facebook-outage-analysis https://www.thousandeyes.com/blog/facebook-outage-analysis Not sure if MS is going through the same issue but DNS could be a reason why
- red-iron-pine 2y agoIt's not DNS There's no way it's DNS It was DNS
- barkingcat 2y agoDNS is definitely capable of widescale disruption. A dns error can take down the entire internet.
- deleted 2y ago[deleted]
- ctyc 2y agoAirports are also at a standstill. Handwriting tickets in KC at the moment and “awaiting the manifest before we can board” - where do you think they store that manifest, a drawer in Mr Airline’s basement?? Uhhh same place as their entire system, which is DOWN
- BandButcher 2y ago"Big Paper and Pencil" can thank the recent MS and CDK Global hacks for making writing fashionable again
- deleted 2y ago[deleted]
- xtracto 2y agoIt's funny how this story has not that much traction because it is Azure ... if it was AWS, half of the internet would be in flames.
- mike503 2y agoXbox Live was totally messed up earlier, I bet it's related.
- hnburnsy 2y agoFrontier Airlines is dead in the water... >Our systems are currently impacted by a Microsoft outage, which is also affecting other companies. During this time booking, check-in, access to your boarding pass, and some flights may be impacted. We appreciate your patience.
- shscs911 2y agoThis seems to be affecting Windows VMs hosted in AWS, as well. I'm seeing status check failure for Windows VMs across multiple AWS accounts in our Org.
- whoknowsidont 2y agoWe definitely won't be getting a postmortem.
- red-iron-pine 2y agonever do. i can recall just before COVID hit seeing Azure AD go down -- broke all of our shit. but you look at the dashboard and it's 100% green. "it's fine" they said, and fought us for months.