11 ms·
Microsoft 365 Outage
- AmateurAlert 6y agohttps://twitter.com/msft365status?lang=en https://twitter.com/msft365status?lang=en
- x87678r 6y ago> We're not observing an increase in successful connections after rolling back a recent change. We're working to evaluate additional mitigation solutions while we investigate the root cause. Oooh that's panic stations.
- annoyingnoob 6y agoCategory 503 hurricane centered over Redmond, WA
- tyingq 6y agoYep, I can imagine the feeling of doom when the rollback didn't work. Push back from the desk, head droops down, hand on forehead. Then I whisper, "Fuck".
- tyingq 6y agoWe’re rerouting traffic to alternate infrastructure to improve the user experience while we continue to investigate the issue. Imagine there's some foxhole prayers happening.
- tyingq 6y agoWe’re seeing improvement for multiple services after applying mitigation steps and we’ll continue monitoring the services to ensure full recovery. Big exhale.
- joemaller1 6y agoSeems to be Azure-related, all sorts of stuff is not accessible right now.
- mastry 6y agoYep. I can’t get into the azure portal.
- DrPhish 6y agoSpecifically Azure Active Directory We (like many other orgs, I expect) have gone all-in on AAD, so this has pretty much taken down everything company wide with the exception of the factories. Good thing the plant floor isn't tied in.
- rrmm 6y agoThis sounds like a pretty strong argument to make sure the plant is tied in too. I'd hate for everyone to get a break but them. How frequently do you think this sort of thing would have to happen before your org considered a significant change in arch?
- NovemberWhiskey 6y agoYeah; that's the real lead here. Outage on Office web stuffs is bad. Outage on Azure AD is catastrophic.
- cookingmyserver 6y agoWhich subsequently impacts Azure AD B2C. Our SSO is down on all of our sites.
- neilsimp1 6y agohttps://status.azure.com/en-us/status https://status.azure.com/en-us/status Looking at it now, Azure AD is the only service that isn't green.
- thanatos519 6y agoIs this /really/ the status page? It looks like it was pasted straight out of an email to Jason Zander from whoever is trying to fix the problem. https://status.office365.com/ https://status.office365.com/ Title: Can't access Microsoft 365 services User Impact: Users may be unable to access multiple Microsoft 365 services. More info: Any Microsoft 365 service that leverages Azure Active Directory (AAD) authentication may be impacted by this issue. Current status: We've identified and are reverting a recent change to the service which may be causing or contributing to impact. Scope of impact: Any user may experience access problems for Microsoft 365 services.
- stefan_ 6y agoThe ©2019 Microsoft Corporation is a nice touch. There isn't a date on this entire page!
- thanatos519 6y agoSeriously!!! How about the "Add this page to your favorites" link? I guess the 365 target audience still hasn't learned how to bookmark. Anyways the Azure status page is a bit more grown up: https://status.azure.com/en-us/status https://status.azure.com/en-us/status
- lucasmullens 6y agoThat button doesn't work in Chrome, which almost certainly means it doesn't even work in Microsoft's own browser (since Edge is Chromium-based now)
- tyingq 6y agoYes, that's pretty nice. A clear, concise, and accurate description of the problem, noting that the change is bring reverted.
- thanatos519 6y agoAbsolutely! It's more about the presentation. "Title:" makes it look like a grade school book report.
- Animats 6y ago"No one was ever fired for buying Microsoft" just ended.
- riffic 6y agoOh please! This sort of outage is regular for O365.
- thanatos519 6y agoMaybe they should have called it O364.
- rachelbythebay 6y agoO365 makes sense if it's down all day, since this is a leap year, after all.
- pgn674 6y agoThe exact-ish length of a year is 365.2422 days. 365 days a year is three nines of uptime (99.93369%).
- deleted 6y ago[deleted]
- fendy3002 6y agoIn 10 years it'll be Office 404 Edit: maybe not that short, in 20, 30 years maybe?
- vishaaall 6y ago39 is my guess. 365+39=404
- dylan604 6y agoThey didn't call it O365/24/7 for a reason. 1990s hosting company: Linux Server 99.99% uptime. Windows - no guarantee. 2020 - same
- imperialdrive 6y agoI'm using this to test for return of service: $loop = "loop" do {$response = $null; Start-Sleep -Seconds 5 try {$response = Invoke-WebRequest "https://login.microsoftonline.com/common/oauth2/ https://login.microsoftonline.com/common/oauth2/" -ErrorAction SilentlyContinue | select status*} catch {Write-Host "Down $(get-date)"} finally {} if ($response) {Write-Host "OK" ($response).StatusCode (Get-Date)} } while ($loop -eq "loop")
- joenathanone 6y agoDoesn't work for me, I get: At line:1 char:16 + $loop = "loop" do {$response = $null; Start-Sleep -Seconds 5 try {$re ... + ~~ Unexpected token 'do' in expression or statement. + CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException + FullyQualifiedErrorId : UnexpectedToken
- imperialdrive 6y agoSorry about that - I have no idea how to format comments. I placed a copy here: https://www.codepile.net/pile/gz77EXzd https://www.codepile.net/pile/gz77EXzd
- joenathanone 6y agoThis version works for me, thank you :-)
- douglasrank 6y agoCleaned-up version while ($true) { $response = $null; try { $response = Invoke-WebRequest "https://login.microsoftonline.com/common/oauth2/ https://login.microsoftonline.com/common/oauth2/" -ErrorAction SilentlyContinue | select status*; if ($response) { "OK $($response.StatusCode) $(Get-Date)" | Write-Host; } } catch { "Down $(Get-Date)" | Write-Host; }; Start-Sleep -Seconds 5; }
- joenathanone 6y ago
- annoyingnoob 6y agoYou could play status page roulette... https://status.azure.com/en-us/status https://status.azure.com/en-us/status https://status.office365.com https://status.office365.com https://portal.office.com/servicestatus https://portal.office.com/servicestatus
- vsareto 6y agoI like how this isn't Critical (yet)
- annoyingnoob 6y agoIts only a matter of time before there are no more 'existing sessions'. I also like how they call it 'non-regional' and not say 'global'.
- filmgirlcw 6y ago#HugOps to the SRE's rolling back changes. I can’t reliably access anything right now so I’ll just use this as an excuse to take a brief walk.
- x86_64Ubuntu 6y agoYou hope that the walk only need to be brief...
- filmgirlcw 6y agoI mean, indeed. But it’s good to get some air (wearing a mask, of course) anyway.
- nicwolff 6y ago"Current status: Rolling back the previously described change did not resolve the incident as expected. We're evaluating additional options to remediate the problem."
- OkGoDoIt 6y agoI'm assuming this is related: I can't do any development using Visual Studio 2019 at the moment because it calls home to verify the license and it is currently unable to log me in to to verify my license. So VS is basically bricked for me until this gets fixed :-( Screenshot: https://rog.gy/ss/3b14f7a2ac.png https://rog.gy/ss/3b14f7a2ac.png
- immigrantsheep 6y agoFunnily enough the community/free version works without issues. That really sucks if you're a paying customer.
- takeda 6y agonot that surprising, since it probably doesn't have to phone home.
- bleepblorp 6y agoIt does, but seemingly not as frequently. The community version seems to phone home every few months. Now, the pirated version never phones home and always works even when MS decides to steal your products back.
- orloffm 6y agoCommunity needs to re-update your license every 30 days, so it would also stop working for someone who had that period ending during the outage.
- fendy3002 6y agoWoah that must be suck! Online license validation can backfire like drm did: you cannot use offline feature without online service available.
- takeda 6y agoisn't it pretty much the same thing?
- bluedino 6y agoThe best part: "Add this page to your favorites" The link is javascript:window.external.AddFavorite(location.href,%20document.title);
- webdood90 6y agoThis seems catastrophically bad.
- merrvk 6y agoMight need to rename to Microsoft 364 after today.
- CobrastanJorji 6y agoOn the contrary, 2020 is a leap year. In such a year, "Microsoft 365" asserts only two 9s of uptime.
- bleepblorp 6y agoThere's a running joke on /r/sysadmin that Office 365 should be called Office 360. I'm not sure if even that number is low enough.
- sys_64738 6y agoRemember when x5 9s uptime was a thing?
- dotancohen 6y agoFor an office suit? If my computer is on, the LibreOffice is running. I think I once did something stupid like try to patch GoOffice into OpenOffice and couldn't get it to build. But the distro packages still worked!
- nrr 6y agoIt still is, but five nines availability is a different matter entirely!
- PascLeRasc 6y agoYou had to specify if you wanted the nines to be leading or trailing when you signed the SLA.
- dotancohen 6y agoIt's called Office 365 for a reason. 2020 is a leap year, but there is no reason for the outage to occur exactly on February 29 or December 31. Wasn't there also a widely-publicized outage in 2016?
- william_T 6y agoThere was a big one in 2018 involving authentication. A handful of datacenters outside of Houston couldn't get enough power and the shutdown procedures prioritized saving data. Some critical Azure system service was only located in that geographic zone and the data recovery priority caused that low level service to fail its migration. It caused a huge authentication failure across all Azure services globally. MS got blasted about it at Ignite pretty soon after.
- dutch3000 6y agois this a national risk? - think other countries taking out these massive, saas offerings. must be easier than simultaneously targeting thousands of disparately configured, onprem implementations
- tsumnia 6y agoI know many posts in this thread have been jokes, but I do hope this outage helps MS reconsider their offline iterations of Office. I use Office on a near constant basis (writing and building lecture PowerPoints). The subscription model isn't feasible and would ultimately be more expensive in the long term. When Office 2019 was released I quickly picked it up out of worry that MS would shift their academic licensing programs to 365, which they ultimately did.
- muststopmyths 6y agohttps://www.engadget.com/microsoft-non-subscription-office-suite-2021-072127104.html https://www.engadget.com/microsoft-non-subscription-office-s...
- tsumnia 6y agoThanks! I was not aware of this news and I'm incredibly happy to hear it.
- acwan93 6y agoI just finished convincing old-school management to move to Office365/Azure/Teams after we've been doing everything on-premise, saying we can reduce our maintenance and increase in reliability since we're a small operation. Needless to say this isn't a good look for me.
- outworlder 6y agoTheir reliability only looks bad in comparison with other cloud providers. Microsoft can still probably do a much better job compared to a small team.
- Nextgrid 6y agoThe variable here is not just the size & skill of the team but their needs. In most cases a single company or team will need a simpler, smaller-scale deployment, probably running on one or a handful of machines and not requiring the insane complexity that a public cloud provider is running behind the scenes to provide their service. This eliminates a lot of moving parts and things that can fail. Imagine an Nginx server running on a single machine. There's very little that can go wrong here beyond hardware failure (and certain types of failures can be mitigated with things like RAID), and yet it is probably enough to host most internal websites. Now compare this to something like Azure App Service which is obviously more complex to be able to support many tenants, load-balance, etc. There is much more that can go wrong with the entire App Service infrastructure (due to its complexity and moving parts) than with a single machine running Nginx, and the complexity will also delay disaster recovery efforts (this outage is now lasting for more than an hour - you can reinstall an entire Linux web server from scratch in half that time).
- userbinator 6y agoAn internal team is also going to be far more accessible and responsive to the company's needs.
- acwan93 6y ago
- haolez 6y agoI've gone all in on Microsoft for my company (high Office usage) with O365, Azure and friends and I'm regretting this decision. OneDrive for Business is almost unusable on Windows 10. The web apps are incredibly slow. Teams is very inconsistent for users between desktop and mobile usage. It's a mess. I'm planning on switching to Amazon next year. I hope I can find an alternative to the Office situation (maybe offline licenses).
- madengr 6y agoI find One Drive frustrating as they don’t have a gitignore type of option. A few thousand other users agree, but MS has not implemented it.
- chillfox 6y agoDepending upon your needs then perhaps Libre Office, or the WordPerfect Office suite would do?
- mindfulhack 6y agoIf I were offering a free (as in price) alternative to MS Office, I'd first recommend WPS Office over LibreOffice, even though it's closed-source. It has a much better interface out of the box. But I'd use something to block it from phoning home to its probably Chinese government-beholden servers, like OpenSnitch for Linux* or LuLu for macOS. (*) https://github.com/gustavo-iniguez-goya/opensnitch https://github.com/gustavo-iniguez-goya/opensnitch
- jaclaz 6y agoIf I may, in my experience Softmaker is not bad at all (still closed source and Commercial, but affordable), LibreOffice simply does not fit, for very light use LibreOffice is fine, but still has - here and there - too many "quirks".
- mindfulhack 6y agoMy apologies, I meant ONLYOFFICE, not WPS Office. I got them mixed up. 'ONLYOFFICE' is FOSS and the one I meant to recommend!
- gigel82 6y agoMy kids' school district is doing all online over MS Teams; things started crashing and burning around 3PM PST (no "Dad Support" can fix MS's servers though).
- top_post 6y agoZerologon?
- AdmiralAsshat 6y agoOh good, I'm not fired. My company's entire communication infrastructure is just down. Was worried there for a second.
- Nextgrid 6y agoAzure DevOps' Git is also down (a git pull hangs forever) which is a bit surprising. I wouldn't have expected SSH key authentication to depend on Azure Active Directory. Microsoft Teams also appears to be down so it's not just authentication that's affected as I was already logged in and it still doesn't work, just hangs trying to load forever. Edit: appears to be resolved at 1:07 am London time.
- dismalpedigree 6y agoDifficult to tell if this is desired functionality for teams or a bug. It’s inline with its normal performance level.
- jostmey 6y agoMy phone keeps asking me to retype in my Microsoft word password every 30 seconds. Agh!
- aksss 6y agoAt least the guy with the change request was vindicated for now. That must have felt good - about to be crucified for putting in a change that brought down AAD globally, they roll it back and shit's still broken. :D hahaha wudn't me mofos!
- ww520 6y agoDo companies get refund from Microsoft for outage? What does a typical MS SLA look like?
- johnbatch 6y agohttps://azure.microsoft.com/en-us/support/legal/sla/active-directory-ds/v1_0/ https://azure.microsoft.com/en-us/support/legal/sla/active-d...
- steverb 6y agoLooks to be fixed, at least partially.
- smcleod 6y agoO365 (aka Microsoft 365 now) has easily been the most unreliable SaaS product I've been forced to use over the past 8 1/2 years, it's been a real disaster story for several of the clients I used to have and for several of them has resulted in loss of profits and general distrust in Cloud/SaaS software. A few times during outages I managed to get hold of someone that worked at Microsoft who stated something along the lines that they only state that a service is degraded or has an outage if enough customers complain - or if the websites they have to sell / demonstrate products to potential customers are unavailable.
- spondyl 6y agoAhh, this explains why I was unable to log into my personal email account on my work laptop. I thought it was just the network here being overly keen to block requests since I'm relatively new and am still finding out all the quirks This also explains why clicking "Sign in" would randomly download an HTML page (ie logout.html) instead of redirecting to the proper auth endpoint!
- anigbrowl 6y agoThat might explain why I've been unable to download onedrive links for the last day.
- microtherion 6y agoI'm reminded of what Leslie Lamport (who ironically went on to work for Microsoft) once observed: "You know you have a distributed system when the crash of a computer you’ve never heard of stops you from getting any work done."
- bleepblorp 6y agoThere's an unconfirmed rumor that this outage has taken out 911 services in multiple parts of the US. https://www.nbcnewyork.com/news/local/nationwide-reports-of-911-system-outages-cause-not-immediately-clear/2640684 https://www.nbcnewyork.com/news/local/nationwide-reports-of-...
- userbinator 6y ago"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." Ironically found on https://www.microsoft.com/en-us/research/publication/distribution/ https://www.microsoft.com/en-us/research/publication/distrib... The cloud is the ultimate rent-seeker's dream. The world's computing power in the hands of a few enormous companies... who then sell it to you in tiny pieces and charge for every little thing you use on their services. And when something goes wrong, you are powerless to do anything about it.
- core-questions 6y agoThis is why my new strategy is to avoid purchasing mission-critical components of my infrastructure from gigantic vendors. For example, right now I could be using Azure SSO, which would have been free with Office 365 at the level we're bought in at, and then all of my other cloud products would be just as inaccessible as Office 365 is. Even when it is working, I could have an edge case issue and never see a feature added or a bug fixed to help me at all. Or, I could go with a 3rd party authentication company, and get a product that their company lives or dies on. My experience is that, if chosen well, you'll get a much better experience even though you have to do some legwork here and there to make things work together.
- jojobas 6y ago>powerless to do anything about it You can't even sack anybody for it.
- Aperocky 6y agoThe sweetspot of cloud is that it is more expensive to operate your own hardware alone, but cheaper than operating your own hardware with paid professionals to manage those hardware.
- verst 6y agoI wish this were only limited to Office 365. It's most services that use authentication backed by Azure Active Directory. https://status.azure.com/en-us/status https://status.azure.com/en-us/status A few days ago a similar outage occurred with Google Accounts.
- maximilianburke 6y agoNot just Azure / Office365 either; we're using Azure ADB2C for our user authentication and it's down too.
- chasd00 6y agoI noticed some strange behavior around 3-4PM central. I was searching my inbox and saw an error in Outlook "we're unable to search the server" or something like that. I have never seen a message like that from Outlook before.