14 ms·
Office 365 global authentication outage
- Bie-ing_Adik 8y agoim just gonna say this: AAAAAH! HAHAHAHAHAHAHA AAAAHH! HAHAHAHAHA! suckers! try something that works: Libre Office. https://www.libreoffice.org/ https://www.libreoffice.org/
- mrmondo 8y agoThe status page reads as below (I've copied the text here as it a) may not be accessible to everyone and b) often has old information removed / edited, c) this status page / information is usually pay-walled and only available once you've authenticated): -- Microsoft 365 Service health status Title: Unable to access Microsoft 365 services User impact: Affected users are unable to authenticate to and access Microsoft 365 services. Current status: We've received reports of an issue affecting access to Microsoft 365 services. We've identified a degraded portion of infrastructure that manages authentication requests and have restarted that infrastructure to mitigate impact. Scope of impact: This issue may potentially affect any of your users attempting to access Microsoft 365 services. Start time: Tuesday, January 29, 2019, at 9:15 PM UTC Preliminary root cause: A portion of infrastructure that manages authentication requests is degraded, affecting access to one or more Microsoft 365 services.
- amaccuish 8y agoPlease don't clutter the comments by copying and pasting what is already a fairly succinct page.
- GuyPostington 8y agoI don't agree with this. Not everyone wants to click an extra link when browsing on a mobile device.
- larkeith 8y agoAlso to note: status pages are, by their nature, transient. Having a record of what it previously said is helpful context for any discussion regarding those prior contents.
- mrmondo 8y agoAs I said below the "status page which a) may not be accessible to everyone and b) often has old information removed"
- amaccuish 8y agoYou made that comment after I had made mine, so I had not seen it.
- mrmondo 8y agoNot sure why I was down-voted for this, it's a direct copy-paste of the service status page which a) may not be accessible to everyone and b) often has old information removed.
- deleted 8y ago[deleted]
- dragonwriter 8y ago> Not sure why I was down-voted for this, it's a direct copy-paste of the service status page Being a direct copy and paste of th page linked to the thread without additional explanation is probably why it was downvoted.
- rickycook 8y ago... there's a pretty damn good explanation given in the comment actually and it's pretty obvious regardless: status pages are highly transient. it's probably good to have a record of what's being discussed?
- amaccuish 8y agoThey edited their comment after which now makes it clearer why copy/pasted.
- seekbeak 8y agoI love how they are just "turning it off and on again".
- SteveNuts 8y agoThat's literally the fix for almost anything Microsoft. If it goes any further than that it's "Restore from backup".
- reaperducer 8y ago"Thank you for calling 310-DELL. Have to tried restarting your computer?" (Older Canadian tech types should recognize that.)
- peterwwillis 8y agoIt's pretty common practice for complex systems. If it's already dead, and nobody knows what's wrong or how to fix it, you try restarting it. (normally you first redirect traffic to a different region, but apparently big orgs are still running critical infrastructure with changes that affect all regions and can't be backed out)
- londons_explore 8y agoKubernetes doesn't even have a "restart this deployment" ability...
- abalone 8y agoUpdate: "Preliminary root cause: A portion of third-party managed network infrastructure that facilitates authentication requests and access to Microsoft 365 services was degraded." (Emphasis mine.)
- snazz 8y agoThat doesn’t seem right. For something so critical to many of their corporate mega-customers, there shouldn’t be any “third-party managed” infrastructure at all.
- mrmondo 8y agoMore so that there should be redundancy if it's a critical system.
- londons_explore 8y ago"network infrastructure" - it'll be a fiber cut or failed router, together with a failover process that didn't work or was overlooked.
- tatersolid 8y agoThat’s not how the internet works. Every large provider has third-party connectivity. There’s no wire from Microsoft to your house. You can buy connectivity from 5 diverse providers per data center, but if one of them continues to advertise your routes while not being able to actually pass traffic to your destination, you’re “down” to some people through no fault of your own. I’ve had this happen with AT&T, XO/Verizon, and Cogent in recent years. It’s not uncommon. In all cases it was mis-configured ISP routers. Usually a support engineer admits a QoS or DDoS-protection configuration went wrong.
- 98codes 8y agoLooks like it's started with a DNS outage @ Level3: https://downdetector.com/status/level3/map/ https://downdetector.com/status/level3/map/
- mrmondo 8y agoWe thought that too - gwmigprda.aadg.windows.net.nsatc.net seems / seemed to be failing to resolve, but then it seemed like it was internal to their infrastructure - which they later added to their status page (and clarified it was their authentication infrastructure which needed to be 'rebooted').
- est31 8y agoI don't understand, could you explain? Level3 may be the biggest backbone provider but I doubt that Microsoft is peering only with them are they? This is maybe something that a smaller company might do but Microsoft seems too large for that.
- zeeZ 8y agoMy ISP (Telefonica) currently doesn't return addresses for login.microsoftonline.de (cloud Germany) while Google's DNS servers do. I wonder if that's related.
- mrmondo 8y agoInteresting, I suspecting their 'restart' of 'authentication infrastructure' may be a rolling restart and perhaps they have not got to Germany yet, perhaps by using Google's DNS different 'authentication infrastructure' is being provided?
- torstenvl 8y agoI wonder if this will be a wake up call to Microsoft to maybe allow offline use of things like OneNote. It's really unacceptable to have something that important so reliant on an online service. (I doubt they'll change their tune. Theee years of complaints about this and they still refuse to allow local notebook storage on Mac, so it seems unlikely that they'd change their minds now.)
- est31 8y agoPiracy of Microsoft products that allow offline use is rampant, especially in poorer countries. A cloud service is a really good tool to prevent piracy.
- judge2020 8y agoThe only issue is that Office is extremely easy to pirate, and it's pirate-able by the (you guessed it!) offline authentication service, KMS. If it's so easy to pirate office then the cloud-based authentication service only serves to cause issues for those who legally purchased the software.
- est31 8y agoI don't mean the downloadable thing, I mean the online thing that runs in the browser.
- amaccuish 8y agoSorry how could you pirate "the online thing that runs in the browser"? The problem is that recent OneNote versions (at least on Mac), require Microsoft signin and can't even open local note files.
- est31 8y agoWell, you can't, that's my point. That's why it's so great for Microsoft: Nobody to pirate it any more, means more revenue.
- mrmondo 8y agoUpdate: This apparently affected all of Microsoft Cloud products - not just Office365 as I stated in the title. Source: https://twitter.com/AzureSupport/status/1090366788972404737 https://twitter.com/AzureSupport/status/1090366788972404737 *Edit: I believe this includes Azure-AD, which I would assume would affect people using it for auth for non-Microsoft cloud products (perhaps even on-prem (if people do that?)).
- save_ferris 8y agocan confirm, our SSO uses Azure. I'm blocked from my remote environments until the service is back up.
- sixdimensional 8y agoAlso confirmed, I was having problems with it all day, working in the Azure portal.
- Animats 8y agoGithub is still up.
- quickthrower2 8y agoThey’re probably still running on their previous systems not Azure
- Haga 8y agoCloud always was a single point of failure that's why it failed in the past that's why it will fail when the first virus ignites a forest fire. The truth is the users needs weight in little compared too drm and control of pricing via shakedown methods.
- robbyt 8y agoA well run service run by a world-class team will provide global availability performance and service that small teams are incapable of. However, in this case, I don't think the service is very well run, so there's downtime. Not really an issue with "the cloud", this is just an issue because of bad management and/or design of the system.
- synaesthesisx 8y agoGlad I purchased a standalone license (although I do have Office365 through work).
- mrmondo 8y agoDoes that mean that you don't (ever) have to login to Microsoft / Office365 online services with the desktop software? I have been hearing that people are having issues with desktop apps asking them to log back in to use them - and they can't.
- acid303 8y agoYou don't have to log in anywhere. MS sells Office 365 and Office 2019, 365 being a subscription service and 2019 a one-time purchase. People you talk about probably had issues with desktop apps that were a part of O365.
- bekimdisha 8y agoFunny enough, GMAIL had regional outages yesterday too.
- theDoug 8y agoCame here to make the same observation. Good on teams at Microsoft to not allow themselves to make a "we're more reliable than G Suite" joke.
- kbumsik 8y agoFaceTime (a little bit different case though) 2 days ago, GMail yesterday, and Office 365 today.
- conception 8y agoOr not funny. Suspicious coincidence. Hope it is just that and not outside actors testing things.
- deleted 8y ago[deleted]
- lelandbatey 8y agoYeah, my theory is that it's just about 2 weeks after everyone is back from holidays, which means all the engineers are back and ready to ship features and changes after nothing's been touched during the holidays. All this "go fever" is leading to somewhat reckless modifications of production systems.
- altmind 8y agoI'm just assembling the non-comprehensive list of office 365 outages or login problems in the last seven months (r/sysadmin): 6days ago: https://www.reddit.com/r/sysadmin/comments/ajavl8/its_that_time_again_anyone_having_office_365/ https://www.reddit.com/r/sysadmin/comments/ajavl8/its_that_t... 1mo: https://www.reddit.com/r/sysadmin/comments/a4w7ap/office365_outage/ https://www.reddit.com/r/sysadmin/comments/a4w7ap/office365_... 2mo: https://www.reddit.com/r/sysadmin/comments/9ygjpv/psa_microsoft_azure_mfa_is_down_limited/ https://www.reddit.com/r/sysadmin/comments/9ygjpv/psa_micros... 2mo: https://www.reddit.com/r/sysadmin/comments/a0vepo/office_365_mfa_down_again/ https://www.reddit.com/r/sysadmin/comments/a0vepo/office_365... 3mo: https://www.reddit.com/r/sysadmin/comments/9nbst7/office_365_outage/ https://www.reddit.com/r/sysadmin/comments/9nbst7/office_365... 4mo: https://www.reddit.com/r/sysadmin/comments/9cuwhs/loginmicrosoftcom_down_for_everyone_or_just_me/ https://www.reddit.com/r/sysadmin/comments/9cuwhs/loginmicro... 5mo: https://www.reddit.com/r/sysadmin/comments/9bihjl/office_365_down_for_anyone_else_uk/ https://www.reddit.com/r/sysadmin/comments/9bihjl/office_365... 7mo: https://www.reddit.com/r/sysadmin/comments/8r9y3w/365_outage/ https://www.reddit.com/r/sysadmin/comments/8r9y3w/365_outage...
- PascLeRasc 8y agoI love the top comment: "Pretty sure It's Office 361 at this point"
- mrmondo 8y agoIf that, been an enterprise customer with a company that is a gold partner and been using Office365 for 7 years now - it's down so often we stopped monitoring it, easily the most flaky, unreliable, slow and poorly managed online service I've personally used. (Words are my own, not necessarily those of my employeer(s) etc...)
- brobdingnagians 8y agoMy company has a subscription and the search functionality mysteriously and randomly doesn't work at times, or is really slow... I have to guess which one it is _this_ time when it starts acting up...
- yegle 8y agoLovely, there's this link on the page (doesn't work on Chrome): Add this page to your favorites
- ronsor 8y agoIt's using the old IE window.external
- sqd 8y agoNot sure if it is related, but the safelink service (the thing that wraps a link with a jump) is down at my school's mail service (that uses MS suite).
- mrmondo 8y agoThis also impacted "US government cloud resources": https://twitter.com/AzureSupport/status/1090379738424569859 https://twitter.com/AzureSupport/status/1090379738424569859 Also from ITNews: https://www.itnews.com.au/news/microsoft-hit-by-global-office-365-login-problems-518580 https://www.itnews.com.au/news/microsoft-hit-by-global-offic...
- shereadsthenews 8y agoIncredibly this has taken down typingclub.com
- ed_elliott_asc 8y agoWww.dayssincelastazuremajoroutage.com is constantly down because it is hosted on azure
- mrmondo 8y agoSaid domain is not registered, While I suspect your comment is just aimed at being humorous it may be taken by some as a bit of a troll comment.
- phs318u 8y agoSomeone pointed me to this [0], and I wonder if it is related? Maybe Microsoft tried some changes in response? [0] https://medium.com/@lukeberner/how-i-abused-2fa-to-maintain-persistence-after-a-password-change-google-microsoft-instagram-7e3f455b71a1 https://medium.com/@lukeberner/how-i-abused-2fa-to-maintain-...
- jenna_talia 8y agoHow is it that everyone keeps screwing themselves with MSO when even if you pay for commercial use for libre office you come out way ahead. If you want to collaborate on a document or spreadsheet, you dont need to prostitute yourself to microsoft, its a case of being so self important and lazy that one more click or shifting between a telepresence application and an office suite application is to hard. All you people complaining about being down because msoffice products are unavailable need to get a grip on how to use a computer.
- mrmondo 8y agoWhile I agree with your sentiments about the products, MS as a company and their software model, some of us are forced to use their products at work and of course some of us have tried to improve things and move away to more standards based and ethical software we don't always win (no pun intended) (and words are my own etc...)
- jugg1es 8y agoMany people's work computers come preinstalled with MS products and have no choice. You might need to get a grip on how the corporate world works.
- code4tee 8y agoMicrosoft’s auth infrastructure has been a hot mess over the last 12 month. These outages are getting old.
- a-dub 8y agoit was a marketing stunt! "hey everybody! you know that big google outage today? we have a cloud too! it's big, and it goes down every once in a while just like google's and amazon's! we're a real internet company now too!"
- post_break 8y agoDoes anyone know how to get office 365 pro plus installed on a server to stop deactivating itself? Should we have just paid the insane amount for office enterprise or whatever the one time fee version?
- jammygit 8y agoI know people make a big deal out of availability, but I personally don't usually mind that much when things go offline. I'm sure it could cause serious problems in specific situations though
- pointytrees 8y agoI see these comment sections get somewhat rowdy and negative. Anyone care to comment on the other side of things? Are there any "it may be down, but it is up quite often and saves us heaps of time and labor, totally worth the occasional outage" type experiences? We're considering G Suite and O365 soon.
- altmind 8y agoYes, there are some great things about O365. For example, I often hear that O365 licenses are a bliss compared to complexity of microsoft enterprise licensing, when even ms reps don't agree how much licenses you need.
- pointytrees 8y agoYes, we're quite familiar with MS license audits and the auditors don't even know how everything should be licensed. Good point, thanks.
- techfoolery 8y agoTo be fair, "the licensing isn't completely miserable like their other product" isn't exactly a selling point. My two cents from what I've seen (note mainly see GSuite sie of things).. You can get much more productivity & actual collaboration with G Suite, dependent on a few factors. One, there has to be total buy-in @ the executive level because you most likely need to completely re-think how work gets done across every function. Often, we do what we've been doing and don't see the full picture because of that existing perspective. So that requires a legit G Suite partner to help execute change management as for O365 I imagine. It's not so much a risk unless you do it for the wrong reasons; They've done it enough times to have a proven migration formula. Saving licensing costs for example should be on the bottom of your considerations because that more or less evens out and becomes irrelevant.
- dingle_thunk 8y agoGo check out /r/office365, this is a fairly irrelevant question to this discussion.
- aboutruby 8y ago> I would say is pretty fair and based on today’s events everyone affected could be eligible for 25% discount on their bill as the service credit for breaching 99.9% SLA is 25% https://www.reddit.com/r/sysadmin/comments/ajavl8/its_that_time_again_anyone_having_office_365/eevk047/ https://www.reddit.com/r/sysadmin/comments/ajavl8/its_that_t...
- mrmondo 8y agoThe Office365 status page now states "There are currently no known issues preventing you from signing in to your Office 365 service health dashboard." and when you click the link and are forced to login (now that auth appears to be working for me at least) you're presented with an error preventing you from seeing service status or historical events stating "You don’t have permission to access this page or perform this action." Screenshots: https://imgur.com/a/pALWiIR https://imgur.com/a/pALWiIR
- grumblepeet 8y agoAnd, while this may not be connected, Microsoft also seem to have deleted all of our Sensitive Information Policies in the Office 365 Security and Compliance Center overnight too. Policies that we running yesterday all seem to have been turned off or deleted this morning. This is a HUGE security risk. As I say it could be coinicidence but I'm not so sure.
- tonyedgecombe 8y agoAround the world millions of accountants were able to go home early as they couldn't twiddle some cells in their spreadsheets, middle managers were spared from death by PowerPoint and countless people weren't told about their use of passive voice by Word's grammar checker.
- fredley 8y agoChrist I cannot stand Office 365. For some reason, I cannot open files that are in my OneDrive folder without O365 authentication shitting itself, and refusing to save files. This means to edit files that are in OneDrive, I have to move them out of OneDrive, open them, edit them and save them, then move the back into OneDrive. The situation with having a different fucking login/account for every single fucking Microsoft service (Skype, Office, etc.) even when you're on the business tier for them all is insane and endlessly frustrating.
- perlgeek 8y agoThere seems to be an attempt at a single-sign on solution, but when I used it about 2 years ago, it sent me a on trip to about 6 different domains (live.com, microsoft.com, office365.com, something with sharepoint in the name, and several subdomains of those). In the past, I've given folks the tip to not open links if they send you to a different domain than the link suggests -- by that standard, the SSO looks like a complete scam. It takes some serious knowledge of the Microsoft products to know that those domains all belong to the same corporate entity.
- rcarmo 8y agoIt is also a pain for parental controls if your kids use Kahoot or micro:bit with MS auth. I've had to whitelist a bunch of non-obvious subdomains as well.
- mavhc 8y agoOffice 365 Message Encryption looks exactly like phishing. (and is also useless as it doesn't have a second channel) Good news: You can pay more to get phishing warnings in Office 365.... spooky
- jedieaston 8y agoCan you clarify what you mean by second channel?
- 8y ago
- jxdxbx 8y agoI need 365 since it can do some things Google docs can’t do. But, a while ago there was a widespread bug making is so that some percentage of people who had used the Mac or iOS apps couldn’t sign in. It was never officially acknowledged, and only fixed due to random people getting journalists to write about an emailing randome people on other teams within Microsoft. Their cloud stuff is good but the auth and account stuff is a total mess.
- lisper 8y agoI'm still using my copy of Office2008. It still works just fine. It never has an "authentication outage". And, as a bonus, it costs me nothing to continue to use it. I would feel sorry for all of the poor shmoes who are locked out of their office suites, but I'm sorry, you brought this no yourselves when you accepted a subscription model. If people just said no to the subscription model, no one would have this problem.
- deusex_ 8y agoYour personal home use case of Office is not really relevant here. Good luck organizing a huge company on that.
- lisper 8y agoCompanies are even more relevant because they have more leverage as buyers.
- dingle_thunk 8y agoI don't think the complaints are about office suite access, as that has a fallback in case of failed auth. It's more about IM/VOIP/email, OneDrive/SharePoint, Azure cloud, etc.
- flippyhead 8y agoMaybe it is just me but... The interest we here at HN have in showcasing service outages seems like it's mostly because we want to point to some big/other company and say "see they went down!" so we can feel better about it when our own services go down.
- brodsky 8y agoIt may be that to a degree, but there is also a desire to point out reliability deficiencies in a product or platform that inevitably gets pushed down everyone's throats by management with low resistance to kool-aid.
- forgotAgain 8y agoIronic to read all the negativity here when on the day after the OP Microsoft will announce quarterly earnings which will undoubtedly be driven by continued "Enterprise buy in to Microsoft Cloud Solutions".
- DigiMortal 8y agoCould this have affected my ability to log into Skype for Business? I could log in with my home wifi, but not my cell phone hotspot. Didn't think much of it at the time!
- mrmondo 8y agoYes, Skype / Skype for business uses Microsoft’s centralised servers including for authentication.
- mrmondo 8y agoand today Microsoft's "Xbox live" cloud service is down and it's causing Xbox's to fail to boot: https://techcrunch.com/2019/01/30/xbox-one-black-screen/ https://techcrunch.com/2019/01/30/xbox-one-black-screen/