10 ms·

Azure Active Directory down

91 points by Decabytes 6y ago

bombcar 6y ago
I was seeing: DX10501:+Signature+validation+failed.+Unable+to+match+keys:+ kid:+'[PII+is+hidden]',+ token:+'[PII+is+hidden]'. and sure enough, it seems it was some authentication thing going sideways
EMM_386 6y ago
It seems to be coming back online now. Strange that HN was down at exactly the same time, although this is reported to be unrelated. > CURRENT STATUS: Engineering teams are currently rolling out mitigation worldwide. Customers should begin seeing recovery at this time, with full mitigation expected within 60 minutes. This message was last updated at 21:12 UTC on 15 March 2021
deathanatos 6y ago
It is a huge pet peeve of mine that, a. the status page update took as long as it did. b. the status page still only claims that AAD is down. You're not "up" if a required dependency of yours is down, IMO, and most of the Azure services right now are severely degraded to the point of not being useful due to the AAD outage. Now, that would make your status page look bad, yes. (And I would clearly message on it that the other outages are fallout of AAD on that.) c. I don't have any real expectation that we'll get a public PM about this. (And a PM must include sufficient information for me to understand what went wrong, and what's been done to prevent it going forward.) I had been starting to wonder what sort of incident it would take for them to actually update the status page. I guess this is it. We've witnessed a number of outages in various services over the past few months, none of which made it to the status page. I pushed a support rep on the issue, and was told that they don't want to cause a panic.
- jrockway 6y ago
  I'm also here because my work is blocked by the inability to do anything on Azure right now. I looked at their past status histories, and they have provided pretty detailed postmortems for much more minor incidents. I don't have any reason to believe that they won't do that this time. I'm new to Azure, so this kind of outage really looks bad in my mind, but it's par for the course with Cloud stuff. They'll fix it and there will be one less thing to break in the future.
  - spondyl 6y ago
    As someone who has been using Azure for 6 months, it's fairly bad but most of the Azure pain is more death by 1000 cuts than large explosions. Documentation that contains triple negatives, UI elements that have no consistency (clicking off a modal on one page does nothing while doing so on another closes it and clears your data), widgets and modals forgetting state when you navigate to different submenus. The sort of unexciting stuff that by itself is ok but it slowly makes you descend into madness.
    - NDizzle 6y ago
      Just wait until you see their made up rules for using their Redis implementation. You have to roll your own Redis to do anything interesting.
      - zanecodes 6y ago
        Their managed PostgreSQL offering also has fun quirks, such as that you have to connect using `username@db_name`, but your actual username in the database is still just `username`, so any third party software needs to support using a different username to connect than it uses to perform user-related queries (they have some sort of application-level load balancer in front of it that uses `db_name` to route the connection).
        bombcar 6y ago
        This is ... something - wow - they couldn't just make the actual username "username@db_name"
        zanecodes 6y ago
        Nope. Their load balancer seems to strip the `@db_name` from connections, and anyway some software doesn't like having an @ in Postgres usernames (which is probably also a sign it's vulnerable to SQL injections)
        6y ago
- breatheoften 6y ago
  Bad status pages are a very bad sign in my opinion. I had to use the IBM cloud for a project once upon a time and the status monitoring was almost always wrong ... I really hope that as customers we can succeed in making "excellent status reporting" an absolute minimum requirement for cloud services. Observing the quality of the status page reporting history is one of the first things I do now when evaluating a new service -- and no I'm not just looking for a page that shows all greens -- I'm looking for a page that connects to some conceivably meaningful metric as well as detailed reports about major and minor outages ... something to prove that they actually even know if their service is working at a given point in time ... The IBM cloud page was a page that tended to update in response to my own issue reports (usually if I also pointed out that nothing about the outage appeared on the status page ...). I don't think that's at all an acceptable standard ...
  - jmartens 6y ago
    Are status pages the right answer though? They seem to be a failure all around, so let's find a better solution.
- jmartens 6y ago
  Honest question for the sake of discussion/learning: What impact would it have on you if (1) the status page was updated instantly and (2) All impacted capabilities were marked as down/degraded?
NDizzle 6y ago
Affecting Teams, Azdo, and likely more services. It turned the whole afternoon into learning time at our company. Thankfully our Okta integration goes through our on prem AD servers and not purely AAD. Otherwise I wouldn't be able to get to learning resources which authenticate through AD!
spondyl 6y ago
As someone working at a company that pays for Azure support, it's pretty sad that Microsoft 365's Status Twitter account has been providing more insight than Azure's own status page https://twitter.com/MSFT365Status/status/1371554704518352896 https://twitter.com/MSFT365Status/status/1371554704518352896 They actually acknowledged it was due to a "recent change to an authentication system" where Azure Support just vaguely mentions an outage
- llama052 6y ago
  I have zero expectations from Azure support as a whole, in fact 80% of the time I'll rebuild production services in Azure to avoid having them tell me to do just that in most tickets. It is abysmal levels of actual support.
heyflyguy 6y ago
I am naive, but it is crazy to me that we experience outages of any magnitude like this from any major supplier (GOOG/MS/AMZN). I get alot of crap for being crotchety about making my backups in and out of the cloud, and it's stuff like this that proves my point!
- ocdtrekkie 6y ago
  "The more they overthink the plumbing, the easier it is to stop up the drain." - Scotty Trying to provide globally available, replicated service that meets every and all needs 24/7/365 is basically making Active Directory about 600x more complicated than it would be otherwise. It's basically impossible for a service that complicated to meet the uptime of... installing Windows Server 2019 on a VM and patching it monthly. (Bear in mind, if I know when my business is not affected by an outage, I can do it without being disruptive. Microsoft, by definition, cannot. Every moment is critical since their customers are everywhere.) There are types of businesses to which the former might be a necessary solution, but most would be better off with the latter. It'll be really interesting when people start realizing the cloud is mostly just a scam to get people on subscription revenue streams, and not actually providing any greater reliability or less management overhead than what they had before.
  - heyflyguy 6y ago
    I agree!
  - Spooky23 6y ago
    Plus, with a massive customer base and eating their own dog food makes their schedule be a priority over other customers. Your priority is lower than theirs. Microsoft is always a shitshow this time of year. Infrastructure changes usually land ahead of spring releases.
    - ocdtrekkie 6y ago
      Indeed, no matter how much we pay Microsoft, they will not care about our business as we care about our business.
Severian 6y ago
Teams completely borked because of this. My employer hugely depends on this as we all WFH, and basically has stopped all communication outside of email (thank god we are on-prem).
king_magic 6y ago
This should be a huge shitshow for Microsoft. This never should have happened. Azure AD taking down pretty much the entirety of Azure / O365 / Teams is frankly inexcusable. Astounding incompetence, and an astonishing single point of failure that needs to be re-architected.
- macintux 6y ago
  In fairness to Microsoft, there's no way to make authentication not a single point of failure. You have to make the service as resilient as possible, which clearly they failed at, but you can't very well fail over from your AD service to something else.
  - king_magic 6y ago
    Microsoft is rolling back some update they pushed that borked Azure AD. Early signs are not pointing to this as a service resilience issue - this appears to be sheer incompetence - pushing an update that was likely not tested well enough that broke pretty much everything. More than that, why aren't updates to Azure AD being rolled out regionally? Why is Azure AD architected in such a way that the entire thing going down can do so much damage? It's pretty hard to be understanding with a preventable screwup with this kind of global scale.
    - adflux 6y ago
      You are making a whole bunch of assumptions here without ANY info about how Microsoft rolls out updates to its Azure services.
      - king_magic 6y ago
        Sure, but those assumptions are largely backed up by a single update to Azure AD knocking it out worldwide. End of the day, whatever they pushed was not nearly well tested enough, and they paid for it with a massive global outage that borked pretty much everything they sell.
      - 6y ago
ocdtrekkie 6y ago
Every time someone tells me we should move to the cloud, it seems like there's an outage the same day. Microsoft isn't actually any good at managing your IT infrastructure for you. Between Exchange 365 outages and Azure AD outages, I'm not sure how anyone thinks moving Windows infrastructure to the cloud is not an existential risk to business operations.
hateful 6y ago
cloud=a single point of failure for the whole world!
chevman 6y ago
Was interesting watching this happen today at a large Fortune xx that is a year or so in to wide scale Azure rollout. Took out Outlook, Teams and all meetings over the course of 5 minutes, then moved on to internal service alarms going off across multiple (10+) major customer facing digital products that got vaporized due (presumably) to various dependencies on AD. I think some of this was service accounts on backend that lost connection to various dbs, file systems, etc. What a total shit show. Company basically down from 2pm CST and barely coming back up 5 hours later. IT Buddy who is 2 people away from CIO pinged me and said “DNS issue, data center power outage or Azure just crapped its pants.”
dnndev 6y ago
I run a saas company and we use azure ad. B2B sales. Good thing is we are down but so are they. This is strangely not that bad. Left the office early today
_Understated_ 6y ago
You know, in some ways I'm actually glad that Azure AD is down: I ran a SQL update and right after it I started seeing errors and thought I'd done something wrong... I've been shitting myself for an hour :D
JCM9 6y ago
Azure AD instability has been a big pain point for us for a while. We’ve had more of these outages than I care to remember. One the one hand the ability to have one central auth between on-prem and other services is great but every time this happens so many services just stop working. Microsoft really needs to fix their stability issues.
jiggawatts 6y ago
What's crazy is that apparently there's a "fallback" Azure AD authentication endpoint that is a delayed read-only replica of the production endpoint with a subset of the features. During the last global AAD outage, they claimed that they learned their lesson and improved the failover to the fallback endpoint. That didn't seem to happen... What is more crazy is that low level service-to-service authentication in Azure is primarily based on Azure AD. This includes Key Vault and Storage Accounts. Both were mentioned in the Microsoft notification page as affected by this outage. How can anyone build a reliable service on top of Azure if what is essentially the failure of the "Office 365 login" also breaks your VM's access to its SSL certificates (or whatever)?

91 points by Decabytes 6y ago

I was seeing: DX10501:+Signature+validation+failed.+Unable+to+match+keys:+ kid:+'[PII+is+hidden]',+ token:+'[PII+is+hidden]'. and sure enough, it seems it was some authentication thing going sideways

EMM_386 6y ago

It seems to be coming back online now. Strange that HN was down at exactly the same time, although this is reported to be unrelated. > CURRENT STATUS: Engineering teams are currently rolling out mitigation worldwide. Customers should begin seeing recovery at this time, with full mitigation expected within 60 minutes. This message was last updated at 21:12 UTC on 15 March 2021

deathanatos 6y ago

It is a huge pet peeve of mine that, a. the status page update took as long as it did. b. the status page still only claims that AAD is down. You're not "up" if a required dependency of yours is down, IMO, and most of the Azure services right now are severely degraded to the point of not being useful due to the AAD outage. Now, that would make your status page look bad, yes. (And I would clearly message on it that the other outages are fallout of AAD on that.) c. I don't have any real expectation that we'll get a public PM about this. (And a PM must include sufficient information for me to understand what went wrong, and what's been done to prevent it going forward.) I had been starting to wonder what sort of incident it would take for them to actually update the status page. I guess this is it. We've witnessed a number of outages in various services over the past few months, none of which made it to the status page. I pushed a support rep on the issue, and was told that they don't want to cause a panic.

jrockway 6y ago
I'm also here because my work is blocked by the inability to do anything on Azure right now. I looked at their past status histories, and they have provided pretty detailed postmortems for much more minor incidents. I don't have any reason to believe that they won't do that this time. I'm new to Azure, so this kind of outage really looks bad in my mind, but it's par for the course with Cloud stuff. They'll fix it and there will be one less thing to break in the future.
- spondyl 6y ago
  As someone who has been using Azure for 6 months, it's fairly bad but most of the Azure pain is more death by 1000 cuts than large explosions. Documentation that contains triple negatives, UI elements that have no consistency (clicking off a modal on one page does nothing while doing so on another closes it and clears your data), widgets and modals forgetting state when you navigate to different submenus. The sort of unexciting stuff that by itself is ok but it slowly makes you descend into madness.
  - NDizzle 6y ago
    Just wait until you see their made up rules for using their Redis implementation. You have to roll your own Redis to do anything interesting.
    - zanecodes 6y ago
      Their managed PostgreSQL offering also has fun quirks, such as that you have to connect using `username@db_name`, but your actual username in the database is still just `username`, so any third party software needs to support using a different username to connect than it uses to perform user-related queries (they have some sort of application-level load balancer in front of it that uses `db_name` to route the connection).
      - bombcar 6y ago
        This is ... something - wow - they couldn't just make the actual username "username@db_name"
        zanecodes 6y ago
        Nope. Their load balancer seems to strip the `@db_name` from connections, and anyway some software doesn't like having an @ in Postgres usernames (which is probably also a sign it's vulnerable to SQL injections)
      - 6y ago
breatheoften 6y ago
Bad status pages are a very bad sign in my opinion. I had to use the IBM cloud for a project once upon a time and the status monitoring was almost always wrong ... I really hope that as customers we can succeed in making "excellent status reporting" an absolute minimum requirement for cloud services. Observing the quality of the status page reporting history is one of the first things I do now when evaluating a new service -- and no I'm not just looking for a page that shows all greens -- I'm looking for a page that connects to some conceivably meaningful metric as well as detailed reports about major and minor outages ... something to prove that they actually even know if their service is working at a given point in time ... The IBM cloud page was a page that tended to update in response to my own issue reports (usually if I also pointed out that nothing about the outage appeared on the status page ...). I don't think that's at all an acceptable standard ...
- jmartens 6y ago
  Are status pages the right answer though? They seem to be a failure all around, so let's find a better solution.
jmartens 6y ago
Honest question for the sake of discussion/learning: What impact would it have on you if (1) the status page was updated instantly and (2) All impacted capabilities were marked as down/degraded?

NDizzle 6y ago

Affecting Teams, Azdo, and likely more services. It turned the whole afternoon into learning time at our company. Thankfully our Okta integration goes through our on prem AD servers and not purely AAD. Otherwise I wouldn't be able to get to learning resources which authenticate through AD!

spondyl 6y ago

As someone working at a company that pays for Azure support, it's pretty sad that Microsoft 365's Status Twitter account has been providing more insight than Azure's own status page https://twitter.com/MSFT365Status/status/1371554704518352896 https://twitter.com/MSFT365Status/status/1371554704518352896 They actually acknowledged it was due to a "recent change to an authentication system" where Azure Support just vaguely mentions an outage

llama052 6y ago
I have zero expectations from Azure support as a whole, in fact 80% of the time I'll rebuild production services in Azure to avoid having them tell me to do just that in most tickets. It is abysmal levels of actual support.

heyflyguy 6y ago

I am naive, but it is crazy to me that we experience outages of any magnitude like this from any major supplier (GOOG/MS/AMZN). I get alot of crap for being crotchety about making my backups in and out of the cloud, and it's stuff like this that proves my point!

ocdtrekkie 6y ago
"The more they overthink the plumbing, the easier it is to stop up the drain." - Scotty Trying to provide globally available, replicated service that meets every and all needs 24/7/365 is basically making Active Directory about 600x more complicated than it would be otherwise. It's basically impossible for a service that complicated to meet the uptime of... installing Windows Server 2019 on a VM and patching it monthly. (Bear in mind, if I know when my business is not affected by an outage, I can do it without being disruptive. Microsoft, by definition, cannot. Every moment is critical since their customers are everywhere.) There are types of businesses to which the former might be a necessary solution, but most would be better off with the latter. It'll be really interesting when people start realizing the cloud is mostly just a scam to get people on subscription revenue streams, and not actually providing any greater reliability or less management overhead than what they had before.
- heyflyguy 6y ago
  I agree!
- Spooky23 6y ago
  Plus, with a massive customer base and eating their own dog food makes their schedule be a priority over other customers. Your priority is lower than theirs. Microsoft is always a shitshow this time of year. Infrastructure changes usually land ahead of spring releases.
  - ocdtrekkie 6y ago
    Indeed, no matter how much we pay Microsoft, they will not care about our business as we care about our business.

Severian 6y ago

Teams completely borked because of this. My employer hugely depends on this as we all WFH, and basically has stopped all communication outside of email (thank god we are on-prem).

king_magic 6y ago

This should be a huge shitshow for Microsoft. This never should have happened. Azure AD taking down pretty much the entirety of Azure / O365 / Teams is frankly inexcusable. Astounding incompetence, and an astonishing single point of failure that needs to be re-architected.

macintux 6y ago
In fairness to Microsoft, there's no way to make authentication not a single point of failure. You have to make the service as resilient as possible, which clearly they failed at, but you can't very well fail over from your AD service to something else.
- king_magic 6y ago
  Microsoft is rolling back some update they pushed that borked Azure AD. Early signs are not pointing to this as a service resilience issue - this appears to be sheer incompetence - pushing an update that was likely not tested well enough that broke pretty much everything. More than that, why aren't updates to Azure AD being rolled out regionally? Why is Azure AD architected in such a way that the entire thing going down can do so much damage? It's pretty hard to be understanding with a preventable screwup with this kind of global scale.
  - adflux 6y ago
    You are making a whole bunch of assumptions here without ANY info about how Microsoft rolls out updates to its Azure services.
    - king_magic 6y ago
      Sure, but those assumptions are largely backed up by a single update to Azure AD knocking it out worldwide. End of the day, whatever they pushed was not nearly well tested enough, and they paid for it with a massive global outage that borked pretty much everything they sell.
    - 6y ago

ocdtrekkie 6y ago

Every time someone tells me we should move to the cloud, it seems like there's an outage the same day. Microsoft isn't actually any good at managing your IT infrastructure for you. Between Exchange 365 outages and Azure AD outages, I'm not sure how anyone thinks moving Windows infrastructure to the cloud is not an existential risk to business operations.

hateful 6y ago

cloud=a single point of failure for the whole world!

chevman 6y ago

Was interesting watching this happen today at a large Fortune xx that is a year or so in to wide scale Azure rollout. Took out Outlook, Teams and all meetings over the course of 5 minutes, then moved on to internal service alarms going off across multiple (10+) major customer facing digital products that got vaporized due (presumably) to various dependencies on AD. I think some of this was service accounts on backend that lost connection to various dbs, file systems, etc. What a total shit show. Company basically down from 2pm CST and barely coming back up 5 hours later. IT Buddy who is 2 people away from CIO pinged me and said “DNS issue, data center power outage or Azure just crapped its pants.”

dnndev 6y ago

I run a saas company and we use azure ad. B2B sales. Good thing is we are down but so are they. This is strangely not that bad. Left the office early today

_Understated_ 6y ago

You know, in some ways I'm actually glad that Azure AD is down: I ran a SQL update and right after it I started seeing errors and thought I'd done something wrong... I've been shitting myself for an hour :D

JCM9 6y ago

Azure AD instability has been a big pain point for us for a while. We’ve had more of these outages than I care to remember. One the one hand the ability to have one central auth between on-prem and other services is great but every time this happens so many services just stop working. Microsoft really needs to fix their stability issues.

jiggawatts 6y ago

What's crazy is that apparently there's a "fallback" Azure AD authentication endpoint that is a delayed read-only replica of the production endpoint with a subset of the features. During the last global AAD outage, they claimed that they learned their lesson and improved the failover to the fallback endpoint. That didn't seem to happen... What is more crazy is that low level service-to-service authentication in Azure is primarily based on Azure AD. This includes Key Vault and Storage Accounts. Both were mentioned in the Microsoft notification page as affected by this outage. How can anyone build a reliable service on top of Azure if what is essentially the failure of the "Office 365 login" also breaks your VM's access to its SSL certificates (or whatever)?