3 ms·

Microsoft is rolling back some update they pushed that borked Azure AD. Early signs are not pointing to this as a service resilience issue - this appears to be

Microsoft is rolling back some update they pushed that borked Azure AD. Early signs are not pointing to this as a service resilience issue - this appears to be sheer incompetence - pushing an update that was likely not tested well enough that broke pretty much everything. More than that, why aren't updates to Azure AD being rolled out regionally? Why is Azure AD architected in such a way that the entire thing going down can do so much damage? It's pretty hard to be understanding with a preventable screwup with this kind of global scale.

adflux 6y ago
You are making a whole bunch of assumptions here without ANY info about how Microsoft rolls out updates to its Azure services.
- king_magic 6y ago
  Sure, but those assumptions are largely backed up by a single update to Azure AD knocking it out worldwide. End of the day, whatever they pushed was not nearly well tested enough, and they paid for it with a massive global outage that borked pretty much everything they sell.
- regecks 6y ago
  To add another datapoint, we recently faced an issue where Microsoft rolled out a change to their OIDC endpoints, which broke our client's tenant, but not other tenants. (It was crashing with an HTTP 500 if the Accept request header was a certain value: the default value in OpenJDK 8). Some days later and after our complaints, Microsoft rolled back the change. I would call this evidence that they do practice gradual deployments of changes. I would guess that this outage was caused by something more complex than MS not slow-rolling their deployment.
- extrapickles 6y ago
  For something as critical as auth, I would expect most changes to be rolled out over a few days just to make sure something like this cannot happen to everyone. Ideally a change would be shadowed for a few % of traffic before it was allowed to start rolling out.

adflux 6y ago

You are making a whole bunch of assumptions here without ANY info about how Microsoft rolls out updates to its Azure services.

king_magic 6y ago
Sure, but those assumptions are largely backed up by a single update to Azure AD knocking it out worldwide. End of the day, whatever they pushed was not nearly well tested enough, and they paid for it with a massive global outage that borked pretty much everything they sell.
regecks 6y ago
To add another datapoint, we recently faced an issue where Microsoft rolled out a change to their OIDC endpoints, which broke our client's tenant, but not other tenants. (It was crashing with an HTTP 500 if the Accept request header was a certain value: the default value in OpenJDK 8). Some days later and after our complaints, Microsoft rolled back the change. I would call this evidence that they do practice gradual deployments of changes. I would guess that this outage was caused by something more complex than MS not slow-rolling their deployment.
extrapickles 6y ago
For something as critical as auth, I would expect most changes to be rolled out over a few days just to make sure something like this cannot happen to everyone. Ideally a change would be shadowed for a few % of traffic before it was allowed to start rolling out.