6 ms·
Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/ https://azure.status.microsoft/en-gb/status/history/ Timeline 15:45 UTC
by croemer 1y ago
Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/ https://azure.status.microsoft/en-gb/status/history/
Timeline
15:45 UTC on 29 October 2025 – Customer impact began.
16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered.
16:15 UTC on 29 October 2025 – We began the investigation and started to examine configuration changes within AFD.
16:18 UTC on 29 October 2025 – Initial communication posted to our public status page.
16:20 UTC on 29 October 2025 – Targeted communications to impacted customers sent to Azure Service Health.
17:26 UTC on 29 October 2025 – Azure portal failed away from Azure Front Door.
17:30 UTC on 29 October 2025 – We blocked all new customer configuration changes to prevent further impact.
17:40 UTC on 29 October 2025 – We initiated the deployment of our ‘last known good’ configuration.
18:30 UTC on 29 October 2025 – We started to push the fixed configuration globally.
18:45 UTC on 29 October 2025 – Manual recovery of nodes commenced while gradual routing of traffic to healthy nodes began after the fixed configuration was pushed globally.
23:15 UTC on 29 October 2025 - PowerApps mitigation of dependency, and customers confirm mitigation.
00:05 UTC on 30 October 2025 – AFD impact confirmed mitigated for customers.
- onionisafruit 1y agoAt 16:04 “Investigation commenced”. Then at 16:15 “We began the investigation”. Which is it?
- not_a_bot_4sho 1y agoI read it as the second investigation being specific to AFD. The first more general.
- onionisafruit 1y agoI think you’re right. I missed that subtlety on first reading.
- ssss11 1y agoQuick coffee run before we get stuck in mate
- ozim 1y agoLoad some carbs with chocolate chip cookies as well, that’s what I would do. You don’t want to debug stuff with low sugar.
- normie3000 1y agoOne crash after another
- red-iron-pine 1y agoburn a smoko and take a leak
- taco_emoji 1y ago16:04 Started running around screaming 16:15 Sat down & looked at logs
- xnorswap 1y ago33 minutes from impact to status page for a complete outage is a joke.
- neya 1y agoIn Microsoft's defense, Azure has always been a complete joke. It's extremely developer unfriendly, buggy and overpriced.
- michaelt 1y agoIf you call that defending microsoft, I'd hate to see what attacking them looks like :)
- dude250711 1y agoSave it for when they stick Copilot into Azure portal.
- antonvdi 1y agoThey're already doing that.
- alias_neo 1y agoHa, you haven't used it recently have you? Copilot is already there, and it can't do a single useful thing. Me: "How do I connect [X] to [Y] using [Z]?" Copilot: "Please select the AKS cluster you'd like to delete"
- GuestFAUniverse 1y agoPerfect answer /s
- nflekkhnnn 1y agoActually one of the inventors of k8s was the project lead for copilot in the azure portal, and deployed it over a year ago.
- oofbey 1y ago“Our protection mechanisms, to validate and block any erroneous deployments, failed due to a software defect which allowed the deployment to bypass safety validations.” Very circular way of saying “the validator didn’t do its job”. This is AFAICT a pretty fundamental root cause of the issue. It’s never good enough to have a validator check the content and hope that finds all the issues. Validators are great and can speed a lot of things up. But because they are independent code paths they will always miss something. For critical services you have to assume the validator will be wrong, and be prepared to contain the damage WHEN it is wrong.
- notorandit 1y agoWhat puzzles me too is the time it took to recognize an outage. Looks like there was no monitoring and no alerts. Which is kinda weird.
- hinkley 1y agoI've seen sensitivity get tuned down to avoid false positives during deployments or rolling restarts for host updates. And to a lesser extent for autoscaling noise. It can be hard to get right. I think it's perhaps a gap in the tools. We apply the same alert criteria at 2 am that we do while someone is actively running deployment or admin tasks and there's a subset that should stay the same, like request failure rate, and others that should be tuned down, like overall error rate and median response times. And it means one thing if the failure rate for one machine is 90% and something else if the cluster failure rate is 5%, but if you've only got 18 boxes it's hard to discern the difference. And which is the higher priority error may change from one project to another.
- deadbolt 1y agoJust what you want in a cloud provider, right?
- neop1x 1y ago>> We began the investigation and started to examine configuration changes within AFD. Troubleshooting has completed Troubleshooting was unable to automatically fix all of the issues found. You can find more details below. >> We initiated the deployment of our ‘last known good’ configuration. System Restore can help fix problems that might be making your computer run slowly or stop responding. System Restore does not affect any of your documents, pictures, or other personal data. Recently installed programs and drivers might be uninstalled. Confirm your restore point Your computer will be restored to the state it was in before the event in the Description field below.