3 ms·

I wonder if Azure DevOps will have a day long outage like VSTS had last week? That had a legitimately productivity cost to a lot of companies, and the more vest

I wonder if Azure DevOps will have a day long outage like VSTS had last week? That had a legitimately productivity cost to a lot of companies, and the more vested in the VSTS toolchain you were the harder hit you were. I'm legitimately surprised they're launching this so soon after that, particularly considering there's been no real post-mortem, and no update on how they'll stop this happening again. "It is Azure's fault" isn't an answer, and re-branding this Azure DevOps makes that excuse almost hilarious. I'm well aware that Azure AD was down, and that VSTS has a strong dependency on it, but ultimately we're still talking about a single data center bringing down VSTS for all of North America for an entire work-day. It has been a week and now they're trying to encourage you to push your whole deployment toolchain (manage, build, test) into their cloud.

ethomson 8y ago
I certainly hope not. I'm not going to argue that this was anything but a disappointment to our customers - and one that we're eager to prevent from happening again. But we have never said that this was Azure's fault. It wasn't. We've put the blame where it belongs: on us. We're nearly done writing up the root cause analysis and an analysis of our next steps to prevent this from happening in the future.
- Someone1234 8y ago
  Do you happen to know where the root cause analysis will be posted if it is made public?
  - ethomson 8y ago
    It will be on our service availability blog: https://blogs.msdn.microsoft.com/vsoservice/ https://blogs.msdn.microsoft.com/vsoservice/, and I'll reply here with a direct link once it's online.
    - ethomson 8y ago
      Update: the Director of Engineering for Azure DevOps has posted this. https://blogs.msdn.microsoft.com/vsoservice/?p=17485 https://blogs.msdn.microsoft.com/vsoservice/?p=17485
- GordonS 8y ago
  3 weeks ago there was a different outage that meant no hosted build agents were working in the west europe region, and it lasted for more than 24 hours. Now, what I hated the most about this was the communication - there wasn't any. Why not email impacted users to let them know you're aware and working on a fix?
  - ethomson 8y ago
    You can subscribe to get emails for availability incidents on our service availability blog, https://blogs.msdn.microsoft.com/vsoservice/ https://blogs.msdn.microsoft.com/vsoservice/ (it's in the right-hand sidebar).
    - GordonS 8y ago
      Huh, I hadn't noticed that - I've subscribed now. I do still think it would be better if you proactively sent out emails only to affected customers though.

ethomson 8y ago

I certainly hope not. I'm not going to argue that this was anything but a disappointment to our customers - and one that we're eager to prevent from happening again. But we have never said that this was Azure's fault. It wasn't. We've put the blame where it belongs: on us. We're nearly done writing up the root cause analysis and an analysis of our next steps to prevent this from happening in the future.

Someone1234 8y ago
Do you happen to know where the root cause analysis will be posted if it is made public?
- ethomson 8y ago
  It will be on our service availability blog: https://blogs.msdn.microsoft.com/vsoservice/ https://blogs.msdn.microsoft.com/vsoservice/, and I'll reply here with a direct link once it's online.
  - ethomson 8y ago
    Update: the Director of Engineering for Azure DevOps has posted this. https://blogs.msdn.microsoft.com/vsoservice/?p=17485 https://blogs.msdn.microsoft.com/vsoservice/?p=17485
GordonS 8y ago
3 weeks ago there was a different outage that meant no hosted build agents were working in the west europe region, and it lasted for more than 24 hours. Now, what I hated the most about this was the communication - there wasn't any. Why not email impacted users to let them know you're aware and working on a fix?
- ethomson 8y ago
  You can subscribe to get emails for availability incidents on our service availability blog, https://blogs.msdn.microsoft.com/vsoservice/ https://blogs.msdn.microsoft.com/vsoservice/ (it's in the right-hand sidebar).
  - GordonS 8y ago
    Huh, I hadn't noticed that - I've subscribed now. I do still think it would be better if you proactively sent out emails only to affected customers though.