12 ms·

Final Root Cause Analysis of Nov 18 Azure Service Interruption

198 points by asyncwords 12y ago

coldcode 12y ago
Shit happens and at this scale it happens big. I wish everyone would provide details like this when the fan gets hit or your security fails. I'm glad I never have to deal with scale like this, it's pretty scary.
- jmartinpetersen 12y ago
  I've seen several companies where analysis like this would be for management only. I guess it's just human nature to want to sweep mistakes and accidents under the rug, but it does also speak volumes about the culture in such companies. Kudos to Microsoft and every other big player that communicates these things.
  - Someone1234 12y ago
    It reminds me of the NTSB's crash investigations. Instead of looking for a scapegoat or someone to blame, they look for the cause, and then look even deeper to find the root cause. For example they discover a pilot made a mistake. But they don't end it there, they then look at the airline's training materials, see if other pilots would repeat the same mistakes, and so on until they reach a point where they have a "this won't happen again" resolution (rather than simply discovering what happened). I feel like with Microsoft's breakdown they did the "this is what happened" post-mortem but then went to the next level and said "here's why this happened, and here is why it won't happen again."
    - epochwolf 12y ago
      Nitpick: Crash investigations are done by the NTSB, not the FAA. The NTSB has no authority to enforce its recommendations. That's up to the FAA. The idea behind that is the NTSB is more likely to be impartial.
      - Someone1234 12y ago
        Valid correction. I've edited it in. But it did originally say FAA, not NTSB.
  - smackfu 12y ago
    >I've seen several companies where analysis like this would be for management only I've found it to be pretty standard in the hosting world. I assume because if you have unexplained outages, customers leave.
    - Alupis 12y ago
      especially when it's an outage for 5-11 hours (depending on the customer) as this one was.
- cortesoft 12y ago
  Eh, its fun. After pushing to 15k servers, everything else seems unimportant!
mrb 12y ago
"These Virtual Machines were recreated by repeating the VM provisioning step. Linux Virtual Machines were not affected." So Azure supports Linux VMs?! Microsoft does so little Azure advertising that I had to learn this fact from their RCA. Apparently they do support it since 2012: http://www.techrepublic.com/blog/linux-and-open-source/microsoft-now-offering-linux-on-azure-what-does-this-mean/ http://www.techrepublic.com/blog/linux-and-open-source/micro... but it is likely that many non-users of Azure do not know this.
- shanselman 12y ago
  In fact, something like 20% of Azure is running Linux VMs. Works great, you can use Chef, Puppet, Vagrant, and the open-source cross platform CLI to manage them. There's a thousand Linux VMs to choose from here https://vmdepot.msopentech.com/List/Index https://vmdepot.msopentech.com/List/Index
  - Alupis 12y ago
    There's also the "Microsoft Loves Linux" campaign going on[1]. It's part of their new attempt to embrace the FOSS world and generally be more "open". [1] https://twitter.com/jniccolai/status/524281997632745472 https://twitter.com/jniccolai/status/524281997632745472
- mynameisvlad 12y ago
  It's on the homepage (http://azure.com http://azure.com) as "Launch Windows Server and Linux in minutes". Plus it has an entire section in their VMs page (http://azure.microsoft.com/en-us/services/virtual-machines/ http://azure.microsoft.com/en-us/services/virtual-machines/). Although yes, their advertising of Azure and Azure features isn't very huge. The ads that I have seen are for "Microsoft Cloud" (http://www.microsoft.com/enterprise/microsoftcloud/default.aspx#fbid=4RaZt1Pt9FK http://www.microsoft.com/enterprise/microsoftcloud/default.a...) which is a combination of products and technologies.
- YorkianTones 12y ago
  Yup, you can get up and running with a Linux VM in seconds. I've found Azure to be cheaper than AWS for micro VM instances. Running a twitter bot on AWS cost me ~ $10/month but after switching over to a CentOS VM on Azure I'm paying about $6/month. I also have dedicated Ubuntu and Windows Server VMs for personal projects and hosting.
pfortuny 12y ago
Impressive non-jargonized report. I would have "quantified" the "small number" but kudos anyway to Microsoft for taking this path towards transparency.
nchelluri 12y ago
I'm pretty impressed with the openness of this statement.
- sybhn 12y ago
  idem
sandis 12y ago
> The engineer fixing the Azure Table storage performance issue believed that because the change had already been flighted on a portion of the production infrastructure for several weeks, enabling this across the infrastructure was low risk. Ugh, I wouldn't want to be that guy (even if there would be no direct repercussions). That said, and as others have highlighted - kudos on the writeup and openness.
- cheez 12y ago
  Forever known as the guy who knocked Azure offline.
  - click170 12y ago
    Probably less than you think. At least among his peers. These kinds of "almost took X offline" happen All The Time, its just that most of the time they get caught before it gets too far. Its inevitable that a few will squeak through the nets. Mistakes can and will happen anywhere we allow them to. If you want to prevent mistakes, write tools to help reduce the "attack surface" (areas where mistakes can be made). Eg Don't want someone to be able to do "sudo reboot" accidentally? Alias reboot to something else. It won't stop hackers but it might help fight fat fingers.
  - epochwolf 12y ago
    I've halted production in 14 manufacturing plants before. Ran a query against the wrong server and locked up the entire ERP system for 30 minutes. Accidents happen.
- angersock 12y ago
  It's kind of shitty--it kinda seems like it had passed initial testing, and then got rushed out to full-scale production because it also fixed other customer issues (as mentioned in the writeup). That's the thing about these kinds of bugs...they are, by definition, tricky enough to have passed testing unseen.
- jschmitz28 12y ago
  The sentence you quoted seems ambiguous to me, since (from what I understood reading the article) there are two separate storage mechanisms using the new feature. The two possible beliefs the engineer may have had are: 1. Since we tested the change on a subset of A for a few weeks, we can assume it will work for all of A. 2. Since we tested the change on a subset of A for a few weeks, we can assume it will work for all of A and all of B. #1 seems reasonable, but #2 is what needed to hold true in order for there to be no problems, since the change was actually enabled for all of A and B. But was the engineer actually advocating to enable the change in B, or was that an accident during the manual deployment?
sybhn 12y ago
TL;DR >In summary, Microsoft Azure had clear operating guidelines but there was a gap in the deployment tooling that relied on human decisions and protocol.
larrystrange 12y ago
OSS is alive and well on the Azure Platform www.microsoft.com/openness
ChuckMcM 12y ago
Nice writeup. I hope that the engineer in question didn't get fired or anything. One of the challenges in SRE/Ops type organizations is to be responsible, take ownership, put in the extra time to fix things you break, but keep the nerve to push out changes. Once an ops team loses its willingness to push large changes, the infrastructure calcifies and you have a much bigger problem on your hand.
- qeorge 12y ago
  I agree, and am reminded of the quote from Thomas Watson (IBM) on this: “Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?”
  - tedunangst 12y ago
    Almost certainly apocryphal. There are multiple versions of the quote floating around. Also: Who was Watson talking to? How did this story get from him to us?
    - michaelcampbell 12y ago
      I heard the exact same story in a Wall St. trading firm I worked for - except the story was about a trade gone wrong (trades X shares instead of $X worth of shares, causing the Dow to slide and all sorts of havok).
    - smackfu 12y ago
      It just sounds like an anecdote someone would use during a talk.
    - CurtMonash 12y ago
      There's a pretty credible one about AOL's board not firing Steve Case after spending $5 million on him. (For those who have forgotten, Steve didn't actually found AOL; he was brought in from the outside.)
- 12y ago
runT1ME 12y ago
Does anyone else see the missing piece to this post mortem? An infinite loop made its way onto a majority(? all?) of production servers, and the immediate response is more or less 'we shouldn't have deployed to as many customers, failure should have only happened to a small subset'? I agree that improvements made to their deployment tooling were good and necessary, take the human temptation to skip steps out of the equation. But this exemplifies a major problem our industry suffers from, in that it just taken as a given that critical errors will sometimes make their way into production servers and the best we can do is reduce the impact. I find this absolutely unacceptable. How about we short circuit the process and identify ways to stop that from happening? Were there enough code reviews? Did automated testing fail here? Yes I'm familiar with the halting problem and limitations of formal verification on turing complete languages, but I don't believe it's an excuse. This is tantamount to saying "yeah sometimes our airplanes crash, so from now on we'll just make sure we have less passengers ride in the newer models".
- daeken 12y ago
  > This is tantamount to saying "yeah sometimes our airplanes crash, so from now on we'll just make sure we have less passengers ride in the newer models". Which would be a pretty reasonable thing to say if you had a large portion of the population on a single plane. What this all comes down to is: problems (small or big; stupidly simple or ridiculously complex) will happen. Isolating problems to the smallest number of people possible is the responsible course of action. That, of course, isn't mutually exclusive with doing better from a software engineering front.
- wvenable 12y ago
  It's not a missing piece, it's in the release: "Azure Table storage Front-Ends, the configuration switch was incorrectly enabled for Azure Blob storage Front-Ends." Not only was it deployed to everyone but it was deployed, untested, to the wrong place. Otherwise deploy would have been just as broad but successful.
- Someone1234 12y ago
  > An infinite loop made its way onto a majority(? all?) of production servers, and the immediate response is more or less 'we shouldn't have deployed to as many customers, failure should have only happened to a small subset'? All server software has one or more "infinite loops." It is a fundamental object in all listeners. Plus when they say infinite loop, I assumed they meant it continuously entered a crash/restart cycle rather than a while(true) {} in a line of code. I think the reality on the ground is that bug-free software is a myth. All you can do is have processes (like gradual deployment) to mitigate the damage it can do, rather than making it your goal to write the mythical perfect code. > But this exemplifies a major problem our industry suffers from, in that it just taken as a given that critical errors will sometimes make their way into production servers and the best we can do is reduce the impact. I find this absolutely unacceptable It is a major problem. It costs billions every year. But what can be done? If there was a magic wand solution I'm sure people would be scrambling to deploy it as it saves them money. > How about we short circuit the process and identify ways to stop that from happening? Were there enough code reviews? Did automated testing fail here? This seems like a somewhat naive view of software development in general. Like what I'd call a "mathematician's view," in the sense that they think large complex systems can be reduced to a simple quantifiable process. Code reviews and more importantly unit tests can help find bugs. But inter-connectivity between large complex systems is harder to test again, and harder to code review (because the bugs don't exist on any single line of code, or in any single block even).
forgotAgain 12y ago
I hope the engineer in question did not get fired. I also hope that no one who recommended Azure to their employer got fired either.
jabanico 12y ago
"Unfortunately, the configuration tooling did not have adequate enforcement of this policy of incrementally deploying the change across the infrastructure." They relied on tooling to do the review of the last step of the process? I would have thought there were a few layers of approval that goes along with that final push into mission critical infrastructure.
- NeutronBoy 12y ago
  That's exactly what they mean - the workflow tool they used didn't enforce approvals from all the concerned parties.
ha292 12y ago
This is a good effort. I do have some concerns about it. A true root cause would go deeper and ask why is it that an engineer could solely decide to roll out to all slices ? The surface-level answer is that Azure platform lacked tooling. Is that the cause or an effect ? I think it is an effect. There are deeper root causes. Let's ask -- why was it that the design allowed one engineer to effectively bring down Azure ? We often stop at these RCAs when it gets uncomfortable and it starts to point upwards. I say this to the engineer who pressed the buttons: Bravo! You did something that exposed a massive hole in Azure which may have very well prevented a much bigger embarrassment.
- Lewisham 12y ago
  A true root cause would go deeper and ask why is it that an engineer could solely decide to roll out to all slices? Because writing code which contains a large number of checks and balances is generally orders of magnitude more expensive than human trust/judgment on the Ops team. Reading the postmortem makes me think that this sort of failure could have happened to anyone, and no-one really did anything wrong. The mistake was the blob store config flag not getting flipped, which is just a natural human error. The engineer who did the roll out could have been any of us. Given what he/she knew, he/she thought they had a good soak test (and a couple of weeks is a pretty good soak test) and made a call, similar calls he/she makes a number of times every day. This one didn't pan out. I would hazard that most companies have a big red rollout button that is reserved for trusted engineers that will do a rollout without all the checks you're requesting.
  - smackfu 12y ago
    Just a second level of approval can be very useful, without requiring orders of magnitude costs. In part because it usually requires that the change be explained in writing to the second approver, and that can often reveal issues.
    - Lewisham 12y ago
      It's not clear he/she didn't notify a secondary person, who would have likely had the same knowledge he/she did. Given the same knowledge, the same push might well have happened.
tytso 12y ago
The really big missing piece that I found in this post mortem is if it only took 30 minutes to revert the original change, why did it take over ten hours to restart the Azure Blob storage servers? This was neatly elided in the last sentence of this paragraph of their writeup: ".... We reverted the change globally within 30 minutes of the start of the issue which protected many Azure Blob storage Front-Ends from experiencing the issue. The Azure Blob storage Front-Ends which already entered the infinite loop were unable to accept any configuration changes due to the infinite loop. These required a restart after reverting the configuration change, extending the time to recover." The ten+ hours extension was the vast majority of the outage time; why wasn't the reason for this given? More importantly, what will be done to prevent a similar extension in the time Azure spends belly up if at some point in the future, the Blob servers go insane and have to be restarted?
- nemothekid 12y ago
  Only a guess but from how its worded it seems that the storage frontend that had already entered infinite loops may have taken the tem+ hours to restart.
  - CoreySanders 12y ago
    Mark, the Azure CTO, gives a good breakdown of the time taken for each portion of the incident recovery in this video: http://channel9.msdn.com/posts/Inside-the-Azure-Storage-Outage-of-November-18th http://channel9.msdn.com/posts/Inside-the-Azure-Storage-Outa... That may help address these questions. Just FYI, I am an engineer in the Azure compute team.
deleted 12y ago
[deleted]
mooneater 12y ago
Pros: -they are sharing info -they allowed some caustic comments to remain at the bottom of the page (so far). Cons: -This is almost 30 days after the incident -Look at the regions, it was global! -This was a whole chain of issues. I count it as 5 separate issues. This goes deep into how they operate and it does not paint a picture of operational maturity: 1: configuration change for the Blob Front-Ends exposed a bug in the Blob Front-Ends 2: Blob Front-Ends infinite loop delayed the fix (I count this as a separate issue though I expect some may not) 3: As part of a plan to improve performance of the Azure Storage Service, the decision was made to push the configuration change to the entire production service 4: Update was made across most regions in a short period of time due to operational error 5: Azure infrastructure issue that impacted our ability to provide timely updates via the Service Health Dashboard That is quite a list. [Edit : formatting only]
- markveronda 12y ago
  > -This is almost 30 days after the incident What would have been the optimal response time? They fixed the immediate problem as fast as they could and gave a preliminary RCA, then they did a longer-term RCA and fix. I feel this shows maturity by not rushing to immediate conclusions and trying to do a 5-Whys drill-down to fix the underlying cause. Furthermore, they also took steps to actually fix the problem by pointing out they moved the human out of the loop in one aspect and that's always a good thing (unless the replacement software is faulty itself of course). Also, in response to the list, I believe [3&4] are actually the same thing, are they not? The operator was the one who made the 'decision' by accidentally ignoring the incremental config change policy that was in place and did it all at once. This was identified as a human error and they fixed it by enforcing incremental changes.
- keypusher 12y ago
  They actually posted a blog the day after it happened with their initial triage and updates. https://news.ycombinator.com/item?id=8633633 https://news.ycombinator.com/item?id=8633633 I agree with you though, and said so at the time. These issues seem systemic, not isolated. Cascading failures, some in code, some in ops procedures, indicates to me they still have work to do.
teyc 12y ago
If utility computing is to is be taken seriously, then it has to institute the same kind of discipline that we see occurring in the airline industry. Recent examples come to mind: pilot letting songstress hold the wheels and wearing the pilots cap - fired. Airline executive overruling pilot over macadamia nuts - million dollar fine. If we wish for a future where cloud computing will be considered reliable enough for air traffic control systems, then management of these infrastructure requires a level of dedication and commitment to process and training. Failover zones need to be isolated not only physically, but also from command and control. A lone engineer should not have sufficient authority or capability to operationally control more than one zone. It is extremely unnerving for enterprises to see that a significant infrastructure like Azure has a root account which can take down the whole of Azure.
spudlyo 12y ago
After analysis was complete, we released an update to our deployment system tooling to enforce compliance to the above testing and flighting policies for standard updates, whether code or configuration. Hopefully there is a way to disable this policy adherence for when you really need to push out a configuration or code change everywhere quickly.
markveronda 12y ago
I cannot believe how many times I have seen a PROD (or new env X) deployment go bad from configuration issues. At least they separate configuration deployments from code deployments, that's a good sign. Why not take it a step further and instead of doing config deployments, use a config server?
- internetisthesh 12y ago
  In the end, you would still have config deployments, but to the config server. And if you can push config to nodes needing it. you have one less point-of-failure, right? I'm not too familiar with the concept of a config server.
Redsquare 12y ago
Why are they so quiet about SLA credit? Not a word for a month and for a year I have been wasting good money on doubling up services to be inside the SLA + also deploying cross region to ensure zero downtime, what a joke. Surely Azure are not hoping we will forget?
billarmstrong 12y ago
Only one question: Will the engineer be fired?
- billarmstrong 12y ago
  I want to know the reality. I don't want to see another PR show. Dear Information diggers, please let us know whether the guy was fired!

198 points by asyncwords 12y ago

Shit happens and at this scale it happens big. I wish everyone would provide details like this when the fan gets hit or your security fails. I'm glad I never have to deal with scale like this, it's pretty scary.

jmartinpetersen 12y ago
I've seen several companies where analysis like this would be for management only. I guess it's just human nature to want to sweep mistakes and accidents under the rug, but it does also speak volumes about the culture in such companies. Kudos to Microsoft and every other big player that communicates these things.
- Someone1234 12y ago
  It reminds me of the NTSB's crash investigations. Instead of looking for a scapegoat or someone to blame, they look for the cause, and then look even deeper to find the root cause. For example they discover a pilot made a mistake. But they don't end it there, they then look at the airline's training materials, see if other pilots would repeat the same mistakes, and so on until they reach a point where they have a "this won't happen again" resolution (rather than simply discovering what happened). I feel like with Microsoft's breakdown they did the "this is what happened" post-mortem but then went to the next level and said "here's why this happened, and here is why it won't happen again."
  - epochwolf 12y ago
    Nitpick: Crash investigations are done by the NTSB, not the FAA. The NTSB has no authority to enforce its recommendations. That's up to the FAA. The idea behind that is the NTSB is more likely to be impartial.
    - Someone1234 12y ago
      Valid correction. I've edited it in. But it did originally say FAA, not NTSB.
- smackfu 12y ago
  >I've seen several companies where analysis like this would be for management only I've found it to be pretty standard in the hosting world. I assume because if you have unexplained outages, customers leave.
  - Alupis 12y ago
    especially when it's an outage for 5-11 hours (depending on the customer) as this one was.
cortesoft 12y ago
Eh, its fun. After pushing to 15k servers, everything else seems unimportant!

mrb 12y ago

"These Virtual Machines were recreated by repeating the VM provisioning step. Linux Virtual Machines were not affected." So Azure supports Linux VMs?! Microsoft does so little Azure advertising that I had to learn this fact from their RCA. Apparently they do support it since 2012: http://www.techrepublic.com/blog/linux-and-open-source/microsoft-now-offering-linux-on-azure-what-does-this-mean/ http://www.techrepublic.com/blog/linux-and-open-source/micro... but it is likely that many non-users of Azure do not know this.

shanselman 12y ago
In fact, something like 20% of Azure is running Linux VMs. Works great, you can use Chef, Puppet, Vagrant, and the open-source cross platform CLI to manage them. There's a thousand Linux VMs to choose from here https://vmdepot.msopentech.com/List/Index https://vmdepot.msopentech.com/List/Index
- Alupis 12y ago
  There's also the "Microsoft Loves Linux" campaign going on[1]. It's part of their new attempt to embrace the FOSS world and generally be more "open". [1] https://twitter.com/jniccolai/status/524281997632745472 https://twitter.com/jniccolai/status/524281997632745472
mynameisvlad 12y ago
It's on the homepage (http://azure.com http://azure.com) as "Launch Windows Server and Linux in minutes". Plus it has an entire section in their VMs page (http://azure.microsoft.com/en-us/services/virtual-machines/ http://azure.microsoft.com/en-us/services/virtual-machines/). Although yes, their advertising of Azure and Azure features isn't very huge. The ads that I have seen are for "Microsoft Cloud" (http://www.microsoft.com/enterprise/microsoftcloud/default.aspx#fbid=4RaZt1Pt9FK http://www.microsoft.com/enterprise/microsoftcloud/default.a...) which is a combination of products and technologies.
YorkianTones 12y ago
Yup, you can get up and running with a Linux VM in seconds. I've found Azure to be cheaper than AWS for micro VM instances. Running a twitter bot on AWS cost me ~ $10/month but after switching over to a CentOS VM on Azure I'm paying about $6/month. I also have dedicated Ubuntu and Windows Server VMs for personal projects and hosting.

pfortuny 12y ago

Impressive non-jargonized report. I would have "quantified" the "small number" but kudos anyway to Microsoft for taking this path towards transparency.

nchelluri 12y ago

I'm pretty impressed with the openness of this statement.

sybhn 12y ago
idem

sandis 12y ago

> The engineer fixing the Azure Table storage performance issue believed that because the change had already been flighted on a portion of the production infrastructure for several weeks, enabling this across the infrastructure was low risk. Ugh, I wouldn't want to be that guy (even if there would be no direct repercussions). That said, and as others have highlighted - kudos on the writeup and openness.

cheez 12y ago
Forever known as the guy who knocked Azure offline.
- click170 12y ago
  Probably less than you think. At least among his peers. These kinds of "almost took X offline" happen All The Time, its just that most of the time they get caught before it gets too far. Its inevitable that a few will squeak through the nets. Mistakes can and will happen anywhere we allow them to. If you want to prevent mistakes, write tools to help reduce the "attack surface" (areas where mistakes can be made). Eg Don't want someone to be able to do "sudo reboot" accidentally? Alias reboot to something else. It won't stop hackers but it might help fight fat fingers.
- epochwolf 12y ago
  I've halted production in 14 manufacturing plants before. Ran a query against the wrong server and locked up the entire ERP system for 30 minutes. Accidents happen.
angersock 12y ago
It's kind of shitty--it kinda seems like it had passed initial testing, and then got rushed out to full-scale production because it also fixed other customer issues (as mentioned in the writeup). That's the thing about these kinds of bugs...they are, by definition, tricky enough to have passed testing unseen.
jschmitz28 12y ago
The sentence you quoted seems ambiguous to me, since (from what I understood reading the article) there are two separate storage mechanisms using the new feature. The two possible beliefs the engineer may have had are: 1. Since we tested the change on a subset of A for a few weeks, we can assume it will work for all of A. 2. Since we tested the change on a subset of A for a few weeks, we can assume it will work for all of A and all of B. #1 seems reasonable, but #2 is what needed to hold true in order for there to be no problems, since the change was actually enabled for all of A and B. But was the engineer actually advocating to enable the change in B, or was that an accident during the manual deployment?

sybhn 12y ago

TL;DR >In summary, Microsoft Azure had clear operating guidelines but there was a gap in the deployment tooling that relied on human decisions and protocol.

larrystrange 12y ago

OSS is alive and well on the Azure Platform www.microsoft.com/openness

ChuckMcM 12y ago

Nice writeup. I hope that the engineer in question didn't get fired or anything. One of the challenges in SRE/Ops type organizations is to be responsible, take ownership, put in the extra time to fix things you break, but keep the nerve to push out changes. Once an ops team loses its willingness to push large changes, the infrastructure calcifies and you have a much bigger problem on your hand.

qeorge 12y ago
I agree, and am reminded of the quote from Thomas Watson (IBM) on this: “Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?”
- tedunangst 12y ago
  Almost certainly apocryphal. There are multiple versions of the quote floating around. Also: Who was Watson talking to? How did this story get from him to us?
  - michaelcampbell 12y ago
    I heard the exact same story in a Wall St. trading firm I worked for - except the story was about a trade gone wrong (trades X shares instead of $X worth of shares, causing the Dow to slide and all sorts of havok).
  - smackfu 12y ago
    It just sounds like an anecdote someone would use during a talk.
  - CurtMonash 12y ago
    There's a pretty credible one about AOL's board not firing Steve Case after spending $5 million on him. (For those who have forgotten, Steve didn't actually found AOL; he was brought in from the outside.)
12y ago

runT1ME 12y ago

Does anyone else see the missing piece to this post mortem? An infinite loop made its way onto a majority(? all?) of production servers, and the immediate response is more or less 'we shouldn't have deployed to as many customers, failure should have only happened to a small subset'? I agree that improvements made to their deployment tooling were good and necessary, take the human temptation to skip steps out of the equation. But this exemplifies a major problem our industry suffers from, in that it just taken as a given that critical errors will sometimes make their way into production servers and the best we can do is reduce the impact. I find this absolutely unacceptable. How about we short circuit the process and identify ways to stop that from happening? Were there enough code reviews? Did automated testing fail here? Yes I'm familiar with the halting problem and limitations of formal verification on turing complete languages, but I don't believe it's an excuse. This is tantamount to saying "yeah sometimes our airplanes crash, so from now on we'll just make sure we have less passengers ride in the newer models".

daeken 12y ago
> This is tantamount to saying "yeah sometimes our airplanes crash, so from now on we'll just make sure we have less passengers ride in the newer models". Which would be a pretty reasonable thing to say if you had a large portion of the population on a single plane. What this all comes down to is: problems (small or big; stupidly simple or ridiculously complex) will happen. Isolating problems to the smallest number of people possible is the responsible course of action. That, of course, isn't mutually exclusive with doing better from a software engineering front.
wvenable 12y ago
It's not a missing piece, it's in the release: "Azure Table storage Front-Ends, the configuration switch was incorrectly enabled for Azure Blob storage Front-Ends." Not only was it deployed to everyone but it was deployed, untested, to the wrong place. Otherwise deploy would have been just as broad but successful.
Someone1234 12y ago
> An infinite loop made its way onto a majority(? all?) of production servers, and the immediate response is more or less 'we shouldn't have deployed to as many customers, failure should have only happened to a small subset'? All server software has one or more "infinite loops." It is a fundamental object in all listeners. Plus when they say infinite loop, I assumed they meant it continuously entered a crash/restart cycle rather than a while(true) {} in a line of code. I think the reality on the ground is that bug-free software is a myth. All you can do is have processes (like gradual deployment) to mitigate the damage it can do, rather than making it your goal to write the mythical perfect code. > But this exemplifies a major problem our industry suffers from, in that it just taken as a given that critical errors will sometimes make their way into production servers and the best we can do is reduce the impact. I find this absolutely unacceptable It is a major problem. It costs billions every year. But what can be done? If there was a magic wand solution I'm sure people would be scrambling to deploy it as it saves them money. > How about we short circuit the process and identify ways to stop that from happening? Were there enough code reviews? Did automated testing fail here? This seems like a somewhat naive view of software development in general. Like what I'd call a "mathematician's view," in the sense that they think large complex systems can be reduced to a simple quantifiable process. Code reviews and more importantly unit tests can help find bugs. But inter-connectivity between large complex systems is harder to test again, and harder to code review (because the bugs don't exist on any single line of code, or in any single block even).

forgotAgain 12y ago

I hope the engineer in question did not get fired. I also hope that no one who recommended Azure to their employer got fired either.

jabanico 12y ago

"Unfortunately, the configuration tooling did not have adequate enforcement of this policy of incrementally deploying the change across the infrastructure." They relied on tooling to do the review of the last step of the process? I would have thought there were a few layers of approval that goes along with that final push into mission critical infrastructure.

NeutronBoy 12y ago
That's exactly what they mean - the workflow tool they used didn't enforce approvals from all the concerned parties.

ha292 12y ago

This is a good effort. I do have some concerns about it. A true root cause would go deeper and ask why is it that an engineer could solely decide to roll out to all slices ? The surface-level answer is that Azure platform lacked tooling. Is that the cause or an effect ? I think it is an effect. There are deeper root causes. Let's ask -- why was it that the design allowed one engineer to effectively bring down Azure ? We often stop at these RCAs when it gets uncomfortable and it starts to point upwards. I say this to the engineer who pressed the buttons: Bravo! You did something that exposed a massive hole in Azure which may have very well prevented a much bigger embarrassment.

Lewisham 12y ago
A true root cause would go deeper and ask why is it that an engineer could solely decide to roll out to all slices? Because writing code which contains a large number of checks and balances is generally orders of magnitude more expensive than human trust/judgment on the Ops team. Reading the postmortem makes me think that this sort of failure could have happened to anyone, and no-one really did anything wrong. The mistake was the blob store config flag not getting flipped, which is just a natural human error. The engineer who did the roll out could have been any of us. Given what he/she knew, he/she thought they had a good soak test (and a couple of weeks is a pretty good soak test) and made a call, similar calls he/she makes a number of times every day. This one didn't pan out. I would hazard that most companies have a big red rollout button that is reserved for trusted engineers that will do a rollout without all the checks you're requesting.
- smackfu 12y ago
  Just a second level of approval can be very useful, without requiring orders of magnitude costs. In part because it usually requires that the change be explained in writing to the second approver, and that can often reveal issues.
  - Lewisham 12y ago
    It's not clear he/she didn't notify a secondary person, who would have likely had the same knowledge he/she did. Given the same knowledge, the same push might well have happened.

tytso 12y ago

The really big missing piece that I found in this post mortem is if it only took 30 minutes to revert the original change, why did it take over ten hours to restart the Azure Blob storage servers? This was neatly elided in the last sentence of this paragraph of their writeup: ".... We reverted the change globally within 30 minutes of the start of the issue which protected many Azure Blob storage Front-Ends from experiencing the issue. The Azure Blob storage Front-Ends which already entered the infinite loop were unable to accept any configuration changes due to the infinite loop. These required a restart after reverting the configuration change, extending the time to recover." The ten+ hours extension was the vast majority of the outage time; why wasn't the reason for this given? More importantly, what will be done to prevent a similar extension in the time Azure spends belly up if at some point in the future, the Blob servers go insane and have to be restarted?

nemothekid 12y ago
Only a guess but from how its worded it seems that the storage frontend that had already entered infinite loops may have taken the tem+ hours to restart.
- CoreySanders 12y ago
  Mark, the Azure CTO, gives a good breakdown of the time taken for each portion of the incident recovery in this video: http://channel9.msdn.com/posts/Inside-the-Azure-Storage-Outage-of-November-18th http://channel9.msdn.com/posts/Inside-the-Azure-Storage-Outa... That may help address these questions. Just FYI, I am an engineer in the Azure compute team.

deleted 12y ago

[deleted]

mooneater 12y ago

Pros: -they are sharing info -they allowed some caustic comments to remain at the bottom of the page (so far). Cons: -This is almost 30 days after the incident -Look at the regions, it was global! -This was a whole chain of issues. I count it as 5 separate issues. This goes deep into how they operate and it does not paint a picture of operational maturity: 1: configuration change for the Blob Front-Ends exposed a bug in the Blob Front-Ends 2: Blob Front-Ends infinite loop delayed the fix (I count this as a separate issue though I expect some may not) 3: As part of a plan to improve performance of the Azure Storage Service, the decision was made to push the configuration change to the entire production service 4: Update was made across most regions in a short period of time due to operational error 5: Azure infrastructure issue that impacted our ability to provide timely updates via the Service Health Dashboard That is quite a list. [Edit : formatting only]

markveronda 12y ago
> -This is almost 30 days after the incident What would have been the optimal response time? They fixed the immediate problem as fast as they could and gave a preliminary RCA, then they did a longer-term RCA and fix. I feel this shows maturity by not rushing to immediate conclusions and trying to do a 5-Whys drill-down to fix the underlying cause. Furthermore, they also took steps to actually fix the problem by pointing out they moved the human out of the loop in one aspect and that's always a good thing (unless the replacement software is faulty itself of course). Also, in response to the list, I believe [3&4] are actually the same thing, are they not? The operator was the one who made the 'decision' by accidentally ignoring the incremental config change policy that was in place and did it all at once. This was identified as a human error and they fixed it by enforcing incremental changes.
keypusher 12y ago
They actually posted a blog the day after it happened with their initial triage and updates. https://news.ycombinator.com/item?id=8633633 https://news.ycombinator.com/item?id=8633633 I agree with you though, and said so at the time. These issues seem systemic, not isolated. Cascading failures, some in code, some in ops procedures, indicates to me they still have work to do.

teyc 12y ago

If utility computing is to is be taken seriously, then it has to institute the same kind of discipline that we see occurring in the airline industry. Recent examples come to mind: pilot letting songstress hold the wheels and wearing the pilots cap - fired. Airline executive overruling pilot over macadamia nuts - million dollar fine. If we wish for a future where cloud computing will be considered reliable enough for air traffic control systems, then management of these infrastructure requires a level of dedication and commitment to process and training. Failover zones need to be isolated not only physically, but also from command and control. A lone engineer should not have sufficient authority or capability to operationally control more than one zone. It is extremely unnerving for enterprises to see that a significant infrastructure like Azure has a root account which can take down the whole of Azure.

spudlyo 12y ago

After analysis was complete, we released an update to our deployment system tooling to enforce compliance to the above testing and flighting policies for standard updates, whether code or configuration. Hopefully there is a way to disable this policy adherence for when you really need to push out a configuration or code change everywhere quickly.

markveronda 12y ago

I cannot believe how many times I have seen a PROD (or new env X) deployment go bad from configuration issues. At least they separate configuration deployments from code deployments, that's a good sign. Why not take it a step further and instead of doing config deployments, use a config server?

internetisthesh 12y ago
In the end, you would still have config deployments, but to the config server. And if you can push config to nodes needing it. you have one less point-of-failure, right? I'm not too familiar with the concept of a config server.

Redsquare 12y ago

Why are they so quiet about SLA credit? Not a word for a month and for a year I have been wasting good money on doubling up services to be inside the SLA + also deploying cross region to ensure zero downtime, what a joke. Surely Azure are not hoping we will forget?

billarmstrong 12y ago

Only one question: Will the engineer be fired?

billarmstrong 12y ago
I want to know the reality. I don't want to see another PR show. Dear Information diggers, please let us know whether the guy was fired!