13 ms·

Postmortem for outage of us-east-1

83 points by akerl_ 12y ago

jffry 12y ago
It's great to see a postmortem with this level of detail, fairly quickly. It's also great to see Joyent hang the blame on the system that allowed rebooting every server, and the poor recovery from that failure, rather than continuing to throw the operator under the bus: "...we will be rethinking what tools are necessary over the coming days and weeks so that "full power" tools are not the only means by which to accomplish routine tasks."
- blantonl 12y ago
  More importantly, I hope that the person that issued this command still keeps his job. He learned an important lesson and unless there is sheer incompetence here, this individual will have a medal on his chest indicating he has been in the worst "combat" that a sysadmin can endure. Everyone makes mistakes, and judging by the language of the postmortem it appears it was just that. Would love to have that sysadmin on my team, because he will never do that again....
  - bcantrill 12y ago
    We debated whether or not to make this explicit in the postmortem, but yes, the operator in question still has their job, and for exactly the reasons that you outlined: it was an honest mistake, they were deeply apologetic (as one might imagine) -- and we know that they (certainly!) won't be making that mistake again. Mistakes like this are their own punishment; additional punitive action serves only to instill fear rather than effect the changes necessary to not repeat the failure.
    - dmourati 12y ago
      "Five why's" might be appropriate to suss out how the ultimate mistake was even possible.
  - patio11 12y ago
    It is said, perhaps apocryphally, that the head of the trading desk at Mizuho, when asked whether he fired the woman who physically keyed in an order for 600k shares at 1 yen rather than an order for 1 share at 600k yen ($300 million or so in losses), said "Why would I do that, after spending $300 million to make her the least likely person in Japan to typo trade instructions."
    - bagels 12y ago
      That quote is interesting because because the conclusion doesn't follow. It's basically the Monte Carlo fallacy. She's probably less likely than before due to fear or guilt, but I don't know that this makes her less error prone than every other person in the country, including those that double check every time, for instance.
- bobbles 12y ago
  >continuing to throw the operator under the bus There has been no evidence of them doing this at all.. I don't know why people keep saying it.
  - jffry 12y ago
    That was poorly phrased... In my mind, I meant it in a "sadly it's too often that the human making the mistake gets blamed, rather than the systems that didn't have appropriate safeguards to make said mistake non-trivial"
  - paulwolf 12y ago
    They threw an open-source contributor under the bus for rejecting a pull request that changed a code comment. Knee-jerk responses are definitely part of their modus operandi.
    - chris_wot 12y ago
      Agreed. They seem to be good at threatening to fire employees of other companies, but not their own. I suppose that's a net positive. Still.
JBiserkov 12y ago
>The command to reboot the select set of new systems that needed to be updated was mis-typed, and instead specified all servers in the datacenter. Substitute reboot with 'upgrade to win 7' and data center with university at get a story from a month? ago. rm ./*.* - delete all files in current dir rm /*.* - never do this Can you spot the difference? A colleague did the later on our testing server. No clients were disturbed, but us devs were left working on things that can run locally (much more pleasant stuff for sure, yet the schedule suffered a lot).
- adamfeldman 12y ago
  The incident was at Emory University: http://thenextweb.com/shareables/2014/05/16/emory-university-server-accidentally-sends-reformat-request-windows-pcs-including/ http://thenextweb.com/shareables/2014/05/16/emory-university...
- jacquesm 12y ago
  Your 'never do this' line would erase initrd.img on most machines, which you'd only find out about after the next reboot if nobody told you about it (and good luck getting that fixed). There are a large number of varieties of this particular error. Some with terrible results. rm -rf * .bak for instance (especially when executed in the root directory). That '#' prompt is there for a reason. The way to solve these sort of issues is to first get the files using 'find' until you're totally happy about the result and then to use 'rm' as the command passed to find. Of course, nobody does this ;)
  - jethro_tell 12y ago
    >Your 'never do this' line Would also erase everything else on the box so you might not know for a few seconds, but then the database errors start and the lib and pid files start disappearing and now your the king of a mountain of shit. It won't take till next reboot to notice. If you ever get a chance, rm -rf / a box before you throw it out and just play around with it for a couple minutes while it eats itself. You can spare your self most errors like that with tab complete, the built in sanity checker. I tab complete everything since I'm dyslexic, and it saves me a shit ton of time cause I'm never far from the error when I notice it.
incision 12y ago
>'In our experience, platforms with this network device will encounter this boot-time issue about 10% of the time. The mitigation for this is for an operator to simply initiate another reboot, which we performed on those afflicted nodes as soon as we identified them.' This bit bothers me more than anything else. It's not just a die roll configuration, but a known one with an operator required to do the re-rolling. Everything is a value judgement I guess, but knowingly leaving that one 'mitigated' would drive me insane.
- bcantrill 12y ago
  Sorry if we didn't adequately convey our frustration with this particular issue. It's one that's been with us for a long time and it absolutely sucks -- and after trying (and failing) to work with the vendor to get the issue understood (it's essentially a firmware-level issue), we ultimately decided to move away from that particular NIC vendor entirely. If we could wave a wand and be rid of these particular parts, we gladly would -- but until then, this transient boot-time issue needs to be manually mitigated with an additional reboot.
  - incision 12y ago
    Sure, I mean to relate more than judge - I've been through similar hoops though surely on a smaller scale.
  - AceJohnny2 12y ago
    Why the shyness about saying the brand's name? If their product and support was subpar to the point of blacklisting the entire vendor, it could be useful to spread the information to A) warn others of potential problems and B) put pressure on the vendor to improve their products.
    - bcantrill 12y ago
      I'm with you, but somehow a postmortem for our own outage seemed like the wrong place to name-and-shame a vendor...
adamfeldman 12y ago
I don't have a lot of experience with datacenters, and am trying to understand this sentence: "Because there was a simultaneous reboot of every system in the datacenter, there was extremely high contention on the TFTP boot infrastructure, which like all of our infrastructure, normally has throttles in place to ensure that it cannot run away with a machine." What does "cannot run away with a machine" mean? Why would you want to by default restrict the speed at which that system runs?
- jacquesm 12y ago
  I think they're trying to say that that machine has other duties except for TFTP and that the process is running at a priority low enough that it can be pre-empted by other processes. So if there is a lot of contention on the TFTP boot infrastructure the throttles limit the amount of work a single machine can do, which in turn can cause starvation. The reason why is probably related to what it does besides TFTP, which under normal circumstances is probably more important than new nodes joining the network. During a mass reboot that will bite you because then the chances of starvation will go up quite a bit as the throttles cause one machine after another to time-out and retry their boot sequence.
- ChargingWookie 12y ago
  They wanted to prevent all of the servers trying to contact the PXE server at the same time to avoid overloading it. Its basically to prevent an accidently self DDOS.
- CHY872 12y ago
  I also have no experience, but my guess is that when you are running normally, you want to share the bandwidth of a server fairly between all clients. When you've just accidentally rebooted all servers, you want to do it completely unfairly - so that instead of 10 computers taking an hour to download something, the first takes 6 minutes, the second takes 12 minutes etc. This allows at least some of the servers to get started and your infrastructure to start getting back online. I.e. you want to change from round robin scheduling to shortest job first scheduling (which gives the minimum average waiting time).
- deleted 12y ago
geetarista 12y ago
On behalf of all of Joyent, we are extremely sorry for this outage, and the severe inconvenience it may have caused to you, and your customers. I hate it when people apologize saying maybe it was inconvenient. If someone is using your service and you fucked up, it is an inconvenience.
- RussianCow 12y ago
  Doesn't mean everyone is affected by it. If I was hosting some unimportant website on their platform, I probably wouldn't even notice, and it wouldn't be an inconvenience to me. Silly thing to feel so strongly about.
- FireBeyond 12y ago
  Disagree. They acknowledge it may be severely inconvenient, not just an 'oops'.
bignaj 12y ago
>The command to reboot the select set of new systems that needed to be updated was mis-typed, and instead specified all servers in the datacenter. Unfortunately the tool in question does not have enough input validation to prevent this from happening without extra steps/confirmation, and went ahead and issued a reboot command to every server in us-east-1 availability zone without delay. "To make error is human. To propagate error to all server in automatic way is #devops." -@DEVOPS_BORAT [1] [1]https://twitter.com/DEVOPS_BORAT/status/41587168870797312 https://twitter.com/DEVOPS_BORAT/status/41587168870797312
sathio 12y ago
so, basically it was like a "DELETE FROM table;" without limit :)
- le_meta 12y ago
  It seems your fleet is running uncertified applications from unknown parties. your fleet has been rebooted for your protection.
- phil21 12y ago
  One of the worst days of my life was when one of my techs called me as I hopped on a plane... Tech: "hey phil, I just broke $bigcustomer's database" Phil: "..." Tech: "I typed drop database ImportantDB;2" Tech: "instead of drop database Importantdb2;" Even worse because this was lets say.. a very undersized install due to customer cheapness. And all we had for backups (since who wants to PAY for backups?!?!) was a day-old mysqldump off a slave. It took multiple days to re-import all that data. Customer was not pleased.
hueving 12y ago
This is the same company that chose to publicly shame one of the best contributors to node.js into quitting because he closed a pull request changing a gendered pronoun in a code comment. They place very little value on engineering excellence and more on 'being cool'. This leads to half-baked tools using <buzzword technology> like the one described in the article that allowed every server to be simultaneously rebooted without confirmation.
- mirashii 12y ago
  For those wondering why this is downvoted, read the "shaming" and the reasons behind it for yourself. http://www.joyent.com/blog/the-power-of-a-pronoun http://www.joyent.com/blog/the-power-of-a-pronoun
  - hueving 12y ago
    That's the Joyent side explaining that they would have fired him over it. Here is a news outlet report of it saying that it was mainly a language barrier issue. https://gigaom.com/2013/12/02/slap-fight-in-node-js-land/ https://gigaom.com/2013/12/02/slap-fight-in-node-js-land/ Either way, it was just shocking to see how quickly they were willing to throw a contributor under the bus who put more than all of Joyent combined into libuv without even trying to understand the his reasoning.
  - vacri 12y ago
    I've always hated that essay. "We value empathy here at Joyent. That's why we would rather fire an [theoretical] employee than retrain them".
    - chris_wot 12y ago
      It's funny how Cantrill is so willing to throw someone else under a bus, yet he's the guy who once asked if another developer had "ever kissed a girl"?
deleted 12y ago
[deleted]
- bcantrill 12y ago
  I'm not sure which design issue you're referring to; we absolutely did consider this scenario and the system behaved essentially as designed in this regard (in particular, the contingency that we can determine a compute node's platform image while the control plane was impaired was critical for our relatively quick recovery). The only real issue with the recovery of compute nodes was the CPU throttling that we encountered as the entire fleet rebooted simultaneously, but that was quickly discovered and remediated (and would have only resulted in a longer recovery time, not an inability to recover). Edit: the comment that I was replying to was apparently deleted. I won't attribute it, but here is the text I was replying to: I'm generally the last person to finger point but wow! You guys didn't think about and test what happens when a lot of your servers reboot simultaneously? That happens any time you lose power! This exact issue has been raised and discussed every single time I've deployed any substantial number of PXE booting machines. There are actually a number of issues with relying on PXE booting and this is just one of them. This design make is far worse than the mistyped command.
samstave 12y ago
There should be Zero method to reboot the whole dc by one operator. There should be means where only issuance of seperate commands from separate operators can, when jointly submitted, will result in simultaneous dc reboot. Tpi and all that.
- paulwolf 12y ago
  As a non-security focused company, they probably don't think there is an issue giving so much power to one command that can be issued by a single person.

83 points by akerl_ 12y ago

jffry 12y ago

It's great to see a postmortem with this level of detail, fairly quickly. It's also great to see Joyent hang the blame on the system that allowed rebooting every server, and the poor recovery from that failure, rather than continuing to throw the operator under the bus: "...we will be rethinking what tools are necessary over the coming days and weeks so that "full power" tools are not the only means by which to accomplish routine tasks."

blantonl 12y ago
More importantly, I hope that the person that issued this command still keeps his job. He learned an important lesson and unless there is sheer incompetence here, this individual will have a medal on his chest indicating he has been in the worst "combat" that a sysadmin can endure. Everyone makes mistakes, and judging by the language of the postmortem it appears it was just that. Would love to have that sysadmin on my team, because he will never do that again....
- bcantrill 12y ago
  We debated whether or not to make this explicit in the postmortem, but yes, the operator in question still has their job, and for exactly the reasons that you outlined: it was an honest mistake, they were deeply apologetic (as one might imagine) -- and we know that they (certainly!) won't be making that mistake again. Mistakes like this are their own punishment; additional punitive action serves only to instill fear rather than effect the changes necessary to not repeat the failure.
  - dmourati 12y ago
    "Five why's" might be appropriate to suss out how the ultimate mistake was even possible.
- patio11 12y ago
  It is said, perhaps apocryphally, that the head of the trading desk at Mizuho, when asked whether he fired the woman who physically keyed in an order for 600k shares at 1 yen rather than an order for 1 share at 600k yen ($300 million or so in losses), said "Why would I do that, after spending $300 million to make her the least likely person in Japan to typo trade instructions."
  - bagels 12y ago
    That quote is interesting because because the conclusion doesn't follow. It's basically the Monte Carlo fallacy. She's probably less likely than before due to fear or guilt, but I don't know that this makes her less error prone than every other person in the country, including those that double check every time, for instance.
bobbles 12y ago
>continuing to throw the operator under the bus There has been no evidence of them doing this at all.. I don't know why people keep saying it.
- jffry 12y ago
  That was poorly phrased... In my mind, I meant it in a "sadly it's too often that the human making the mistake gets blamed, rather than the systems that didn't have appropriate safeguards to make said mistake non-trivial"
- paulwolf 12y ago
  They threw an open-source contributor under the bus for rejecting a pull request that changed a code comment. Knee-jerk responses are definitely part of their modus operandi.
  - chris_wot 12y ago
    Agreed. They seem to be good at threatening to fire employees of other companies, but not their own. I suppose that's a net positive. Still.

JBiserkov 12y ago

>The command to reboot the select set of new systems that needed to be updated was mis-typed, and instead specified all servers in the datacenter. Substitute reboot with 'upgrade to win 7' and data center with university at get a story from a month? ago. rm ./*.* - delete all files in current dir rm /*.* - never do this Can you spot the difference? A colleague did the later on our testing server. No clients were disturbed, but us devs were left working on things that can run locally (much more pleasant stuff for sure, yet the schedule suffered a lot).

adamfeldman 12y ago
The incident was at Emory University: http://thenextweb.com/shareables/2014/05/16/emory-university-server-accidentally-sends-reformat-request-windows-pcs-including/ http://thenextweb.com/shareables/2014/05/16/emory-university...
jacquesm 12y ago
Your 'never do this' line would erase initrd.img on most machines, which you'd only find out about after the next reboot if nobody told you about it (and good luck getting that fixed). There are a large number of varieties of this particular error. Some with terrible results. rm -rf * .bak for instance (especially when executed in the root directory). That '#' prompt is there for a reason. The way to solve these sort of issues is to first get the files using 'find' until you're totally happy about the result and then to use 'rm' as the command passed to find. Of course, nobody does this ;)
- jethro_tell 12y ago
  >Your 'never do this' line Would also erase everything else on the box so you might not know for a few seconds, but then the database errors start and the lib and pid files start disappearing and now your the king of a mountain of shit. It won't take till next reboot to notice. If you ever get a chance, rm -rf / a box before you throw it out and just play around with it for a couple minutes while it eats itself. You can spare your self most errors like that with tab complete, the built in sanity checker. I tab complete everything since I'm dyslexic, and it saves me a shit ton of time cause I'm never far from the error when I notice it.

incision 12y ago

>'In our experience, platforms with this network device will encounter this boot-time issue about 10% of the time. The mitigation for this is for an operator to simply initiate another reboot, which we performed on those afflicted nodes as soon as we identified them.' This bit bothers me more than anything else. It's not just a die roll configuration, but a known one with an operator required to do the re-rolling. Everything is a value judgement I guess, but knowingly leaving that one 'mitigated' would drive me insane.

bcantrill 12y ago
Sorry if we didn't adequately convey our frustration with this particular issue. It's one that's been with us for a long time and it absolutely sucks -- and after trying (and failing) to work with the vendor to get the issue understood (it's essentially a firmware-level issue), we ultimately decided to move away from that particular NIC vendor entirely. If we could wave a wand and be rid of these particular parts, we gladly would -- but until then, this transient boot-time issue needs to be manually mitigated with an additional reboot.
- incision 12y ago
  Sure, I mean to relate more than judge - I've been through similar hoops though surely on a smaller scale.
- AceJohnny2 12y ago
  Why the shyness about saying the brand's name? If their product and support was subpar to the point of blacklisting the entire vendor, it could be useful to spread the information to A) warn others of potential problems and B) put pressure on the vendor to improve their products.
  - bcantrill 12y ago
    I'm with you, but somehow a postmortem for our own outage seemed like the wrong place to name-and-shame a vendor...

adamfeldman 12y ago

I don't have a lot of experience with datacenters, and am trying to understand this sentence: "Because there was a simultaneous reboot of every system in the datacenter, there was extremely high contention on the TFTP boot infrastructure, which like all of our infrastructure, normally has throttles in place to ensure that it cannot run away with a machine." What does "cannot run away with a machine" mean? Why would you want to by default restrict the speed at which that system runs?

jacquesm 12y ago
I think they're trying to say that that machine has other duties except for TFTP and that the process is running at a priority low enough that it can be pre-empted by other processes. So if there is a lot of contention on the TFTP boot infrastructure the throttles limit the amount of work a single machine can do, which in turn can cause starvation. The reason why is probably related to what it does besides TFTP, which under normal circumstances is probably more important than new nodes joining the network. During a mass reboot that will bite you because then the chances of starvation will go up quite a bit as the throttles cause one machine after another to time-out and retry their boot sequence.
ChargingWookie 12y ago
They wanted to prevent all of the servers trying to contact the PXE server at the same time to avoid overloading it. Its basically to prevent an accidently self DDOS.
CHY872 12y ago
I also have no experience, but my guess is that when you are running normally, you want to share the bandwidth of a server fairly between all clients. When you've just accidentally rebooted all servers, you want to do it completely unfairly - so that instead of 10 computers taking an hour to download something, the first takes 6 minutes, the second takes 12 minutes etc. This allows at least some of the servers to get started and your infrastructure to start getting back online. I.e. you want to change from round robin scheduling to shortest job first scheduling (which gives the minimum average waiting time).
deleted 12y ago

geetarista 12y ago

On behalf of all of Joyent, we are extremely sorry for this outage, and the severe inconvenience it may have caused to you, and your customers. I hate it when people apologize saying maybe it was inconvenient. If someone is using your service and you fucked up, it is an inconvenience.

RussianCow 12y ago
Doesn't mean everyone is affected by it. If I was hosting some unimportant website on their platform, I probably wouldn't even notice, and it wouldn't be an inconvenience to me. Silly thing to feel so strongly about.
FireBeyond 12y ago
Disagree. They acknowledge it may be severely inconvenient, not just an 'oops'.

bignaj 12y ago

>The command to reboot the select set of new systems that needed to be updated was mis-typed, and instead specified all servers in the datacenter. Unfortunately the tool in question does not have enough input validation to prevent this from happening without extra steps/confirmation, and went ahead and issued a reboot command to every server in us-east-1 availability zone without delay. "To make error is human. To propagate error to all server in automatic way is #devops." -@DEVOPS_BORAT [1] [1]https://twitter.com/DEVOPS_BORAT/status/41587168870797312 https://twitter.com/DEVOPS_BORAT/status/41587168870797312

sathio 12y ago

so, basically it was like a "DELETE FROM table;" without limit :)

le_meta 12y ago
It seems your fleet is running uncertified applications from unknown parties. your fleet has been rebooted for your protection.
phil21 12y ago
One of the worst days of my life was when one of my techs called me as I hopped on a plane... Tech: "hey phil, I just broke $bigcustomer's database" Phil: "..." Tech: "I typed drop database ImportantDB;2" Tech: "instead of drop database Importantdb2;" Even worse because this was lets say.. a very undersized install due to customer cheapness. And all we had for backups (since who wants to PAY for backups?!?!) was a day-old mysqldump off a slave. It took multiple days to re-import all that data. Customer was not pleased.

hueving 12y ago

This is the same company that chose to publicly shame one of the best contributors to node.js into quitting because he closed a pull request changing a gendered pronoun in a code comment. They place very little value on engineering excellence and more on 'being cool'. This leads to half-baked tools using <buzzword technology> like the one described in the article that allowed every server to be simultaneously rebooted without confirmation.

mirashii 12y ago
For those wondering why this is downvoted, read the "shaming" and the reasons behind it for yourself. http://www.joyent.com/blog/the-power-of-a-pronoun http://www.joyent.com/blog/the-power-of-a-pronoun
- hueving 12y ago
  That's the Joyent side explaining that they would have fired him over it. Here is a news outlet report of it saying that it was mainly a language barrier issue. https://gigaom.com/2013/12/02/slap-fight-in-node-js-land/ https://gigaom.com/2013/12/02/slap-fight-in-node-js-land/ Either way, it was just shocking to see how quickly they were willing to throw a contributor under the bus who put more than all of Joyent combined into libuv without even trying to understand the his reasoning.
- vacri 12y ago
  I've always hated that essay. "We value empathy here at Joyent. That's why we would rather fire an [theoretical] employee than retrain them".
  - chris_wot 12y ago
    It's funny how Cantrill is so willing to throw someone else under a bus, yet he's the guy who once asked if another developer had "ever kissed a girl"?

deleted 12y ago

[deleted]

bcantrill 12y ago
I'm not sure which design issue you're referring to; we absolutely did consider this scenario and the system behaved essentially as designed in this regard (in particular, the contingency that we can determine a compute node's platform image while the control plane was impaired was critical for our relatively quick recovery). The only real issue with the recovery of compute nodes was the CPU throttling that we encountered as the entire fleet rebooted simultaneously, but that was quickly discovered and remediated (and would have only resulted in a longer recovery time, not an inability to recover). Edit: the comment that I was replying to was apparently deleted. I won't attribute it, but here is the text I was replying to: I'm generally the last person to finger point but wow! You guys didn't think about and test what happens when a lot of your servers reboot simultaneously? That happens any time you lose power! This exact issue has been raised and discussed every single time I've deployed any substantial number of PXE booting machines. There are actually a number of issues with relying on PXE booting and this is just one of them. This design make is far worse than the mistyped command.

samstave 12y ago

There should be Zero method to reboot the whole dc by one operator. There should be means where only issuance of seperate commands from separate operators can, when jointly submitted, will result in simultaneous dc reboot. Tpi and all that.

paulwolf 12y ago
As a non-security focused company, they probably don't think there is an issue giving so much power to one command that can be issued by a single person.