6 ms·

It just came back up it seems (9AM EST). I imagine a room full of Amazon engineers all standing around freaking out and possibly yelling at interns.

amarka 10y ago
Can confirm that doing a search from the homepage search bar (before 9AM EST) threw a 500, seems to be back up.
- 3pt14159 10y ago
  Sorry to be pedantic, but you mean 9AM EDT or just 9AM Eastern Time if you want to say "9AM in New York" which is what most people mean. There are still areas of Canada (Southhampton Island) and Central America (Panama IIRC) that are on EST year round.
  - _quick_q 10y ago
    They were responding to the comment where "9AM EST" was explicitly used. I don't understand the value of your comment as it doesn't serve to clarify - I think most if not all readers would understand the reference.
    - GFK_of_xmaspast 10y ago
      There's an hour difference between "9am est" and "9am edt".
      - _quick_q 10y ago
        I understand, but that was not the point of my comment.
    - 3pt14159 10y ago
      Yeah, I saw that but I figured both people would see the response if I put it on the child comment since people often only look at comment replies and their children. Conceptually I was replying to both.
  - cwilkes 10y ago
    To out pedantic you: the problem only involved Amazon.com not Amazon.ca ;)
colmmacc 10y ago
One of my responsibilities at Amazon is "call leader". It's a rotation of long-time Amazon folks who facilitate the recovery process whenever there are outage events; call leaders run the conference call (there's always a conference call) and help people focus on the right things in the moment as well as making sure we gather everything we'll need for a rigorous post mortem. The first and most important principle that every call leader promotes is "stay calm". It would be natural for folks to freak out and get stressed when the clock is ticking on an outage. There's an innate human sense that our emotional state has to match the urgency of a problem. But this would only lead to a kind frenetic energy that isn't helpful. Instead, maintaining a collected composure quickly cools everything down so that actions can be thought through and communicated more clearly. Many people are surprised that when they join a call, especially with an experienced team, that it is so calm. If you listen to the audio of Michael Schumacher as an F1 driver, or to fighter pilots performing complex maneuvers, it can also be striking just how calm they are. It's very effective and a great lesson for teams under pressure.
- lazyant 10y ago
  conference calls are terrible for troubleshooting, let the tech do their jobs (they can use chat) and then for the postmortem you can conference.
  - colmmacc 10y ago
    In general, our conference calls are structured and laser focused on recovery; "what can we rollback, undo, failover or change to restore availability?" There's much less emphasis on figuring out the root cause, or the "why" of the issue, in that moment. That's another natural tendency - to want to understand the issue rather than fix it - that can be counterproductive. In that situation, and especially at scale, it's very useful to keep everyone on the same page about what actions are happening when, and to help folks out by showing them how to find things which may have changed (keep in mind the change may have been in a totally different system than the one that ultimately breaks). For example: it would slow things down if someone tried to flip data centers at the same time that someone else was rolling back software. If on a call it becomes clear that a team needs to make some kind of non-rollback change, like a code change, or push a new config, and thankfully that's very rare, we'll generally recommend they break off and work on it in isolation, while nominating a point person to liaise with everyone else. We do use chat too and that can be more effective as more people join. For postmortems we try hard to do in writing first, and then an in person review. It's very helpful to be able to re-assure and guide each other. I can back up the "Blame the system, not the engineer" guidance others have written about here too.

amarka 10y ago

Can confirm that doing a search from the homepage search bar (before 9AM EST) threw a 500, seems to be back up.

3pt14159 10y ago
Sorry to be pedantic, but you mean 9AM EDT or just 9AM Eastern Time if you want to say "9AM in New York" which is what most people mean. There are still areas of Canada (Southhampton Island) and Central America (Panama IIRC) that are on EST year round.
- _quick_q 10y ago
  They were responding to the comment where "9AM EST" was explicitly used. I don't understand the value of your comment as it doesn't serve to clarify - I think most if not all readers would understand the reference.
  - GFK_of_xmaspast 10y ago
    There's an hour difference between "9am est" and "9am edt".
    - _quick_q 10y ago
      I understand, but that was not the point of my comment.
  - 3pt14159 10y ago
    Yeah, I saw that but I figured both people would see the response if I put it on the child comment since people often only look at comment replies and their children. Conceptually I was replying to both.
- cwilkes 10y ago
  To out pedantic you: the problem only involved Amazon.com not Amazon.ca ;)

colmmacc 10y ago

One of my responsibilities at Amazon is "call leader". It's a rotation of long-time Amazon folks who facilitate the recovery process whenever there are outage events; call leaders run the conference call (there's always a conference call) and help people focus on the right things in the moment as well as making sure we gather everything we'll need for a rigorous post mortem. The first and most important principle that every call leader promotes is "stay calm". It would be natural for folks to freak out and get stressed when the clock is ticking on an outage. There's an innate human sense that our emotional state has to match the urgency of a problem. But this would only lead to a kind frenetic energy that isn't helpful. Instead, maintaining a collected composure quickly cools everything down so that actions can be thought through and communicated more clearly. Many people are surprised that when they join a call, especially with an experienced team, that it is so calm. If you listen to the audio of Michael Schumacher as an F1 driver, or to fighter pilots performing complex maneuvers, it can also be striking just how calm they are. It's very effective and a great lesson for teams under pressure.

lazyant 10y ago
conference calls are terrible for troubleshooting, let the tech do their jobs (they can use chat) and then for the postmortem you can conference.
- colmmacc 10y ago
  In general, our conference calls are structured and laser focused on recovery; "what can we rollback, undo, failover or change to restore availability?" There's much less emphasis on figuring out the root cause, or the "why" of the issue, in that moment. That's another natural tendency - to want to understand the issue rather than fix it - that can be counterproductive. In that situation, and especially at scale, it's very useful to keep everyone on the same page about what actions are happening when, and to help folks out by showing them how to find things which may have changed (keep in mind the change may have been in a totally different system than the one that ultimately breaks). For example: it would slow things down if someone tried to flip data centers at the same time that someone else was rolling back software. If on a call it becomes clear that a team needs to make some kind of non-rollback change, like a code change, or push a new config, and thankfully that's very rare, we'll generally recommend they break off and work on it in isolation, while nominating a point person to liaise with everyone else. We do use chat too and that can be more effective as more people join. For postmortems we try hard to do in writing first, and then an in person review. It's very helpful to be able to re-assure and guide each other. I can back up the "Blame the system, not the engineer" guidance others have written about here too.