3 ms·

One of my responsibilities at Amazon is "call leader". It's a rotation of long-time Amazon folks who facilitate the recovery process whenever there are outage e

One of my responsibilities at Amazon is "call leader". It's a rotation of long-time Amazon folks who facilitate the recovery process whenever there are outage events; call leaders run the conference call (there's always a conference call) and help people focus on the right things in the moment as well as making sure we gather everything we'll need for a rigorous post mortem. The first and most important principle that every call leader promotes is "stay calm". It would be natural for folks to freak out and get stressed when the clock is ticking on an outage. There's an innate human sense that our emotional state has to match the urgency of a problem. But this would only lead to a kind frenetic energy that isn't helpful. Instead, maintaining a collected composure quickly cools everything down so that actions can be thought through and communicated more clearly. Many people are surprised that when they join a call, especially with an experienced team, that it is so calm. If you listen to the audio of Michael Schumacher as an F1 driver, or to fighter pilots performing complex maneuvers, it can also be striking just how calm they are. It's very effective and a great lesson for teams under pressure.

lazyant 10y ago
conference calls are terrible for troubleshooting, let the tech do their jobs (they can use chat) and then for the postmortem you can conference.
- colmmacc 10y ago
  In general, our conference calls are structured and laser focused on recovery; "what can we rollback, undo, failover or change to restore availability?" There's much less emphasis on figuring out the root cause, or the "why" of the issue, in that moment. That's another natural tendency - to want to understand the issue rather than fix it - that can be counterproductive. In that situation, and especially at scale, it's very useful to keep everyone on the same page about what actions are happening when, and to help folks out by showing them how to find things which may have changed (keep in mind the change may have been in a totally different system than the one that ultimately breaks). For example: it would slow things down if someone tried to flip data centers at the same time that someone else was rolling back software. If on a call it becomes clear that a team needs to make some kind of non-rollback change, like a code change, or push a new config, and thankfully that's very rare, we'll generally recommend they break off and work on it in isolation, while nominating a point person to liaise with everyone else. We do use chat too and that can be more effective as more people join. For postmortems we try hard to do in writing first, and then an in person review. It's very helpful to be able to re-assure and guide each other. I can back up the "Blame the system, not the engineer" guidance others have written about here too.
- jcrites 10y ago
  If you know exactly where the problem lies, then I agree that in some cases a conference call might not help. However, more often than not, part of the challenge in large-impact outages is figuring out what's going on and where the problem actually lies. In a microservices or SOA architecture, services can layer on top of other services, that call more services, that put messages in a queue for handling by yet another service, etc., belonging to different teams. Unwinding this to identify the problem is something that's much faster with a phone call. "Connecting the dots", especially across teams. A technical discussion within a team is likely to take place by chat. I've participated in a number of tech conference calls over the years, and I've always been impressed with the professionalism: the discussion is calm and focused on identifying the problem and recovering. I think folks strive for the "cool radio voice", the Houston Center Voice like was described in the SR-71 Ultimate Ground Speed Check story: http://oppositelock.kinja.com/favorite-sr-71-story-1079127041 http://oppositelock.kinja.com/favorite-sr-71-story-107912704... > Now the thing to understand about Center controllers, was that whether they were talking to a rookie pilot in a Cessna, or to Air Force One, they always spoke in the exact same, calm, deep, professional, tone that made one feel important. I referred to it as the " Houston Center voice." I have always felt that after years of seeing documentaries on this country's space program and listening to the calm and distinct voice of the Houston controllers, that all other controllers since then wanted to sound like that, and that they basically did. And it didn't matter what sector of the country we would be flying in, it always seemed like the same guy was talking. Over the years that tone of voice had become somewhat of a comforting sound to pilots everywhere. Conversely, over the years, pilots always wanted to ensure that, when transmitting, they sounded like Chuck Yeager, or at least like John Wayne. Better to die than sound bad on the radios.
- nostrebored 10y ago
  But you're not just troubleshooting, you're working closely with many other people to resolve a problem and mitigate negative outcomes

lazyant 10y ago

conference calls are terrible for troubleshooting, let the tech do their jobs (they can use chat) and then for the postmortem you can conference.

colmmacc 10y ago
In general, our conference calls are structured and laser focused on recovery; "what can we rollback, undo, failover or change to restore availability?" There's much less emphasis on figuring out the root cause, or the "why" of the issue, in that moment. That's another natural tendency - to want to understand the issue rather than fix it - that can be counterproductive. In that situation, and especially at scale, it's very useful to keep everyone on the same page about what actions are happening when, and to help folks out by showing them how to find things which may have changed (keep in mind the change may have been in a totally different system than the one that ultimately breaks). For example: it would slow things down if someone tried to flip data centers at the same time that someone else was rolling back software. If on a call it becomes clear that a team needs to make some kind of non-rollback change, like a code change, or push a new config, and thankfully that's very rare, we'll generally recommend they break off and work on it in isolation, while nominating a point person to liaise with everyone else. We do use chat too and that can be more effective as more people join. For postmortems we try hard to do in writing first, and then an in person review. It's very helpful to be able to re-assure and guide each other. I can back up the "Blame the system, not the engineer" guidance others have written about here too.
jcrites 10y ago
If you know exactly where the problem lies, then I agree that in some cases a conference call might not help. However, more often than not, part of the challenge in large-impact outages is figuring out what's going on and where the problem actually lies. In a microservices or SOA architecture, services can layer on top of other services, that call more services, that put messages in a queue for handling by yet another service, etc., belonging to different teams. Unwinding this to identify the problem is something that's much faster with a phone call. "Connecting the dots", especially across teams. A technical discussion within a team is likely to take place by chat. I've participated in a number of tech conference calls over the years, and I've always been impressed with the professionalism: the discussion is calm and focused on identifying the problem and recovering. I think folks strive for the "cool radio voice", the Houston Center Voice like was described in the SR-71 Ultimate Ground Speed Check story: http://oppositelock.kinja.com/favorite-sr-71-story-1079127041 http://oppositelock.kinja.com/favorite-sr-71-story-107912704... > Now the thing to understand about Center controllers, was that whether they were talking to a rookie pilot in a Cessna, or to Air Force One, they always spoke in the exact same, calm, deep, professional, tone that made one feel important. I referred to it as the " Houston Center voice." I have always felt that after years of seeing documentaries on this country's space program and listening to the calm and distinct voice of the Houston controllers, that all other controllers since then wanted to sound like that, and that they basically did. And it didn't matter what sector of the country we would be flying in, it always seemed like the same guy was talking. Over the years that tone of voice had become somewhat of a comforting sound to pilots everywhere. Conversely, over the years, pilots always wanted to ensure that, when transmitting, they sounded like Chuck Yeager, or at least like John Wayne. Better to die than sound bad on the radios.
nostrebored 10y ago
But you're not just troubleshooting, you're working closely with many other people to resolve a problem and mitigate negative outcomes