10 ms·

Having only ever seen one major outage event in person (at a financial institution that hadn't yet come up with an incident response plan; cue three days of mad

Having only ever seen one major outage event in person (at a financial institution that hadn't yet come up with an incident response plan; cue three days of madness), I would love to be a fly on the wall at Google or other well-established engineering orgs when something like this goes down. I'd love to see the red binders come down off the shelf, people organize into incident response groups, and watch as a root cause is accurately determined and a fix out in place. I know it's probably more chaos than art, but I think there would be a lot to learn by seeing it executed well.

tazjin 7y ago
It's interesting to see it go down. There's some chaos involved, but from my perspective it's the constructive[0] kind. If you're interested in how these sorts of incidents are managed, check out the SRE Book[1] - it has a chapter or two on this and many other related topics. Disclosure: I work in Google Cloud, but not SRE. [0]: https://principiadiscordia.com/book/70.php https://principiadiscordia.com/book/70.php [1]: https://landing.google.com/sre/books/ https://landing.google.com/sre/books/
- alasdair_ 7y ago
  Our own version of Netflix's "Chaos Money" is named "Eris" for precisely the reason mentioned in your first footnote.
  - tobych 7y ago
    Chaos Monkey: https://netflix.github.io/chaosmonkey/ https://netflix.github.io/chaosmonkey/
    - deleted 7y ago
      [deleted]
roganartu 7y ago
I used to be an SRE at Atlassian in Sydney on a team that regularly dealt with high-severity incidents, and I was an incident manager for probably 5-10 high severity Jira cloud incidents during my tenure too, so perhaps I can give some insight. I left because the SRE org in general at the time was too reactionary, but their incident response process was quite mature (perhaps by necessity). The first thing I'll say is that most incident responses are reasonably uneventful and very procedural. You do some initial digging to figure out scope if it's not immediately obvious, make sure service owners have been paged, create incident communication channels (at least a slack room if not a physical war room) and you pull people into it. The majority of the time spent by the incident manager is on internal and external comms to stakeholders, making sure everyone is working on something (and often more importantly that nobody is working on something you don't know about), and generally making sure nobody is blocked. To be honest, despite the fact that it's more often dealing with complex systems for which there is a higher rate of change and the failure modes are often surprising, the general sentiment in a well-run incident war room resembles black box recordings of pilots during emergencies. Cool, calm, and collected. Everyone in these kinds of orgs tend to quickly learn that panic doesn't help, so people tend to be pretty chill in my experience. I work in finance now in an org with no formally defined incident response process and the difference is pretty stark in the incidents I've been exposed to, generally more chaotic as you describe.
- opportune 7y ago
  Yes this is also how it's done at other large orgs. But one key to a quick response is for every low-level team to have at least one engineer on call at any given time. This makes it so any SRE team can engage with true "owners" of the offending code ASAP. Also during an incident, fingers are never publicly/embarrassingly pointed nor are people blamed. It's all about identifying and fixing the issue as fast as possible, fixing it, and going back to sleep/work/home. For better or worse, incidents become routine so everyone knows exactly what do and that as long as the incident is resolved soon, it's not the end of the world, so no histrionics are required.
  - ethbro 7y ago
    I've only been tangentially pulled into high severity incidents, but the thing that most impressed me was the quiet. As mentioned in this thread, it's a lot like listening to air traffic comm chatter. People say what they know, and only what they know, and clearly identify anything they're unsure about. Informative and clear communication matters more than brilliance. Most of the traffic is async task identification, dispatch, and then reporting in. And if anyone is screaming or gets emotional, they should not be in that room.
lytfyre 7y ago
You might be interested in https://response.pagerduty.com/ https://response.pagerduty.com/, PagerDuty's major incident response process documentation - a good starting point for that red binder. Having been in the ringmasters seat for major incidents ranging from "relatively routine" to "it's all on fire", and had a ringside seat for a cloud provider outage of comparable magnitude to this one - it still fascinates me how creative solutions can get dreamed up under high pressure, and how effective someone to keep the response calm and _feel like it's in control_ is.
- gbrayut 7y ago
  Anyone know of other public resources like the one from PagerDuty? The SRE book and workbook at https://landing.google.com/sre/books/ https://landing.google.com/sre/books/ have some details, but curious if there are others people would recommend.
  - di4na 7y ago
    Anything out of Mission Control during Apollo. The Army has good stuff too. FEMA has some good stuff on how they apply it on the ground and train. I particularly like Gene Kranz "Failure is not an Option". It is more background but it works. In general, it is not crazy hard. You get the roles, you distribute them. Someone can have multiple roles that depends on the size of the incident. The usual roles i differentiate are Point (think of it as IC if you want), Comms and Logs
achiang 7y ago
Google SRE doesn't have magical incident response beans that we hoard from the rest of the world. What makes Google SRE institutionally strong is that we have senior executive support to execute on all the best practices described in the book: https://landing.google.com/sre/sre-book/toc/index.html https://landing.google.com/sre/sre-book/toc/index.html At my last job, I bought a copy of this book, but we only had the organizational bandwidth to do a few of the things mentioned. At Google, we do all of them. The incident on Sunday basically played out as described in chapters 13 and 14. There is always the fog of war that exists during an incident, so no, it wasn't always people calmly typing into terminals, but having good structure in place keeps the madness manageable. Disclosure: I work in Google NetInfra SRE, and while my department was/is heavily involved in this incident, I personally was not. Also, we're [always] hiring: https://careers.google.com/jobs/results/?company=Google&company=Google%20Fiber&company=YouTube&employment_type=FULL_TIME&hl=en_US&q=SRE&sort_by=relevance https://careers.google.com/jobs/results/?company=Google&comp...
elliekelly 7y ago
Tangentially related, you might find the documentary "Out of the Clear Blue Sky" interesting. It's about bond trading firm Cantor Fitzgerald (headquartered on the top floors of the World Trade Center) in the days after 9/11. Not even the best plan could have helped them open for business in two days after having lost just about everything. Over the last decade or so we've put a lot of emphasis on documentation when it comes to incident response but the movie is really a testament to how leadership and execution are so much more important.

tazjin 7y ago

It's interesting to see it go down. There's some chaos involved, but from my perspective it's the constructive[0] kind. If you're interested in how these sorts of incidents are managed, check out the SRE Book[1] - it has a chapter or two on this and many other related topics. Disclosure: I work in Google Cloud, but not SRE. [0]: https://principiadiscordia.com/book/70.php https://principiadiscordia.com/book/70.php [1]: https://landing.google.com/sre/books/ https://landing.google.com/sre/books/

alasdair_ 7y ago
Our own version of Netflix's "Chaos Money" is named "Eris" for precisely the reason mentioned in your first footnote.
- tobych 7y ago
  Chaos Monkey: https://netflix.github.io/chaosmonkey/ https://netflix.github.io/chaosmonkey/
  - deleted 7y ago
    [deleted]

roganartu 7y ago

I used to be an SRE at Atlassian in Sydney on a team that regularly dealt with high-severity incidents, and I was an incident manager for probably 5-10 high severity Jira cloud incidents during my tenure too, so perhaps I can give some insight. I left because the SRE org in general at the time was too reactionary, but their incident response process was quite mature (perhaps by necessity). The first thing I'll say is that most incident responses are reasonably uneventful and very procedural. You do some initial digging to figure out scope if it's not immediately obvious, make sure service owners have been paged, create incident communication channels (at least a slack room if not a physical war room) and you pull people into it. The majority of the time spent by the incident manager is on internal and external comms to stakeholders, making sure everyone is working on something (and often more importantly that nobody is working on something you don't know about), and generally making sure nobody is blocked. To be honest, despite the fact that it's more often dealing with complex systems for which there is a higher rate of change and the failure modes are often surprising, the general sentiment in a well-run incident war room resembles black box recordings of pilots during emergencies. Cool, calm, and collected. Everyone in these kinds of orgs tend to quickly learn that panic doesn't help, so people tend to be pretty chill in my experience. I work in finance now in an org with no formally defined incident response process and the difference is pretty stark in the incidents I've been exposed to, generally more chaotic as you describe.

opportune 7y ago
Yes this is also how it's done at other large orgs. But one key to a quick response is for every low-level team to have at least one engineer on call at any given time. This makes it so any SRE team can engage with true "owners" of the offending code ASAP. Also during an incident, fingers are never publicly/embarrassingly pointed nor are people blamed. It's all about identifying and fixing the issue as fast as possible, fixing it, and going back to sleep/work/home. For better or worse, incidents become routine so everyone knows exactly what do and that as long as the incident is resolved soon, it's not the end of the world, so no histrionics are required.
- ethbro 7y ago
  I've only been tangentially pulled into high severity incidents, but the thing that most impressed me was the quiet. As mentioned in this thread, it's a lot like listening to air traffic comm chatter. People say what they know, and only what they know, and clearly identify anything they're unsure about. Informative and clear communication matters more than brilliance. Most of the traffic is async task identification, dispatch, and then reporting in. And if anyone is screaming or gets emotional, they should not be in that room.

lytfyre 7y ago

You might be interested in https://response.pagerduty.com/ https://response.pagerduty.com/, PagerDuty's major incident response process documentation - a good starting point for that red binder. Having been in the ringmasters seat for major incidents ranging from "relatively routine" to "it's all on fire", and had a ringside seat for a cloud provider outage of comparable magnitude to this one - it still fascinates me how creative solutions can get dreamed up under high pressure, and how effective someone to keep the response calm and _feel like it's in control_ is.

gbrayut 7y ago
Anyone know of other public resources like the one from PagerDuty? The SRE book and workbook at https://landing.google.com/sre/books/ https://landing.google.com/sre/books/ have some details, but curious if there are others people would recommend.
- di4na 7y ago
  Anything out of Mission Control during Apollo. The Army has good stuff too. FEMA has some good stuff on how they apply it on the ground and train. I particularly like Gene Kranz "Failure is not an Option". It is more background but it works. In general, it is not crazy hard. You get the roles, you distribute them. Someone can have multiple roles that depends on the size of the incident. The usual roles i differentiate are Point (think of it as IC if you want), Comms and Logs

achiang 7y ago

Google SRE doesn't have magical incident response beans that we hoard from the rest of the world. What makes Google SRE institutionally strong is that we have senior executive support to execute on all the best practices described in the book: https://landing.google.com/sre/sre-book/toc/index.html https://landing.google.com/sre/sre-book/toc/index.html At my last job, I bought a copy of this book, but we only had the organizational bandwidth to do a few of the things mentioned. At Google, we do all of them. The incident on Sunday basically played out as described in chapters 13 and 14. There is always the fog of war that exists during an incident, so no, it wasn't always people calmly typing into terminals, but having good structure in place keeps the madness manageable. Disclosure: I work in Google NetInfra SRE, and while my department was/is heavily involved in this incident, I personally was not. Also, we're [always] hiring: https://careers.google.com/jobs/results/?company=Google&company=Google%20Fiber&company=YouTube&employment_type=FULL_TIME&hl=en_US&q=SRE&sort_by=relevance https://careers.google.com/jobs/results/?company=Google&comp...

elliekelly 7y ago

Tangentially related, you might find the documentary "Out of the Clear Blue Sky" interesting. It's about bond trading firm Cantor Fitzgerald (headquartered on the top floors of the World Trade Center) in the days after 9/11. Not even the best plan could have helped them open for business in two days after having lost just about everything. Over the last decade or so we've put a lot of emphasis on documentation when it comes to incident response but the movie is really a testament to how leadership and execution are so much more important.