5 ms·

I used to be an SRE at Atlassian in Sydney on a team that regularly dealt with high-severity incidents, and I was an incident manager for probably 5-10 high sev

I used to be an SRE at Atlassian in Sydney on a team that regularly dealt with high-severity incidents, and I was an incident manager for probably 5-10 high severity Jira cloud incidents during my tenure too, so perhaps I can give some insight. I left because the SRE org in general at the time was too reactionary, but their incident response process was quite mature (perhaps by necessity). The first thing I'll say is that most incident responses are reasonably uneventful and very procedural. You do some initial digging to figure out scope if it's not immediately obvious, make sure service owners have been paged, create incident communication channels (at least a slack room if not a physical war room) and you pull people into it. The majority of the time spent by the incident manager is on internal and external comms to stakeholders, making sure everyone is working on something (and often more importantly that nobody is working on something you don't know about), and generally making sure nobody is blocked. To be honest, despite the fact that it's more often dealing with complex systems for which there is a higher rate of change and the failure modes are often surprising, the general sentiment in a well-run incident war room resembles black box recordings of pilots during emergencies. Cool, calm, and collected. Everyone in these kinds of orgs tend to quickly learn that panic doesn't help, so people tend to be pretty chill in my experience. I work in finance now in an org with no formally defined incident response process and the difference is pretty stark in the incidents I've been exposed to, generally more chaotic as you describe.

opportune 7y ago
Yes this is also how it's done at other large orgs. But one key to a quick response is for every low-level team to have at least one engineer on call at any given time. This makes it so any SRE team can engage with true "owners" of the offending code ASAP. Also during an incident, fingers are never publicly/embarrassingly pointed nor are people blamed. It's all about identifying and fixing the issue as fast as possible, fixing it, and going back to sleep/work/home. For better or worse, incidents become routine so everyone knows exactly what do and that as long as the incident is resolved soon, it's not the end of the world, so no histrionics are required.
- ethbro 7y ago
  I've only been tangentially pulled into high severity incidents, but the thing that most impressed me was the quiet. As mentioned in this thread, it's a lot like listening to air traffic comm chatter. People say what they know, and only what they know, and clearly identify anything they're unsure about. Informative and clear communication matters more than brilliance. Most of the traffic is async task identification, dispatch, and then reporting in. And if anyone is screaming or gets emotional, they should not be in that room.
  - tetha 7y ago
    Someone at our place recently commented that the ops team during an incident strongly feels like NASA mission control in critical moments[1]. I wanted to protest, but that's surprisingly accurate. > And if anyone is screaming or gets emotional, they should not be in that room. If someone starts yelling around in my incident war room for no reason, they get thrown out. I'm a calm and quiet person, but bugging around during a major incident is one of the few things that make me mad. 1: https://youtu.be/Y0yOTanzx-s?t=3059 https://youtu.be/Y0yOTanzx-s?t=3059
    - Moru 7y ago
      When the insident is over, does it look like 55:50 in that video? :-)
      - tetha 7y ago
        We recently had a 15 month long project almost fail due to some stupid shit and a really wonky error no one understood so far. <Almost fail> as in "Keep the customer on the phone we have 3 possible things to try and don't hang up! No one leaves that call until I'm out of hacks to deploy!" That evening we had the entire ops team in the houston-mode for several hours. And yes, once we had a workaround in place the customer accepted, we reacted like that. Except we also had our critical-incident whiskey go around. Then the CEO walked in to congratulate us on that project. Whoops. But he's a good sport, so good times. :)
    - 7y ago
rorykoehler 7y ago
> I left because the SRE org in general at the time was too reactionary It shows in their products (though it's improving)

opportune 7y ago

Yes this is also how it's done at other large orgs. But one key to a quick response is for every low-level team to have at least one engineer on call at any given time. This makes it so any SRE team can engage with true "owners" of the offending code ASAP. Also during an incident, fingers are never publicly/embarrassingly pointed nor are people blamed. It's all about identifying and fixing the issue as fast as possible, fixing it, and going back to sleep/work/home. For better or worse, incidents become routine so everyone knows exactly what do and that as long as the incident is resolved soon, it's not the end of the world, so no histrionics are required.

ethbro 7y ago
I've only been tangentially pulled into high severity incidents, but the thing that most impressed me was the quiet. As mentioned in this thread, it's a lot like listening to air traffic comm chatter. People say what they know, and only what they know, and clearly identify anything they're unsure about. Informative and clear communication matters more than brilliance. Most of the traffic is async task identification, dispatch, and then reporting in. And if anyone is screaming or gets emotional, they should not be in that room.
- tetha 7y ago
  Someone at our place recently commented that the ops team during an incident strongly feels like NASA mission control in critical moments[1]. I wanted to protest, but that's surprisingly accurate. > And if anyone is screaming or gets emotional, they should not be in that room. If someone starts yelling around in my incident war room for no reason, they get thrown out. I'm a calm and quiet person, but bugging around during a major incident is one of the few things that make me mad. 1: https://youtu.be/Y0yOTanzx-s?t=3059 https://youtu.be/Y0yOTanzx-s?t=3059
  - Moru 7y ago
    When the insident is over, does it look like 55:50 in that video? :-)
    - tetha 7y ago
      We recently had a 15 month long project almost fail due to some stupid shit and a really wonky error no one understood so far. <Almost fail> as in "Keep the customer on the phone we have 3 possible things to try and don't hang up! No one leaves that call until I'm out of hacks to deploy!" That evening we had the entire ops team in the houston-mode for several hours. And yes, once we had a workaround in place the customer accepted, we reacted like that. Except we also had our critical-incident whiskey go around. Then the CEO walked in to congratulate us on that project. Whoops. But he's a good sport, so good times. :)
  - 7y ago

rorykoehler 7y ago

> I left because the SRE org in general at the time was too reactionary It shows in their products (though it's improving)