3 ms·
I've only been tangentially pulled into high severity incidents, but the thing that most impressed me was the quiet. As mentioned in this thread, it's a lot li
by ethbro 7y ago
I've only been tangentially pulled into high severity incidents, but the thing that most impressed me was the quiet.
As mentioned in this thread, it's a lot like listening to air traffic comm chatter.
People say what they know, and only what they know, and clearly identify anything they're unsure about. Informative and clear communication matters more than brilliance.
Most of the traffic is async task identification, dispatch, and then reporting in.
And if anyone is screaming or gets emotional, they should not be in that room.
- tetha 7y agoSomeone at our place recently commented that the ops team during an incident strongly feels like NASA mission control in critical moments[1]. I wanted to protest, but that's surprisingly accurate. > And if anyone is screaming or gets emotional, they should not be in that room. If someone starts yelling around in my incident war room for no reason, they get thrown out. I'm a calm and quiet person, but bugging around during a major incident is one of the few things that make me mad. 1: https://youtu.be/Y0yOTanzx-s?t=3059 https://youtu.be/Y0yOTanzx-s?t=3059
- Moru 7y agoWhen the insident is over, does it look like 55:50 in that video? :-)
- tetha 7y agoWe recently had a 15 month long project almost fail due to some stupid shit and a really wonky error no one understood so far. <Almost fail> as in "Keep the customer on the phone we have 3 possible things to try and don't hang up! No one leaves that call until I'm out of hacks to deploy!" That evening we had the entire ops team in the houston-mode for several hours. And yes, once we had a workaround in place the customer accepted, we reacted like that. Except we also had our critical-incident whiskey go around. Then the CEO walked in to congratulate us on that project. Whoops. But he's a good sport, so good times. :)
- di4na 7y agoIt is not surprising at all. Mission Control was forged in the fire (literally for Apollo 1) and they are one of the most visible "incident team" we know about. I highly advise to read Gene Kranz memoirs "Failure is not an Option" if you work in that kind of environment.
- ethbro 7y agoI heard recently that he never said that. Apparently it was mentioned when the Apollo 13 script writers were gathering stories at NASA, they liked it, and then gave it to the Kranz character. Who then decided, "Hey, if everyone thinks I said it..." and titled his memoir.
- di4na 7y agoYep exactly.