3 ms·
Wow, lots of impressive thinking ahead of time. I’m heading up our incident management team at work and this is definitely going in my presentations. A few thi
by karlkatzke 9y ago
Wow, lots of impressive thinking ahead of time. I’m heading up our incident management team at work and this is definitely going in my presentations.
A few things that jumped out:
- Thinking ahead to how he was going to handle an MCA probably saved dozens if not hundreds, even if he made some choices that he shouldn’t have if you’re sitting in our present armchairs.
- This is great demonstration of why the incident commander needs good oversight of the whole situation, and why not everyone makes a good incident commander. The three doctors in Station 1 should have felt empowered to solve the bottlenecks, but they did not... they just kept working within established procedure. That’s ok. You need good operators. But the good leaders in incidents are the people who know how and when to establish and communicate new standard procedures.
- Getting ahead of and staying ahead of a cascade failure is a difficult thing to manage, one that we don’t often accomplish in operations/sre incidents. I know I’ve had one or two incidents like that, mostly DDoS or other attack types. This story shows the value again of staying frosty and planning to handle your next problem before it snowballs and hits you from behind.
- JshWright 9y ago> This story shows the value again of staying frosty and planning to handle your next problem before it snowballs and hits you from behind. My experience in emergency services is my greatest asset in ops work. So, half the datacenter has crashed, and the other half is about to buckle under the load. Is anyone literally going to die in the next 5 minutes? Ok, cool, then let's just sort this out and get on with our lives.