I've done a LOT of incident management and I'm not happy about it. The biggest issue I have run into other than burnout is this: Thinking and reasoning under p

I've done a LOT of incident management and I'm not happy about it. The biggest issue I have run into other than burnout is this: Thinking and reasoning under pressure are the enemy. Make as many decisions in advance as possible. Make flowcharts and decision trees with "decision criteria" already written down. If you have to figure something out or make a "decision" then things are really really bad. That happens sometimes, but when teams don't prep at all for incident management (pre-determined plans for common classes of problem) every incident is "really really bad" If have a low risk, low cost action with low confidence of high reward, I'm going to do it and just tell people it happened. Asking means I just lost a half-hour+ worth of money and if I just did it and I was wrong we would have lost 2 minutes of money. When management asks me why I did that, I point at the doc I wrote that my coworkers reviewed and mostly forgot about. A really common example is "it looks like most the errors are in datacenter X", you fail out of the datacenter. Maybe it was sampling bias or some other issue and it doesn't help, maybe the problem follows the traffic, maybe it just suddenly makes things better. No matter what we get signal. Establish well in advance of a situation what the common "solutions" to problems are and if you are oncall and responding, then just DO them and document+communicate as you do.