It's so frustrating, because when it all boils down to it, the story is really quite simple: 1) What metrics reflect your customer experience? Monitor them, a

It's so frustrating, because when it all boils down to it, the story is really quite simple: 1) What metrics reflect your customer experience? Monitor them, alarm on them. Target them as a priority for improvement. Check in on them weekly at a minimum. 2) What metrics make up the metrics that reflect the customer experience? Monitor them as well, and consider whether alarming on them is the right thing to do. Use these to direct what you target to solve 1. 3) What do you need for SPOF resilience? If you've got something critical, you need a minimum of two servers running it, and no more than, say, 40% total CPU usage so that you've got sufficient overhead to cope with a single host failure and any unexpected work increase.