Time to pop this old chestnut out: https://blog.fastmail.com/2014/12/14/on-duty/ https://blog.fastmail.com/2014/12/14/on-duty/ "At one stage, we decided to tr

Time to pop this old chestnut out: https://blog.fastmail.com/2014/12/14/on-duty/ https://blog.fastmail.com/2014/12/14/on-duty/ "At one stage, we decided to try to avoid having to be woken for some types of failure by using Heartbeat, a high availability solution for Linux, on our frontend servers. The thing is, our servers are actually really reliable, and we found that heartbeat failed more often than our systems - so the end result was reduced reliability! It's counter-intuitive, but automated high-availability often isn't." One of these days we'll finish our new system and I'll blog about that, which is that the automated systems are allowed to take ONE corrective action without paging, at which point they flag that the system is in compromised state. Any further test failures trigger an immediate wake of the on-call.