Sorry I should have used the word probable which is limited to power, network, software and storage failures. The solution to these is redundancy and to test ea

Sorry I should have used the word probable which is limited to power, network, software and storage failures. The solution to these is redundancy and to test each failure condition in succession and combinations thereof. Sure there are more possibilities, but these should all be covered with a complete fallback DR strategy i.e. if not identifiable cause X, Y or Z then assume the worse, snapshot everything for debugging later and fallback to a complete restore from scratch. The trick is to make X, Y and Z self-healing and part of your architecture provisioning. I get the feeling most people here haven't dealt with large deployments and multiple site redundancy, five-nines reliability requirements and extensive DR planning, because the above is pretty obvious to those of us who have.