Recovery-oriented computing. Seperate Bohr bugs (easy to reproduce) from Heisenbugs (difficult to reproduce/change their behaviour once investigated). Bohr bug

Recovery-oriented computing. Seperate Bohr bugs (easy to reproduce) from Heisenbugs (difficult to reproduce/change their behaviour once investigated). Bohr bugs generate an alert and happily meander through normal support channels. Heisenbugs go through phases - 1. Probation. On continued failure, 2. Restart. If the app or service fails after a restart, 3. Reboot. If the app or service fails after a reboot, 4. Re-image. If the app or service fails after re-imaging, 5. Remove/elimate the node.