Unit and Integration Tests don't always catch everything but cluster immune systems do a pretty good job on the important stuff: When someone wanted to do a de

Unit and Integration Tests don't always catch everything but cluster immune systems do a pretty good job on the important stuff: When someone wanted to do a deployment, we had a completely automated system that we called the cluster immune system. This would deploy the change incrementally, one machine at a time. That process would continually monitor the health of those machines, as well as the cluster as a whole, to see if the change was causing problems. If it didn't like what was going on, it would reject the change, do a fast revert, and lock deployments until someone investigated what went wrong. http://www.startuplessonslearned.com/2008/11/five-whys.html http://www.startuplessonslearned.com/2008/11/five-whys.html