I did something similar and had a test suite across a CoreOS cluster that would fail machines in patterns for hours whilst checking service and data integrity.

I did something similar and had a test suite across a CoreOS cluster that would fail machines in patterns for hours whilst checking service and data integrity. One can run the suite against the replica cluster when new deployment features are added or changed. Doing it on live wasn't possible for me because failover resulted in some down time although I consider the former to have been very useful.