better, I also got it set up with selenium doing a set of tests, my own version of a chaos monkey (a powershell script) that created failing instances, all trig

better, I also got it set up with selenium doing a set of tests, my own version of a chaos monkey (a powershell script) that created failing instances, all triggered as part of a TeamCity process.... Too much faff to do it in TFS. The glue code for this is truly horrific, I'm thinking of buying ranerex. I've not yet linked it up to the build quality (which I want to do) and I'm also thinking of making a more Windows Focused Chaos Monkey OS project. I dislike the idea of testing only that a VM stops. I want to see what happens if I hammer say its CPU with a few higher priority threads. Or simulate insane network latency, or some other networking quirk. I worry I am only testing against very well behaved nodes disappearance. Ultimately I'll have a nice little dashboard, which allows me to switch instance between UAT2/PROD, that shows me the available reserve for each environment. I'm toying with making some OWIN extension that allows me to control which components are loaded. For example, we have a 'chat' component, which is obviously not essential. I'd love the idea of automatically failing that. As we add more value add features, certain models and the like which are CPU demanding, these features will not have any uptime guarantee, so I'd like to automatically throttle them, in the case of a catastrophic failure (like a DC loosing connectivity). I am looking for something 99.999%. I've got three physical separate DC sites, each DC site has 3 or more servers. We could run the entire system on one server, easily.