We have different tools for different kinds of failures. Chaos Monkey helps ensure that you are resilient to single instance failure. Kong helps ensure that we

We have different tools for different kinds of failures. Chaos Monkey helps ensure that you are resilient to single instance failure. Kong helps ensure that we are resilient to region failure. Most of the developer-induced pain (which is most frequent source of pain) happens at the service level -- a bad code push that somehow made it through canary, accidentally doing something you shouldn't, misconfiguring something, etc. For tolerating service-level failures, we use different tools that minimize the fallout of the failure injection. Specifically, FIT (and the soon to be revealed ChAP.) These tools allow us to be more surgical in our injection of failure and tie that into our telemetry solutions. We only inject failures we expect to be resilient to. Sadly, that is a subset of the failures that people cause ;)