Yes! If you haven’t had an operational issue with a service in a while, you should force one on a testing to stack to make sure it fails like you expect, you c

Yes! If you haven’t had an operational issue with a service in a while, you should force one on a testing to stack to make sure it fails like you expect, you can recover gracefully, your monitors work, etc. (lots of folks call these “game days”). A service that hasn’t had an ops issue in a long time is a ticking time bomb: when it does fail, nobody will be very familiar with it, and environmental assumptions might have changed, which could result in taking a much longer time to recover. When I design services these days, I try to design them so these failure scenarios are constantly exercised. Eg if I care about multi-AZ resiliency, I try to design it so that it’s forced to fail over to other AZs all the time. Or at the least, write tests for the scenario. Exceptional behavior or code paths are dangerous.