We have some manual QA scripts where we introduce error conditions like manually killing processes, containers, etc and make sure the alert is fired (after noti

We have some manual QA scripts where we introduce error conditions like manually killing processes, containers, etc and make sure the alert is fired (after notifying admins that we will be testing). We use Terraform to keep the configurations in sync - most alerting is done via AWS CloudWatch. This means we can test these conditions in the staging environment without affecting the production environment. It's not a very academically satisfying way to do it but it's a process done every few months and takes about 30 minutes to run through the plan. One day we'll will automate this and perform far more frequent tests but it's quite a pragmatic solution for now - our application changes frequently but the cloud configuration doesn't really.