We started our current project running chaos monkey from almost day one, and a few weeks ago, someone fucked up a tag enforcement script on our dev aws account

We started our current project running chaos monkey from almost day one, and a few weeks ago, someone fucked up a tag enforcement script on our dev aws account and started shutting down every single instance, while all of our asgs kept restarting them. So basically every single service from consul to kafka to our microservices was down to a few or sometimes even no instances. Because we had already been dealing with chaos monkey, everything recovered by itself within a few minutes of the script stopping, and we barely had any alerts from any services going down. Also, when aws was pushing our meltdown patches, we just ignored all the notifications because we didn’t care about instances being shut down, because they go down randomly all the time. It’s a pain in the ass at first to deal with, but it definitely forces you to fix a lot of potential problems unless you want to spend all your time manually fixing stuff every day. Even just aside from chaos monkey, we build new base amis and push out rolling updates daily. That’ll also quickly suss out problems with not having version numbers locked, etc.