4 ms·
There are definitely tools within Azure to perform fault injection, but I have not yet made use of them for our Search Chaos Monkey. Right now I've only taught
by henakama 11y ago
There are definitely tools within Azure to perform fault injection, but I have not yet made use of them for our Search Chaos Monkey. Right now I've only taught the monkey how to perform basic Azure operations like restarting VMs and focused mostly on injecting failure into Azure Search components. It's absolutely something that I want to look into for future iterations.
We've found bugs in production that chaos engineering hasn't caught. I can't give you a percentage, but the monkey is great at catching a majority of bugs where it has coverage (places like our provisioning and scaling workflow). Most of the bugs that have slipped through are as a result of lack of coverage. I haven't added any operations that test the service under heavy load (large amount of data or large number of requests/second), and I feel like we've seen more production issues in that area as a result.