3 ms·

How do other Azure teams inside Microsoft do their testing (SQL Azure, Azure DocDB etc?) I'd assume there's a common platform that allows you to just throw stuf

How do other Azure teams inside Microsoft do their testing (SQL Azure, Azure DocDB etc?) I'd assume there's a common platform that allows you to just throw stuff and inject failures? Further, in production, what percent of bugs if any did you find were not caught by Chaos engineering? Where they of a particular class? Did you still see a necessity for functional tests that test all permutations diligently?

henakama 11y ago
There are definitely tools within Azure to perform fault injection, but I have not yet made use of them for our Search Chaos Monkey. Right now I've only taught the monkey how to perform basic Azure operations like restarting VMs and focused mostly on injecting failure into Azure Search components. It's absolutely something that I want to look into for future iterations. We've found bugs in production that chaos engineering hasn't caught. I can't give you a percentage, but the monkey is great at catching a majority of bugs where it has coverage (places like our provisioning and scaling workflow). Most of the bugs that have slipped through are as a result of lack of coverage. I haven't added any operations that test the service under heavy load (large amount of data or large number of requests/second), and I feel like we've seen more production issues in that area as a result.
henakama 11y ago
Regarding testing, I think that static functional tests that verify permutations of a specific set of inputs are still essential, in the same way that unit tests are needed even when you have comprehensive integration test coverage. Like the old axiom, the earlier you find a bug, the cheaper it is to fix. The Search Chaos Monkey tests run against the current state of the repository, are not very precise and are relatively expensive and time-consuming to run. I only know when a test has failed because we start getting notifications that the service is unavailable, or has degraded performance. I still need to go dig up logs to see how the fault actually impacted the system, then go file a bug against code that's already been committed. If I spend 15 minutes digging through logs only to find that an error was caused by a bug in Azure Search code that could have been caught at a lower level of testing before that code was checked in, I won't be very happy. The monkey is there to test how our code works as the underlying cluster changes states and experiences failures. Before any code gets passed to the monkey, we should be fairly confident through rigorous functional tests that it performs correctly in most circumstances.

henakama 11y ago

There are definitely tools within Azure to perform fault injection, but I have not yet made use of them for our Search Chaos Monkey. Right now I've only taught the monkey how to perform basic Azure operations like restarting VMs and focused mostly on injecting failure into Azure Search components. It's absolutely something that I want to look into for future iterations. We've found bugs in production that chaos engineering hasn't caught. I can't give you a percentage, but the monkey is great at catching a majority of bugs where it has coverage (places like our provisioning and scaling workflow). Most of the bugs that have slipped through are as a result of lack of coverage. I haven't added any operations that test the service under heavy load (large amount of data or large number of requests/second), and I feel like we've seen more production issues in that area as a result.

henakama 11y ago

Regarding testing, I think that static functional tests that verify permutations of a specific set of inputs are still essential, in the same way that unit tests are needed even when you have comprehensive integration test coverage. Like the old axiom, the earlier you find a bug, the cheaper it is to fix. The Search Chaos Monkey tests run against the current state of the repository, are not very precise and are relatively expensive and time-consuming to run. I only know when a test has failed because we start getting notifications that the service is unavailable, or has degraded performance. I still need to go dig up logs to see how the fault actually impacted the system, then go file a bug against code that's already been committed. If I spend 15 minutes digging through logs only to find that an error was caused by a bug in Azure Search code that could have been caught at a lower level of testing before that code was checked in, I won't be very happy. The monkey is there to test how our code works as the underlying cluster changes states and experiences failures. Before any code gets passed to the monkey, we should be fairly confident through rigorous functional tests that it performs correctly in most circumstances.