henakama

3 ms·

1. 1.▲
   by henakama 11y ago

   Hi, kabouseng. Right now, I'm the only engineer working on this. I mentioned this in a comment on the blog post, but the main reason I was able to put this together by myself and do it relatively quickly was that I was able to leverag
2. 2.▲
   by henakama 11y ago

   Thanks for the feedback! I do have an example of a silent failure that would be classified as extreme chaos. Before we went into production, we were having problems with leader election that could cause multiple masters per cluster. This i
3. 3.▲
   by henakama 11y ago

   Regarding testing, I think that static functional tests that verify permutations of a specific set of inputs are still essential, in the same way that unit tests are needed even when you have comprehensive integration test coverage. Like th
4. 4.▲
   by henakama 11y ago

   There are definitely tools within Azure to perform fault injection, but I have not yet made use of them for our Search Chaos Monkey. Right now I've only taught the monkey how to perform basic Azure operations like restarting VMs and f
5. 5.▲
   by henakama 11y ago

   Hi, author of the post here. Happy to answer any questions, would love to hear from any of you who have used similar approaches to managing failure in distributed systems.