4 ms·

You were downvoted but I think you're right. Netflix's Chaos Monkey is a similar idea: https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey https://github.co

• zzzcpan 8y ago
  Yeah, Chaos Monkey is a nice idea. Although it's more like an after the fact solution to the problem, not addressed from the beginning. I mean imagine if you have a distributed system like an object storage or a database. Before deploying it into production you try to evaluate how it performs in various conditions. But you can't really do it thoroughly enough and on a lot of data, you can only scratch the surface and this forms your expectations about performance. You deploy it into production and start monitoring how it performs and relying on that performance that you barely understand. Everything seems fine, until you add nodes to the system or replace an hdd and suddenly it becomes so slow, that nothing works anymore. Turns out everyone, including system designers, had broken assumptions about its performance and didn't even think about it, because testing fault tolerance was always done only separately from normal mode of operation and it was impossible to grasp how they affect each other and come up with better algorithms.