4 ms·
If you've never run a chaos experiment, how do you square up blast radius with running in prod? It seems like this setup works great if built from the get-go b
by jorblumesea 7y ago
If you've never run a chaos experiment, how do you square up blast radius with running in prod?
It seems like this setup works great if built from the get-go but incredibly painful and possibly dangerous if starting with existing applications.
- andreareina 7y agoSet up a parallel deployment, run the experiments there. Document the failures in as granular a way as possible, decree that future deployments aren't allowed to add to the set of known failures. Assign e.g. 20% time to fixing the known problems. When confidence passes a threshold, start running the experiments in prod. It's basically the strangler pattern[1]. It is painful, but can be made arbitrarily safe. [1] https://news.ycombinator.com/item?id=19122973 https://news.ycombinator.com/item?id=19122973