Set up a parallel deployment, run the experiments there. Document the failures in as granular a way as possible, decree that future deployments aren't allowed t

Set up a parallel deployment, run the experiments there. Document the failures in as granular a way as possible, decree that future deployments aren't allowed to add to the set of known failures. Assign e.g. 20% time to fixing the known problems. When confidence passes a threshold, start running the experiments in prod. It's basically the strangler pattern[1]. It is painful, but can be made arbitrarily safe. [1] https://news.ycombinator.com/item?id=19122973 https://news.ycombinator.com/item?id=19122973