This is a technique I've used a lot, A/B testing things to understand how well they work and get detailed understanding of strengths and weaknesses in action.

This is a technique I've used a lot, A/B testing things to understand how well they work and get detailed understanding of strengths and weaknesses in action. But for the love of reason don't build two identical production systems with different hidden internals and ditch one .. that's ridiculous. Maybe it was exaggeration as hyperbole? Instead, the technique I use is when I'm building a system, and a choice comes up where two approaches would work reasonable well, and I've used one and not the other - then I use the approach I hadn't used previously. Just the one approach in that system (as long as it works reasonably well as expected). This means I end up using both approaches (in different systems, at different times) and get for nearly free the synergistic benefit of actual real world A/B testing the two approaches over time.