Implicit in your question is that AB testing is used at the exclusion of other decision making tools. Definitely not the way to go. The right starting point fo

Implicit in your question is that AB testing is used at the exclusion of other decision making tools. Definitely not the way to go. The right starting point for thinking about AB testing is every experience is something that feels safe to put into production. That's what you're doing, but something about it being a "test" makes it not feel real to people. Maybe that threshold is low for some call to action copy, maybe it's much higher for something more critical. Use whatever tools you otherwise would to get to this point. Then you can AB test to see if it actually works. The second part is that in theory AB testing doesn't specifically drive to local optimizations (you can test and iterate on big changes) but in practice it sure does. It also tends to encourage short term thinking, maybe comically so, because those things are faster to test. It's local optimization on the time axis. "Changing the expected ship date for custom work from 90 days to 30 days increased sales 4% with no increase in support tickets!" and they ran the test for 4 whole weeks, so how could it be wrong? Maybe someone should tell manufacturing about this test and how great it went...