The biggest mistake engineers make about A/B testing is not recognizing local maxima. Your test may be super successful, but there may be an even better soluti

The biggest mistake engineers make about A/B testing is not recognizing local maxima. Your test may be super successful, but there may be an even better solution that's significantly different than what you've arrived at. It's important to not only A/B test minor changes, but occasionally throw in some major changes to see if it moves the same metric, possibly even more than your existing success.