For most businesses, focusing on the math and making incremental improvements to the statistical methodology is a waste of time. There is a "good enough" approa

For most businesses, focusing on the math and making incremental improvements to the statistical methodology is a waste of time. There is a "good enough" approach to using tools like Optimizely and VWO. Instead, they should be focusing on the quality of the tests they run. Quality hypotheses, preferably backed by data-driven insights into behavior that test very clear, if not dramatic, changes to the user flow will be what leads to bottom-line improvements, not increasing the mathematical merit and rigor of poorly-conceived tests. Of course, doing both is ideal but I put more fault on the testing software than the companies using it. Keep track of historical conversion rate and adjust/account for noise in your historical conversion rate. Conceive a testing program that focuses on the quality testing and you'll likely see an upward trend in that historical conversion rate.