As soon as significance is achieved, we show a "winner" If you've described that accurately, you're hurting some of your customers. It's not well-known (though

As soon as significance is achieved, we show a "winner" If you've described that accurately, you're hurting some of your customers. It's not well-known (though it should be obvious, in retrospect), but a test can produce a false positive when, in fact, it's a negative. The problem is that there is always noise in customer data and sometimes you'll randomly have enough conversions such that one of your tests will show a winner when it's not. Matts Einersen has some simulation code to show this: http://www.einarsen.no/is-your-ab-testing-effort-just-chasing-statistical-ghosts/ http://www.einarsen.no/is-your-ab-testing-effort-just-chasin... Evan Miller has a more in-depth (but harder to follow) discussion here: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html If, in fact, you show the winner as soon as significance is achieved, it's then trivial to demonstrate that some of your customers will lose money as a result. However, if you take this knowledge and then find a good way to incorporate this into your tool, you may have another great selling point that competitors do not (that being said, the idea is complicated enough that some customers won't understand it).