"The important lesson here, and the one you should take away from our experience, is this: Whenever you think you have enough data for the A/B test, get more! S

"The important lesson here, and the one you should take away from our experience, is this: Whenever you think you have enough data for the A/B test, get more! Sometimes, you will fall into that 0.1%, and your decision will be wrong, and might impact your metrics adversely, and you might never find out." This is actually terrible advice because continuing a test in which one set is significantly better than another has a cost. You are showing an inferior set to a segment of your users and that costs you money (or signups or whatever metric it is you're improving which, at the end of the day, presumably equates to money). As an example, suppose you do a test and discover something that doubles your signup rate (and therefore monetization rate) and you've got a confidence level of 99.9%. It's true, there's a 1 in 1,000 chance your result is flawed and you'll end up with the wrong decision. But there's a 999 in 1,000 that you're showing a significantly inferior signup page to half of your customers, costing you about 25% of potential revenue. It doesn't even take someone who knows what EV stands for to realize his EV on ending the test is huge here.