What's the threshold of "affording to be wrong?" A/B testing is a tool. It gives you data, not answers. Would I make a color change off of 100 data points and

What's the threshold of "affording to be wrong?" A/B testing is a tool. It gives you data, not answers. Would I make a color change off of 100 data points and a p value of 0.05? Well, multiply the risk of being wrong against the cost of being wrong. A 1 in 20 chance of being wrong isn't terrible when the cost of being wrong is low. Would I make sweeping changes to my sales funnel at n=100 and p=0.05? Maybe not. n=10000 and p=0.001? Slam dunk yes. (And you should have probably halted the test earlier... imagine all the potential customers who got the "bad" leg and bounced?) If results are currently at n=5000 and p=0.06001, and it's going to take 6 more months to hit n=10000, and you've only got 5 months of runway... Well then, give up on that test and do something else. Opportunity cost is a cost too! At the end of the day, it's up to you to make an informed business decision using your best judgement. "Bounded rationality." All decisions are made with imperfect information and under time pressure. Like in TFA, if your chargeback rate spikes after a change, is it because the change was bad, or because you're expanding out of early adopters and into choosier (more annoying, more normal) customers? Or it's just a random Thursday and your sales volume is so low that every day is spiky and weird? As @joelrunyon notes above, in the very earliest days, when your total sales number in the double digits, no A/B test in the world is going to be powered enough, short of "normal landing page/404 error". You just have to be good at your job, and lucky.