With a big enough N, the probability of reaching significance approaches 1. Well, not with an A vs A experiment. Then the probability of significance at t

With a big enough N, the probability of reaching significance approaches 1. Well, not with an A vs A experiment. Then the probability of significance at the p < 0.05 level should stay 5%. On the other hand, in an A vs B experiment where B is very slightly different from A you're right that with a large enough population you become very likely to reach a p < 0.05 significance level, just with an extremely tiny effect size. The experiment has told you something about the difference between A and B, and you now have high certainty, but it's just such a small difference that you probably don't care. Have you ever approached a tool in the field that would fail my proposed test? Your proposed test being, run an A vs A experiment and a tool is faulty if it concludes there's a difference significant at the 0.05 level more than 5% of the time? All the tools I've used for A/B testing would do fine on this test. This includes a custom tool at a computational advertising startup I worked at, Google's internal experiment platform, and Google Analytics' Content Experiments. (If I didn't trust a tool to pass this test I'd just use it to get the counts for both sides of the experiment and do that stats on my own.)