This is exactly the reason when working with my team on A/B testing, I'm always careful to use the phrase _meaningful_ difference (or not). With internet-based

This is exactly the reason when working with my team on A/B testing, I'm always careful to use the phrase _meaningful_ difference (or not). With internet-based tests, the volumes of participants can be huge, so finding statistical significance is a hell of a lot easier than finding a meaningful difference. I like to say, "As N tends to infinity, we are guaranteed to find significance." The sad thing is that there are more testing platforms than I can count on one hand that introduce this p-value fallacy to users. They encourage things like repeatedly checking tests. I don't believe that users realize that the true p-value is oscillating around alpha. If your test isn't significant at them moment, just check a few hours later (it likely will be). Even with Bayesian methods, I've learned the hard way that you really have to be patient and let certainty accrue. More and more I'm lead towards Bandit methods for this reason. When sampling real-time data, you're in effect treating a historical sample as a part population with one part in the future --- that's a pretty dangerous assumption. My solution has been to guide my team towards large tests, work to _really_ move the needle (and have high statistical power). This is where statistical analysis has the best chance of providing certainty. In generally, with web experiments it's best to proceed with a healthy amount of humility.