2σ is fine, but the benefit of modern technology is that we can tell exactly what standard deviation would be needed for the null hypothesis to randomly generat

2σ is fine, but the benefit of modern technology is that we can tell exactly what standard deviation would be needed for the null hypothesis to randomly generate our results. Particle physics holds itself to an "industry standard" of 5 sigma, for example. The real conversation to be had is -- what standard deviation will we tolerate? Is this something we'll keep doing, and thus A/B test ourselves into a (perhaps quite horrifying) local minimum based on random noise? Is this a single great experiment? Are lives on the line? Will these results be taken Quite Seriously? Is this a test to say "look, this is worth further investigation"? I'm no statistician but in my opinion this is the first conversation that needs to be had when doing an experiment: what p / σ levels are satisfactory to claim confidence? p<0.05 is a decent heuristic for some experiments, but not all.