>'My go-to phrase: "Even if there were no real difference, we would see a result like this X% of the time anyway."' Hmm, p-values are data-dependent random var

>'My go-to phrase: "Even if there were no real difference, we would see a result like this X% of the time anyway."' Hmm, p-values are data-dependent random variables. Do this simple experiment in R where the null hypothesis of no difference is as true as can be: replicate(2,t.test(rnorm(10),rnorm(10))$p.value) For example, I got: [1] 0.8446991 0.3935238 So, the p-value can vary wildly (actually will follow a uniform(0,1) distribution) even if there were no real difference. Experimenter A would tell you that "we would see a result like this 84% of the time anyway" and experimenter B would say 39%. If enough experiments are performed, eventually someone will calculate a very small p-value. So which experimenter should we listen to? Which experimenters should bother sharing their results? All are correct. In other words, what should be done with that information about "even if there were no difference"? How should it affect our behavior, decisions, and beliefs?