3 ms·
Well, if there is no effect (the effect-size is zero), two different experimenters will likely see two different p-values regardless of how large an experiment
by krona 7y ago
Well, if there is no effect (the effect-size is zero), two different experimenters will likely see two different p-values regardless of how large an experiment either of them runs.
- mehrdadn 7y agoSo by this logic "it is completely consistent with the null hypothesis to see p-values of 0.00000001 and 0.99999999 from two replications of the same damn experiment"? At which point, what is even the point of this statement?
- lonelappde 7y agoThe point is that you need to replicate experiments and not cherry pick the one with the highest p-value.
- penagwin 7y agoMy understanding (Someone please correct me if I'm wrong) is that while the values can range from limx->0 and limx->1 , the actual distribution will be skewed in cases where it is statistically significant. E.g. If you repeat the experiment 100x you could find a p=.999999 but more likely they'll be close to .05 (or whatever alpha you choose)
- kgwgk 7y agoIf the null hypothesis is true the p-value is distributed uniformly in [0 1] (at least ignoring discrete data, composite hypothesis and other cases which are not easy). The distribution will be different when then true hypothesis is not true (but you may also get non-significant results even if the null hypothesis is not true). I’m not sure if that’s what you mean by “the actual distribution will be skewed in cases where it is statistically significant.”
- penagwin 7y agoSorry I was definitely kinda vague - I meant when the alternative hypothesis is true the distribution should be skewed towards 0.