4 ms·
Isn’t a main problem with p-values that you don’t know whether significance (low p-value) is a result of big effect and small sample or big sample and small eff
by rndn 11y ago
Isn’t a main problem with p-values that you don’t know whether significance (low p-value) is a result of big effect and small sample or big sample and small effect. This is why you also need a measure for the effect, for example the distance of the two measurements in terms of standard derivations.
- james1071 11y agoThat is a separate issue. The main problem with p-values is that, without further information, one cannot infer from them how likely it is that a result is genuine.
- jjoonathan 11y agoI agree with TFA that p-hacking is a bigger problem. Low p-value <=> null hypothesis is unlikely. Choose a shitty null hypothesis ("aliens did it!", "everything is Gaussian", etc) and you trivially get low p. Peer review checks this to some extent (you won't get away with "aliens did it") but there's a large gray area of null hypotheses shitty enough to give low p but not shitty enough to be rejected by peer review. Choosing the hypothesis after-the-fact is the most common strategy because it's undetectable except by repeating the experiment, which is hard.