3 ms·
Not quite. That's the common misinterpretation of p-values as probabilities that a study is correct. The p-value isn't the probability of the hypothesis being
by MereInterest 5y ago
Not quite. That's the common misinterpretation of p-values as probabilities that a study is correct. The p-value isn't the probability of the hypothesis being wrong given the data. It is instead the probability of the data given the hypothesis being wrong.
In order to determine what proportion of studies would fail to reproduce, you would also need to know the number of correct hypotheses in the first place. If 100 correct hypotheses and 900 incorrect hypotheses are tested, your sample of "p<0.05" results would contain 95 correct hypotheses and 45 incorrect hypotheses, so about 1/3 of the results would fail to replicate.
This is why there is such a large push for prepublishing of analysis methods, hypotheses, and experimental design, and later publishing of negative results. The initial distribution of hypotheses is important to know how to interpret the resulting body of work.