4 ms·
Not necessarily 5%, no. The p-value is controlling the false-positive rate, i.e. it constrains the likelihood that a seemingly-significant result was really jus
by mjn 13y ago
Not necessarily 5%, no. The p-value is controlling the false-positive rate, i.e. it constrains the likelihood that a seemingly-significant result was really just by chance. Therefore, if we were now given the real ground truth, it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all.
But you seem to be positing the reverse: a hypothetical case where we assume that the results of the original studies were in fact correct (i.e. ground truth), and we want a sort of false-negative rate, the chance that a fact would not be confirmed by a replication study, despite actually being true. That depends on several things about the replication attempt, such as sample size and methodology, summarized into the concept of statistical power: http://en.wikipedia.org/wiki/Statistical_power http://en.wikipedia.org/wiki/Statistical_power
The overall failure-to-replicate rate would involve both aspects, with four possible outcomes at varying likelihood: 1) a correct result that was confirmed; 2) a correct result that failed to be confirmed; 3) an incorrect result that was confirmed anyway; and 4) an incorrect result that failed to be confirmed. Obviously it would be ideal if #1 and #4 were much higher than #2 and #3.
- gabemart 13y agoThanks, very interesting. It was silly of me to assume that the p-value could be used in that way when, as you explain, it's not measuring or indicating that at all.
- gwern 13y agoDon't feel too bad. It's a very common mistake; see some of the citations in https://en.wikipedia.org/wiki/P-value#Misunderstandings https://en.wikipedia.org/wiki/P-value#Misunderstandings and http://lesswrong.com/lw/g13/against_nhst/ http://lesswrong.com/lw/g13/against_nhst/
- yummyfajitas 13y agoit's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all. Not quite. Assume P of the tests are truly positive, and N are truly negative. We expect to see approximately 0.05 x N + A x P positives in the entire sample, where A is the probability of a false negative. So the fraction of true positives not replicated is likely to be 0.05 x N / (0.05 x N + A x P).
- deleted 13y ago[deleted]
- gwern 13y ago> That depends on several things about the replication attempt, such as sample size and methodology, summarized into the concept of statistical power: http://en.wikipedia.org/wiki/Statistical_power http://en.wikipedia.org/wiki/Statistical_power It's not just power, you also need prior odds if you want to make an unconditional estimate. This is pretty much the core of Ioannides's famous paper http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal... - even after taking power into account, the replication rate could be anywhere from 0 to say 80% depending on the prior odds.