4 ms·

The false discovery rate isn't the same as the type I error rate. It's larger. If, for example, 10% of articles have publication bias, your test has 100% power

The false discovery rate isn't the same as the type I error rate. It's larger. If, for example, 10% of articles have publication bias, your test has 100% power to detect these, and the type I rate is 10%, your false discovery rate will be 47%. (That is, if you tested 100 papers, you'd get 10 true positives and 9 false positives, so 9 / 19 = 47%.) So in that ideal scenario, about half of your published findings of bias would be false positives. This means that your analysis of Psychological Science would overestimate the base rate by a factor of 2. Of course, the overestimation amount depends on the true rate, which we don't know. Now, you say the true type I rate is much smaller, so perhaps that solves some of the problem. But I doubt you detect bias with 100% power, and smaller power means a higher false discovery rate. Thanks for the link; that looks like what I was hoping for. It'd be interesting to see something similar for a broader sample of papers (not just those started as theses).

gfrancis 12y ago
Thanks for the clarification about the FDR. Obviously, its calculation very much depends on the base rate probabilities and the sensitivity of the method. The method's ability to detect bias very much depends on the type of bias being applied. When only a file-drawer bias is being used the method is unlikely to detect the presence of bias (because the power values are dramatically overestimated). On the other hand, if optional stopping and a file drawer are used, the method has a good chance of detecting bias (because all the reported experiments stop just as soon as they satisfy the p<.05 criterion). Ioannidis used some of my simulation results to calculate likelihood ratios, which are somewhat related to the FDR. Details are at http://www.sciencedirect.com/science/article/pii/S0022249613000278 http://www.sciencedirect.com/science/article/pii/S0022249613...

gfrancis 12y ago

Thanks for the clarification about the FDR. Obviously, its calculation very much depends on the base rate probabilities and the sensitivity of the method. The method's ability to detect bias very much depends on the type of bias being applied. When only a file-drawer bias is being used the method is unlikely to detect the presence of bias (because the power values are dramatically overestimated). On the other hand, if optional stopping and a file drawer are used, the method has a good chance of detecting bias (because all the reported experiments stop just as soon as they satisfy the p<.05 criterion). Ioannidis used some of my simulation results to calculate likelihood ratios, which are somewhat related to the FDR. Details are at http://www.sciencedirect.com/science/article/pii/S0022249613000278 http://www.sciencedirect.com/science/article/pii/S0022249613...