10 ms·

If you feel that the conclusion of the individual bias results are not convincing, then you probably should not trust the estimated base rate from the analysis

If you feel that the conclusion of the individual bias results are not convincing, then you probably should not trust the estimated base rate from the analysis of the Psych Science articles. I think the results are convincing, and I am not sure why you say there will be a high false discovery rate (false discovery means concluding bias when it does not exist - a Type I error). If bias does not exist, then (in the ideal case where true experimental power is known) the Type I error rate is 0.1. When power is estimated from the data, the Type I error rate is generally much smaller than 0.1. As for a conclusion of bias being "strong", I think we have to define the term. Certainly I would not suggest that a conclusion of bias means that authors are being dishonest or stupid. There are many ways for published results to become biased. However, I think the appearance of bias does mean that scientists should be skeptical about the validity of the reported results. The impetus is on the original authors to provide good support for their theoretical ideas, and biased data generally do not provide such support. The closest example I know of that relates to your description of contrasting registries and published results is a comparison of theses and subsequent journal articles. Details are at http://jom.sagepub.com/content/early/2014/03/18/0149206314527133.full http://jom.sagepub.com/content/early/2014/03/18/014920631452...

capnrefsmmat 12y ago
The false discovery rate isn't the same as the type I error rate. It's larger. If, for example, 10% of articles have publication bias, your test has 100% power to detect these, and the type I rate is 10%, your false discovery rate will be 47%. (That is, if you tested 100 papers, you'd get 10 true positives and 9 false positives, so 9 / 19 = 47%.) So in that ideal scenario, about half of your published findings of bias would be false positives. This means that your analysis of Psychological Science would overestimate the base rate by a factor of 2. Of course, the overestimation amount depends on the true rate, which we don't know. Now, you say the true type I rate is much smaller, so perhaps that solves some of the problem. But I doubt you detect bias with 100% power, and smaller power means a higher false discovery rate. Thanks for the link; that looks like what I was hoping for. It'd be interesting to see something similar for a broader sample of papers (not just those started as theses).
- gfrancis 12y ago
  Thanks for the clarification about the FDR. Obviously, its calculation very much depends on the base rate probabilities and the sensitivity of the method. The method's ability to detect bias very much depends on the type of bias being applied. When only a file-drawer bias is being used the method is unlikely to detect the presence of bias (because the power values are dramatically overestimated). On the other hand, if optional stopping and a file drawer are used, the method has a good chance of detecting bias (because all the reported experiments stop just as soon as they satisfy the p<.05 criterion). Ioannidis used some of my simulation results to calculate likelihood ratios, which are somewhat related to the FDR. Details are at http://www.sciencedirect.com/science/article/pii/S0022249613000278 http://www.sciencedirect.com/science/article/pii/S0022249613...

capnrefsmmat 12y ago

The false discovery rate isn't the same as the type I error rate. It's larger. If, for example, 10% of articles have publication bias, your test has 100% power to detect these, and the type I rate is 10%, your false discovery rate will be 47%. (That is, if you tested 100 papers, you'd get 10 true positives and 9 false positives, so 9 / 19 = 47%.) So in that ideal scenario, about half of your published findings of bias would be false positives. This means that your analysis of Psychological Science would overestimate the base rate by a factor of 2. Of course, the overestimation amount depends on the true rate, which we don't know. Now, you say the true type I rate is much smaller, so perhaps that solves some of the problem. But I doubt you detect bias with 100% power, and smaller power means a higher false discovery rate. Thanks for the link; that looks like what I was hoping for. It'd be interesting to see something similar for a broader sample of papers (not just those started as theses).

gfrancis 12y ago
Thanks for the clarification about the FDR. Obviously, its calculation very much depends on the base rate probabilities and the sensitivity of the method. The method's ability to detect bias very much depends on the type of bias being applied. When only a file-drawer bias is being used the method is unlikely to detect the presence of bias (because the power values are dramatically overestimated). On the other hand, if optional stopping and a file drawer are used, the method has a good chance of detecting bias (because all the reported experiments stop just as soon as they satisfy the p<.05 criterion). Ioannidis used some of my simulation results to calculate likelihood ratios, which are somewhat related to the FDR. Details are at http://www.sciencedirect.com/science/article/pii/S0022249613000278 http://www.sciencedirect.com/science/article/pii/S0022249613...