4 ms·

I'm not sure I completely buy that argument. In the extreme case, you could test 100 articles and find p < 0.05 evidence that 5 of them are biased. If you repor

I'm not sure I completely buy that argument. In the extreme case, you could test 100 articles and find p < 0.05 evidence that 5 of them are biased. If you reported only on those 5, we would be misled, because it's exactly the result we'd expect under the null. I think this is a point Simonsohn made when critiquing your papers, although I didn't follow the discussion after that. Thanks for the link, though; that's exactly the kind of analysis I was hoping to see. I'll have to read through it.

gfrancis 12y ago
If I selectively reported only 5 out of 100 analyses, you would only be misled if you inferred that a high percentage of articles were biased. I am specifically warning you not to make such an inference because the selective reporting invalidates it. Perhaps those 5 articles that failed the analysis are actually unbiased and were (un)lucky in repeatedly rejecting the null with seemingly low-powered experiments. That would be a kind of Type I error regarding the conclusion of bias. But such errors are inevitable when making decisions in a stochastic setting. The findings in those articles _do_ appear odd, so scientists _should_ be skeptical about their conclusions. To do otherwise is to throw out the basic ideas of hypothesis testing (in which case you are skeptical about all of the articles rather than just the 5 odd ones). In one sense, I think you are worrying about prior probabilities. Your concern is that if some of the articles actually generated data or theorised in a unbiased way, then we have to worry about the Type I errors. You can reduce the occurrence of such errors by using a more stringent criterion, but you can never remove them entirely (and the 0.1 criterion commonly used for tests of publication bias seems very conservative, the Type I error rate is close to 0.01 for simulated cases). Either we accept that we will sometimes make errors, or we do not make decisions. Curiously, the publication bias analyses have also been criticised for exactly the opposite prior: that all articles are biased so there is nothing learned when the analysis concludes bias. (Simonsohn makes both arguments, even though they contradict each other.) I think this prior is unjustified. Publication bias seems to be a common problem, but I think it is not warranted to assume that experiments are biased by default. If that negative view is appropriate, then we should all close shop (and funding agencies should stop supporting such flawed research). My view is more pragmatic. There are problems with many experimental studies, but some work appears quite solid. We should make an effort to distinguish between them.
- capnrefsmmat 12y ago
  Another way of putting the argument is that, if the base rate of publication bias is low, your articles will have a very high false discovery rate. That's inevitable, yes, but it means I would not treat your individual bias results as particularly strong. But the survey of Psychological Science papers makes this point moot, since it shows that the base rate is fairly high. It would be interesting to see if there are other approaches to detecting publication bias. In medicine several reviews have spotted outcome reporting bias by comparing clinical trial protocols given to ethics committees with the published papers. Typically the clinical trial protocol includes several outcomes that aren't in the paper. Other reviews search clinical trial registries to find publication bias. Is there something analogous in psychology? You could presumably track down ethics board documents and figure out what fraction of experiments never appear, then do a sensitivity analysis on the published results. I don't know if this has been done. I assume you're familiar with this survey that found about 50% of psychologists admit to publication bias: John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. doi:10.1177/0956797611430953
  - gfrancis 12y ago
    If you feel that the conclusion of the individual bias results are not convincing, then you probably should not trust the estimated base rate from the analysis of the Psych Science articles. I think the results are convincing, and I am not sure why you say there will be a high false discovery rate (false discovery means concluding bias when it does not exist - a Type I error). If bias does not exist, then (in the ideal case where true experimental power is known) the Type I error rate is 0.1. When power is estimated from the data, the Type I error rate is generally much smaller than 0.1. As for a conclusion of bias being "strong", I think we have to define the term. Certainly I would not suggest that a conclusion of bias means that authors are being dishonest or stupid. There are many ways for published results to become biased. However, I think the appearance of bias does mean that scientists should be skeptical about the validity of the reported results. The impetus is on the original authors to provide good support for their theoretical ideas, and biased data generally do not provide such support. The closest example I know of that relates to your description of contrasting registries and published results is a comparison of theses and subsequent journal articles. Details are at http://jom.sagepub.com/content/early/2014/03/18/0149206314527133.full http://jom.sagepub.com/content/early/2014/03/18/014920631452...

gfrancis 12y ago

If I selectively reported only 5 out of 100 analyses, you would only be misled if you inferred that a high percentage of articles were biased. I am specifically warning you not to make such an inference because the selective reporting invalidates it. Perhaps those 5 articles that failed the analysis are actually unbiased and were (un)lucky in repeatedly rejecting the null with seemingly low-powered experiments. That would be a kind of Type I error regarding the conclusion of bias. But such errors are inevitable when making decisions in a stochastic setting. The findings in those articles _do_ appear odd, so scientists _should_ be skeptical about their conclusions. To do otherwise is to throw out the basic ideas of hypothesis testing (in which case you are skeptical about all of the articles rather than just the 5 odd ones). In one sense, I think you are worrying about prior probabilities. Your concern is that if some of the articles actually generated data or theorised in a unbiased way, then we have to worry about the Type I errors. You can reduce the occurrence of such errors by using a more stringent criterion, but you can never remove them entirely (and the 0.1 criterion commonly used for tests of publication bias seems very conservative, the Type I error rate is close to 0.01 for simulated cases). Either we accept that we will sometimes make errors, or we do not make decisions. Curiously, the publication bias analyses have also been criticised for exactly the opposite prior: that all articles are biased so there is nothing learned when the analysis concludes bias. (Simonsohn makes both arguments, even though they contradict each other.) I think this prior is unjustified. Publication bias seems to be a common problem, but I think it is not warranted to assume that experiments are biased by default. If that negative view is appropriate, then we should all close shop (and funding agencies should stop supporting such flawed research). My view is more pragmatic. There are problems with many experimental studies, but some work appears quite solid. We should make an effort to distinguish between them.

capnrefsmmat 12y ago
Another way of putting the argument is that, if the base rate of publication bias is low, your articles will have a very high false discovery rate. That's inevitable, yes, but it means I would not treat your individual bias results as particularly strong. But the survey of Psychological Science papers makes this point moot, since it shows that the base rate is fairly high. It would be interesting to see if there are other approaches to detecting publication bias. In medicine several reviews have spotted outcome reporting bias by comparing clinical trial protocols given to ethics committees with the published papers. Typically the clinical trial protocol includes several outcomes that aren't in the paper. Other reviews search clinical trial registries to find publication bias. Is there something analogous in psychology? You could presumably track down ethics board documents and figure out what fraction of experiments never appear, then do a sensitivity analysis on the published results. I don't know if this has been done. I assume you're familiar with this survey that found about 50% of psychologists admit to publication bias: John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. doi:10.1177/0956797611430953
- gfrancis 12y ago
  If you feel that the conclusion of the individual bias results are not convincing, then you probably should not trust the estimated base rate from the analysis of the Psych Science articles. I think the results are convincing, and I am not sure why you say there will be a high false discovery rate (false discovery means concluding bias when it does not exist - a Type I error). If bias does not exist, then (in the ideal case where true experimental power is known) the Type I error rate is 0.1. When power is estimated from the data, the Type I error rate is generally much smaller than 0.1. As for a conclusion of bias being "strong", I think we have to define the term. Certainly I would not suggest that a conclusion of bias means that authors are being dishonest or stupid. There are many ways for published results to become biased. However, I think the appearance of bias does mean that scientists should be skeptical about the validity of the reported results. The impetus is on the original authors to provide good support for their theoretical ideas, and biased data generally do not provide such support. The closest example I know of that relates to your description of contrasting registries and published results is a comparison of theses and subsequent journal articles. Details are at http://jom.sagepub.com/content/early/2014/03/18/0149206314527133.full http://jom.sagepub.com/content/early/2014/03/18/014920631452...