4 ms·

Yes. Most scientific studies have a sample size too small to reliably detect the effect they're looking for, so they'd have to be very lucky to get a statistic

Yes. Most scientific studies have a sample size too small to reliably detect the effect they're looking for, so they'd have to be very lucky to get a statistically significant result even if there's a real effect. There's just not enough data to distinguish signal from noise. So if you see a paper that says "We ran ten different experiments and every one was statistically significant, proving we were right", you should be less convinced, because this suggests they cheated. Even if all ten experimental hypotheses were right, there's a very small chance they would obtain statistically significant results for all of them. This suggests they conducted other unsuccessful experiments but did not report their results, or that they chose their hypotheses after running the experiments. There's a psychologist, Gregory Francis, who has published a series of papers investigating this. He looks at a series of results published on some phenomenon, calculates how many you'd expect to be statistically significant if the effect genuinely exists, and then shows that this number was greatly exceeded. This suggests there is publication bias. Of course, Francis got into hot water by selectively publishing the results of his studies into publication bias, only reporting a result if he found evidence of bias. So there's a bit of irony in it all.

judk 12y ago
It's only suspect if they claim 10/10 successes with p value of 0.1, right?
- gwern 12y ago
  No, it's not quite that simple: there's a three-way tradeoff between _n_, power, and _p_ - you can tradeoff one to save on another. So the 10/10 will be suspicious depending on the specific values of _n_, power, and _p_. However, usually psychology will always use a p<0.05, so that variable is fixed in advance, and they almost never use big enough _n_ for funding & convenience so you can safely assume _n_ will be 'too small'; with those two variable set, that implies that usually the power will be very small, like 10-50%/0.1-0.5, and then 10/10 would indeed be very suspicious (0.5^10 = a small number; if you flip a fair coin 10 times, it's not often you'll get 10 heads).
gfrancis 12y ago
Nice summary. I wanted to clarify one point. Selective reporting (of publication bias or most other kinds of tests) limits the inferences that can be drawn from the analyses, but it does not necessarily invalidate the properties of the test itself. If my selective reporting indicates that 6 out of 7 articles appear to be biased, it would be improper to infer that 86% of psychology articles are biased. Nevertheless, those 6 articles appear biased whether you make such an inference or not. I think scientists care about the validity of specific data/theory more than about the rate of bias across a field. However, if you want an estimate of the latter, you can look at a paper I published that systematically investigated bias in the journal Psychological Science: http://link.springer.com/article/10.3758%2Fs13423-014-0601-x http://link.springer.com/article/10.3758%2Fs13423-014-0601-x -Greg Francis
- capnrefsmmat 12y ago
  I'm not sure I completely buy that argument. In the extreme case, you could test 100 articles and find p < 0.05 evidence that 5 of them are biased. If you reported only on those 5, we would be misled, because it's exactly the result we'd expect under the null. I think this is a point Simonsohn made when critiquing your papers, although I didn't follow the discussion after that. Thanks for the link, though; that's exactly the kind of analysis I was hoping to see. I'll have to read through it.
  - gfrancis 12y ago
    If I selectively reported only 5 out of 100 analyses, you would only be misled if you inferred that a high percentage of articles were biased. I am specifically warning you not to make such an inference because the selective reporting invalidates it. Perhaps those 5 articles that failed the analysis are actually unbiased and were (un)lucky in repeatedly rejecting the null with seemingly low-powered experiments. That would be a kind of Type I error regarding the conclusion of bias. But such errors are inevitable when making decisions in a stochastic setting. The findings in those articles _do_ appear odd, so scientists _should_ be skeptical about their conclusions. To do otherwise is to throw out the basic ideas of hypothesis testing (in which case you are skeptical about all of the articles rather than just the 5 odd ones). In one sense, I think you are worrying about prior probabilities. Your concern is that if some of the articles actually generated data or theorised in a unbiased way, then we have to worry about the Type I errors. You can reduce the occurrence of such errors by using a more stringent criterion, but you can never remove them entirely (and the 0.1 criterion commonly used for tests of publication bias seems very conservative, the Type I error rate is close to 0.01 for simulated cases). Either we accept that we will sometimes make errors, or we do not make decisions. Curiously, the publication bias analyses have also been criticised for exactly the opposite prior: that all articles are biased so there is nothing learned when the analysis concludes bias. (Simonsohn makes both arguments, even though they contradict each other.) I think this prior is unjustified. Publication bias seems to be a common problem, but I think it is not warranted to assume that experiments are biased by default. If that negative view is appropriate, then we should all close shop (and funding agencies should stop supporting such flawed research). My view is more pragmatic. There are problems with many experimental studies, but some work appears quite solid. We should make an effort to distinguish between them.

judk 12y ago

It's only suspect if they claim 10/10 successes with p value of 0.1, right?

gwern 12y ago
No, it's not quite that simple: there's a three-way tradeoff between _n_, power, and _p_ - you can tradeoff one to save on another. So the 10/10 will be suspicious depending on the specific values of _n_, power, and _p_. However, usually psychology will always use a p<0.05, so that variable is fixed in advance, and they almost never use big enough _n_ for funding & convenience so you can safely assume _n_ will be 'too small'; with those two variable set, that implies that usually the power will be very small, like 10-50%/0.1-0.5, and then 10/10 would indeed be very suspicious (0.5^10 = a small number; if you flip a fair coin 10 times, it's not often you'll get 10 heads).

gfrancis 12y ago

Nice summary. I wanted to clarify one point. Selective reporting (of publication bias or most other kinds of tests) limits the inferences that can be drawn from the analyses, but it does not necessarily invalidate the properties of the test itself. If my selective reporting indicates that 6 out of 7 articles appear to be biased, it would be improper to infer that 86% of psychology articles are biased. Nevertheless, those 6 articles appear biased whether you make such an inference or not. I think scientists care about the validity of specific data/theory more than about the rate of bias across a field. However, if you want an estimate of the latter, you can look at a paper I published that systematically investigated bias in the journal Psychological Science: http://link.springer.com/article/10.3758%2Fs13423-014-0601-x http://link.springer.com/article/10.3758%2Fs13423-014-0601-x -Greg Francis

capnrefsmmat 12y ago
I'm not sure I completely buy that argument. In the extreme case, you could test 100 articles and find p < 0.05 evidence that 5 of them are biased. If you reported only on those 5, we would be misled, because it's exactly the result we'd expect under the null. I think this is a point Simonsohn made when critiquing your papers, although I didn't follow the discussion after that. Thanks for the link, though; that's exactly the kind of analysis I was hoping to see. I'll have to read through it.
- gfrancis 12y ago
  If I selectively reported only 5 out of 100 analyses, you would only be misled if you inferred that a high percentage of articles were biased. I am specifically warning you not to make such an inference because the selective reporting invalidates it. Perhaps those 5 articles that failed the analysis are actually unbiased and were (un)lucky in repeatedly rejecting the null with seemingly low-powered experiments. That would be a kind of Type I error regarding the conclusion of bias. But such errors are inevitable when making decisions in a stochastic setting. The findings in those articles _do_ appear odd, so scientists _should_ be skeptical about their conclusions. To do otherwise is to throw out the basic ideas of hypothesis testing (in which case you are skeptical about all of the articles rather than just the 5 odd ones). In one sense, I think you are worrying about prior probabilities. Your concern is that if some of the articles actually generated data or theorised in a unbiased way, then we have to worry about the Type I errors. You can reduce the occurrence of such errors by using a more stringent criterion, but you can never remove them entirely (and the 0.1 criterion commonly used for tests of publication bias seems very conservative, the Type I error rate is close to 0.01 for simulated cases). Either we accept that we will sometimes make errors, or we do not make decisions. Curiously, the publication bias analyses have also been criticised for exactly the opposite prior: that all articles are biased so there is nothing learned when the analysis concludes bias. (Simonsohn makes both arguments, even though they contradict each other.) I think this prior is unjustified. Publication bias seems to be a common problem, but I think it is not warranted to assume that experiments are biased by default. If that negative view is appropriate, then we should all close shop (and funding agencies should stop supporting such flawed research). My view is more pragmatic. There are problems with many experimental studies, but some work appears quite solid. We should make an effort to distinguish between them.