4 ms·

Most published research results are false

23 points by henning 16y ago

rubidium 16y ago
Note well. This is for clinical research. This says nothing about C.S., Math, Physics, etc... It's harder to publish a big result in Physics, for example, and have it stand for long without being questioned. Just ask Jan Hendrik Schön.
- johndcook 16y ago
  Math and CS are better off than experimental sciences. Theoretical paper often contain errors, but these don't usually invalidate the conclusions. In experimental sciences, hard sciences like physics do have a lower error rate than sciences that rely heavily on statistics. But that's changing. See the second page of this article: http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWModernScience.pdf http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWMode... The author argues that "Nature has gotten more tight-fisted with modern physicists" and that physics is now more dependent on statistics.
- logjam 16y ago
  > Note well. This is for clinical research. My own experience is that there is much constant questioning about published clinical research. In addition, decision-making of clinicians based on clinical research is tempered with other considerations, sometimes quite complex. Stephen Paulker commented in response to the original paper: "From the perspective of an epidemiologist or a statistician, the relevant question is whether the study's hypothesis is true - i.e., is the probability of H1 greater than 0.5. For clinicians and their patients, the relevant question is whether a particular strategy should be followed in an individual patient or a subset of similar patients. That decision (or recommendation to the patient) will depend on the pre-study likelihood of benefit in that patient and on the relative magnitude of benefits and risks of that strategy, if the diagnosis in that patient is uncertain. For many such decisions, the "more likely true than false" criterion may not be the best decision rule. For serious diseases and treatments of only modest risk, post-study probabilities of considerably less than 0.5 may be sufficient to justify treatment."
- Lewisham 16y ago
  From what I've seen, I wouldn't say CS is immune to this at all. CS seems to only be accepting papers that illustrate some big win. In my short PhD career, I've an article in the drawer where the result was inconclusive. This result, in and of itself, was fairly valuable. It spoke much about all the ways things went wrong. However, wrong was not acceptable for publication, it had to be right. Much hand-wringing was done to identify the bits that could be pulled out from it, polished, and then submitted (I was not especially pleased about this). The paper was, rightly, rejected, but all the valuable findings are still in my drawer. The pressure to produce Good findings rather than Bad findings does not meet my expectation of scientific pursuit. I took great joy in seeing a history of science exhibition recently, and reading the notebooks of people like Newton and Darwin, who they themselves spend a lot of time writing out all the things that went wrong, all the concerns they have about their "results", and general humbleness. It speaks volumes that the greatest scientific minds that have ever lived have put more caveats and concerns in their published work than 98% of what is published today does. To get published, it seems you have to be 110% sure that your work is 120% amazing. I don't think this is a good state of affairs.
jakestein 16y ago
Linke to the PLoS article that this post summarizes: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal...
j_baker 16y ago
Just out of curiosity, what is the alternative to using p-values? What should researchers really be doing?
- apl 16y ago
  Hypothesis testing isn't the fundamental problem, it's how a given p-value is perceived. Getting a so-called significant result simply shouldn't be confused with conclusive evidence for the hypothesis.
- Read_the_Genes 16y ago
  Most studies use raw p-values. This, coupled with "exploratory analyses" which are not mentioned in the research, leads to a high false discovery rate. Alternatives include using simulations to estimate false discovery rates (this can be done analytically for some problems). Bayesian frameworks can also be applied, depending on the amount and accuracy of prior knowledge of the system under study. That said, I see nothing wrong with publishing results with nominal p-values, as long as the researchers indicate the weakness in their results. Meta-analyses can always come back later and use the results they publish. What annoys me is seeing 50 or so statistical tests done, and then researchers stating they have found something when 1 of those tests shows a p-value of 0.05. Just using a Bonferroni correction, the simplest of all corrections, would demonstrate that findings like this are not significant.
- johndcook 16y ago
  Here's an article that makes the case that Bayes factors do a better job of summarizing the amount of evidence in favor of a conclusion: http://jnci.oxfordjournals.org/content/early/2010/12/03/jnci.djq493.full.pdf http://jnci.oxfordjournals.org/content/early/2010/12/03/jnci...
- amalcon 16y ago
  The problem isn't with the use of p-values, per se: the problem is that the human brain tends to distort probabilities on either end of the spectrum. People see that there's only a 5% chance that something could have happened by chance, and they put that in the "Not possible" bucket. It's a fundamental weakness of the human brain. There's not a whole lot we can do about it.
kenjackson 16y ago
Given that this proclamation was itself a published research study, does it mean that it is also probably wrong? The barber also publishes papers?
arctangent 16y ago
I always found statistics deeply disturbing when I was studying it at school. You can establish a null hypothesis and then test at some confidence interval (say, 99%) and find that your null hypothesis still holds. But, being somewhat ingenious, you may decide to lower the confidence interval (to, say, 98%) and find that your experimental evidence is now significant enough for you to reject your null hypothesis and accept your alternative hypothesis. Lies, damned lies, and statistics, indeed.
- brent 16y ago
  This is why different communities accept p-values at certain levels (typically <0.05). You have to live with some amount of uncertainty when the data generating mechanism is random... that is unfortunately the nature of the beast.
  - arctangent 16y ago
    I understand this. But let's say I ran some experiments and collected some data in an attempt to disprove theory X. What does it mean for me to say that at the 99% confidence level X is still true, but at the 98% confidence level it is not true? I just find it a bit spooky, is all.
s3graham 16y ago
I enjoyed this related article a few months ago on the problem: http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_...

23 points by henning 16y ago

Note well. This is for clinical research. This says nothing about C.S., Math, Physics, etc... It's harder to publish a big result in Physics, for example, and have it stand for long without being questioned. Just ask Jan Hendrik Schön.

johndcook 16y ago
Math and CS are better off than experimental sciences. Theoretical paper often contain errors, but these don't usually invalidate the conclusions. In experimental sciences, hard sciences like physics do have a lower error rate than sciences that rely heavily on statistics. But that's changing. See the second page of this article: http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWModernScience.pdf http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWMode... The author argues that "Nature has gotten more tight-fisted with modern physicists" and that physics is now more dependent on statistics.
logjam 16y ago
> Note well. This is for clinical research. My own experience is that there is much constant questioning about published clinical research. In addition, decision-making of clinicians based on clinical research is tempered with other considerations, sometimes quite complex. Stephen Paulker commented in response to the original paper: "From the perspective of an epidemiologist or a statistician, the relevant question is whether the study's hypothesis is true - i.e., is the probability of H1 greater than 0.5. For clinicians and their patients, the relevant question is whether a particular strategy should be followed in an individual patient or a subset of similar patients. That decision (or recommendation to the patient) will depend on the pre-study likelihood of benefit in that patient and on the relative magnitude of benefits and risks of that strategy, if the diagnosis in that patient is uncertain. For many such decisions, the "more likely true than false" criterion may not be the best decision rule. For serious diseases and treatments of only modest risk, post-study probabilities of considerably less than 0.5 may be sufficient to justify treatment."
Lewisham 16y ago
From what I've seen, I wouldn't say CS is immune to this at all. CS seems to only be accepting papers that illustrate some big win. In my short PhD career, I've an article in the drawer where the result was inconclusive. This result, in and of itself, was fairly valuable. It spoke much about all the ways things went wrong. However, wrong was not acceptable for publication, it had to be right. Much hand-wringing was done to identify the bits that could be pulled out from it, polished, and then submitted (I was not especially pleased about this). The paper was, rightly, rejected, but all the valuable findings are still in my drawer. The pressure to produce Good findings rather than Bad findings does not meet my expectation of scientific pursuit. I took great joy in seeing a history of science exhibition recently, and reading the notebooks of people like Newton and Darwin, who they themselves spend a lot of time writing out all the things that went wrong, all the concerns they have about their "results", and general humbleness. It speaks volumes that the greatest scientific minds that have ever lived have put more caveats and concerns in their published work than 98% of what is published today does. To get published, it seems you have to be 110% sure that your work is 120% amazing. I don't think this is a good state of affairs.

jakestein 16y ago

Linke to the PLoS article that this post summarizes: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal...

j_baker 16y ago

Just out of curiosity, what is the alternative to using p-values? What should researchers really be doing?

apl 16y ago
Hypothesis testing isn't the fundamental problem, it's how a given p-value is perceived. Getting a so-called significant result simply shouldn't be confused with conclusive evidence for the hypothesis.
Read_the_Genes 16y ago
Most studies use raw p-values. This, coupled with "exploratory analyses" which are not mentioned in the research, leads to a high false discovery rate. Alternatives include using simulations to estimate false discovery rates (this can be done analytically for some problems). Bayesian frameworks can also be applied, depending on the amount and accuracy of prior knowledge of the system under study. That said, I see nothing wrong with publishing results with nominal p-values, as long as the researchers indicate the weakness in their results. Meta-analyses can always come back later and use the results they publish. What annoys me is seeing 50 or so statistical tests done, and then researchers stating they have found something when 1 of those tests shows a p-value of 0.05. Just using a Bonferroni correction, the simplest of all corrections, would demonstrate that findings like this are not significant.
johndcook 16y ago
Here's an article that makes the case that Bayes factors do a better job of summarizing the amount of evidence in favor of a conclusion: http://jnci.oxfordjournals.org/content/early/2010/12/03/jnci.djq493.full.pdf http://jnci.oxfordjournals.org/content/early/2010/12/03/jnci...
amalcon 16y ago
The problem isn't with the use of p-values, per se: the problem is that the human brain tends to distort probabilities on either end of the spectrum. People see that there's only a 5% chance that something could have happened by chance, and they put that in the "Not possible" bucket. It's a fundamental weakness of the human brain. There's not a whole lot we can do about it.

kenjackson 16y ago

Given that this proclamation was itself a published research study, does it mean that it is also probably wrong? The barber also publishes papers?

arctangent 16y ago

I always found statistics deeply disturbing when I was studying it at school. You can establish a null hypothesis and then test at some confidence interval (say, 99%) and find that your null hypothesis still holds. But, being somewhat ingenious, you may decide to lower the confidence interval (to, say, 98%) and find that your experimental evidence is now significant enough for you to reject your null hypothesis and accept your alternative hypothesis. Lies, damned lies, and statistics, indeed.

brent 16y ago
This is why different communities accept p-values at certain levels (typically <0.05). You have to live with some amount of uncertainty when the data generating mechanism is random... that is unfortunately the nature of the beast.
- arctangent 16y ago
  I understand this. But let's say I ran some experiments and collected some data in an attempt to disprove theory X. What does it mean for me to say that at the 99% confidence level X is still true, but at the 98% confidence level it is not true? I just find it a bit spooky, is all.

s3graham 16y ago

I enjoyed this related article a few months ago on the problem: http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_...