7 ms·

Why most published research findings are false (2005)

157 points by Jimmc414 4y ago

T3RMINATED 4y ago
realaleris149 4y ago
including this one?
- planetsprite 4y ago
  predictable joke
  - Sakos 4y ago
    Is it a joke?
fastaguy88 4y ago
This was a very controversial paper when it was published, perhaps because of its incendiary title. But the paper is much more subtle than the title suggests. Basically the idea is that if you try to test phenomena that are completely unexpected, your prior odds are low, so even if you get a positive result, there is a good chance the result is incorrect. So there is a danger that by trying to ensure more correct results, scientists may avoid paradigm changing experiments, which would be a loss. Follow up analysis by Jager and Leek (2014) https://academic.oup.com/biostatistics/article/15/1/1/244509 https://academic.oup.com/biostatistics/article/15/1/1/244509 suggests the false discovery rate is closer to 14% than 50%.
- themitigating 4y ago
  That's a huge difference yet anti-intellectuals will contiune to post this paper until the end of time as proof that you can't trust science
  - diognesofsinope 4y ago
    I mean, the failure of a lot of experiments to replicate is extraordinarily well documented... Not to mention the corollaries from the Wikipedia article (https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...): "In addition to the main result, Ioannidis lists six corollaries for factors that can influence the reliability of published research. Research findings in a scientific field are less likely to be true, 1. the smaller the studies conducted. 2. the smaller the effect sizes. 3. the greater the number and the lesser the selection of tested relationships. 4. the greater the flexibility in designs, definitions, outcomes, and analytical modes. 5. the greater the financial and other interests and prejudices. 6. the hotter the scientific field (with more scientific teams involved)." These are all reasonable criticisms from my own area of expertise: applied econometrics.
    - nonameiguess 4y ago
      Being skeptical of cutting edge research is a lot different than distrusting science. There are plenty of people out there denying special relativity, evolution by natural selection, or believing all of western medicine is invalid, based on extrapolation from stuff like this. I won't speak for all textbooks, but generally stuff you find in there should not be the same as what you find in journals, and is much more settled. Big caveat that that isn't necessarily true for younger sciences without long-established theory, say exercise physiology or social psychology, but something like a chemistry textbook is pretty damn trustworthy. And those are what people who aren't actually scientists should mostly be educating themselves with, not newspaper science reporting sections.
      - epolanski 4y ago
        Doubting a scientific paper results is different from distrusting science. Academia is a complex place and it's full of fake results to get publications. (I'm coauthor of several scientific papers and I know for a fact countless highly rated papers in my former field are not reproducible because they come from adjusted numbers).
  - deleted 4y ago
- glofish 4y ago
  This followup paper seems to summarize p-values reported in 77K papers and uses the distribution of these p-values to compute the FDR. Alas everyone knows that the p-values reported in accepted (!) papers are questionable at best - any analysis that uses them is on shaky foundation. The misuse of p-values in science is well known and an endemic problem - so what do we learn from a re-analysis of made up numbers?
  - Manu40 4y ago
    Not to be pedantic, but considering how many people I have run into who take all research studies as if they are holy gospel; you may want to reconsider saying "everyone knows". Just my 5 cents on the matter.
    - glofish 4y ago
      good point, what I really meant is by now everybody should know that p-values are questionable at best
      - genewitch 4y ago
        can we name one mainstream journalist, or any policy makers that know anything about reproducibility - or, more honestly, the appalling lack thereof - or "p-values are questionable at best"? There's been books, such as "how to lie with statistics" that should be mandatory reading, at least on entry to adulthood. I am one of maybe 3 people i know personally that knows how to actually parse a scientific paper, so when someone starts throwing a half dozen studies at me to back up their point, i can usually - on the scale of quarter hours - nitpick their understanding of the sources they used. This doesn't win anyone friends, and something i've said for nearly 3 decades: Religion and science like to point fingers, but religion seems more likely to bend and change than science. Every time i mention this or an analogue i'm shouted at that it's "actually the opposite". Shut up already, it's science. The science is in. This is science!
        Manu40 4y ago
- 6gvONxR4sf7o 4y ago
  I'd love to know the correlation between failure to replicate and public virality of findings. I wouldn't be surprised of the more exciting findings replicate at a different rate from average.
  - H8crilA 4y ago
    That's kind of the point. More exciting statements are more often "proven" because they're more often attempted to be proven.
- nextos 4y ago
  A good way to address these issues is to frame all experiments as multilevel models. See [1] for a long discussion from Andrew Gelman et al on why this is advisable. Surprisingly, many people working in statistics still ignore the James-Stein theorem, which provides a theoretical justification for multilevel models. In layman terms, said theorem shows that if you are simultaneously estimating many random variables you should borrow information across variables [2]. Estimating them one by one is suboptimal and does not minimize the global mean squared error. Multilevel models "shrink" individual effect sizes by looking at the overall distribution of effect sizes and provide much more realistic estimates. [1] https://arxiv.org/abs/0907.2478 https://arxiv.org/abs/0907.2478 [2] https://en.wikipedia.org/wiki/Stein%27s_example https://en.wikipedia.org/wiki/Stein%27s_example
  - kimukasetsu 4y ago
    This. Multilevel models relax the assumption of independent observations by specifying that the measures of repeated experimental units are dependent on each other. It's a way of telling your model that it has less information than it would have if all observations came from independent units. Therefore, standard errors of effects are usually larger. Otherwise, they are biased [1]. Since most researches are not aware of multilevel models, they design their experiments and aggregate their data to fit the independence assumption, which is rarely a good idea. Many are not even aware of modeling beyond hypothesis tests, and are unable or unwilling to adjust their analysis for confounding factors or non-sampling errors that arise due to experiment design flaws. Also, p-values should be deprecated, since a) nil hypothesis are strawmen at best and false by definition at worst [2] and b) they incentivize researchers to not think hard about effect sizes and uncertainty in their problems. [1] https://academic.oup.com/biomet/article/73/1/13/246001 https://academic.oup.com/biomet/article/73/1/13/246001 [2] http://www.stat.columbia.edu/~gelman/research/published/failure_of.pdf http://www.stat.columbia.edu/~gelman/research/published/fail...
    - nextos 4y ago
      Many articles in e.g. Nature Genetics are cheating by hacking their p-values. These hacks would be much harder to get away with if, as a starter, they were asked to use hierarchical models and continuous explanatory variables, whenever possible. The article from Andrew Gelman you cited explains this quite well. In general the review articles and books he has co-authored are incredibly helpful to learn how to avoid common issues that plague statistical inference. We need to shift away from null hypotheses and p-values towards generative models, model selection and effect sizes. It leads to much more robust inference.
- statheros 4y ago
  So the results of this paper are also false... ironic.
- thedailymail 4y ago
  >>Follow up analysis by Jager and Leek (2014) https://academic.oup.com/biostatistics/article/15/1/1/244509 https://academic.oup.com/biostatistics/article/15/1/1/244509 suggests the false discovery rate is closer to 14% than 50%. Maybe the title should be updated to "Why most published research findings, including this one, are false".
- civilized 4y ago
  The 2014 paper is just another theoretical analysis that gets a different answer. It's just one cute, provocative theory versus another.
- KvanteKat 4y ago
  For anyone who likes academic drama (or who is interested in the underlying methodological disagreements among academic statisticians), it is worth pointing out that Jager and Leek's 2014 paper was a discussion paper, and that Ionnidas was one of the people invited to write a response to be published alongside with the original paper. He did (https://doi.org/10.1093/biostatistics/kxt036 https://doi.org/10.1093/biostatistics/kxt036) and the response is extremely critical of Jaeger and Leeks' methodology and his contempt for the authors is not hard to read between the lines.
  - KvanteKat 4y ago
    For anyone who wants to get really into the weeds, here are all the articles in the sequence of discussion papers: Main Paper (Jager and Leek): ttps://doi.org/10.1093/biostatistics/kxt007 Response papers: - Yoav Benjamini and Yotam Hechtlinger: https://doi.org/10.1093/biostatistics/kxt032 - David R. Cox: https://doi.org/10.1093/biostatistics/kxt033 - Andrew Gelman and Keith O'Rourke: https://doi.org/10.1093/biostatistics/kxt034 - Steven N. Goodman: https://doi.org/10.1093/biostatistics/kxt035 - John P. A. Ioannidis (the spicy response): https://doi.org/10.1093/biostatistics/kxt036 - Martijn J. Schuemie, Patrick B. Ryan, Marc A. Suchard, Zach Shahn, and David Madigan: https://doi.org/10.1093/biostatistics/kxt037 Jaeger and Leeks' rejoinder to the responses: https://doi.org/10.1093/biostatistics/kxt038 https://doi.org/10.1093/biostatistics/kxt038 edit: fixed some formatting and link to main paper
daze42 4y ago
There appears to be an error in the correction posted in August of this year. See if you can spot it.
- usgroup 4y ago
  This one? https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1004085 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... Lol yes, research finding Yes, true relationship No is missing a left bracket in the denominator.
dang 4y ago
Related: Most Published Research Findings Are False (2005) - https://news.ycombinator.com/item?id=19016399 https://news.ycombinator.com/item?id=19016399 - Jan 2019 (39 comments) Why Most Published Research Findings Are False (2005) - https://news.ycombinator.com/item?id=18106679 https://news.ycombinator.com/item?id=18106679 - Sept 2018 (40 comments) Most Published Research Findings Are False–But Little Replication Goes Long Way - https://news.ycombinator.com/item?id=9337355 https://news.ycombinator.com/item?id=9337355 - April 2015 (3 comments) Why Most Published Research Findings Are False - https://news.ycombinator.com/item?id=8340405 https://news.ycombinator.com/item?id=8340405 - Sept 2014 (2 comments) Why most published scientific research is probably false [video] - https://news.ycombinator.com/item?id=6661710 https://news.ycombinator.com/item?id=6661710 - Nov 2013 (53 comments) Most published research results are false - https://news.ycombinator.com/item?id=2207750 https://news.ycombinator.com/item?id=2207750 - Feb 2011 (15 comments) Why Most Published Research Findings Are False - https://news.ycombinator.com/item?id=1825007 https://news.ycombinator.com/item?id=1825007 - Oct 2010 (40 comments) Lies, Damned Lies, and Medical Science - https://news.ycombinator.com/item?id=1793838 https://news.ycombinator.com/item?id=1793838 - Oct 2010 (27 comments) Lies, Damned Lies, and Medical Science - https://news.ycombinator.com/item?id=1787055 https://news.ycombinator.com/item?id=1787055 - Oct 2010 (2 comments) Why Most Published Research Findings Are False (2005) - https://news.ycombinator.com/item?id=833879 https://news.ycombinator.com/item?id=833879 - Sept 2009 (2 comments)
- hdesh 4y ago
  Dang (pun intended), is there a trick to find all these similar articles?
  - dang 4y ago
    I just use HN search (which zaik linked to) but I have a bunch of keyboard shortcuts to make it go faster - https://news.ycombinator.com/item?id=29370676 https://news.ycombinator.com/item?id=29370676.
  - zaik 4y ago
    https://hn.algolia.com/ https://hn.algolia.com/
deleted 4y ago
[deleted]
Dawnyhf5 4y ago
hackandthink 4y ago
Ioannidis paper really impressed me back then. I was surprised when Ioannidis supported shady Covid research. Ideology can catch everyone. Beware. https://sciencebasedmedicine.org/what-the-heck-happened-to-john-ioannidis/ https://sciencebasedmedicine.org/what-the-heck-happened-to-j...
- rossdavidh 4y ago
  Ioannidis was the first major name to point out that the mortality rate for covid-19 was 0.25-0.5%, at a time when figures as high as 1 in 8 were being cited. Abundant evidence since then, including CDC's official estimates in the US, suggest he was correct. So, you know, beware, ideology can catch everyone. Even you.
  - trention 4y ago
    The first estimates from Ioannidis from the flawed (in every possible way) California paper were less than 0.1% IFR. He later revised them to 0.16% IFR, not sure if subsequent "revision" (=admission of being wrong) was done afterwards. Bear in mind, that was his IFR estimate for March-April 2020, when a lot of treatment was being done wrong (=intubate early) and even steroids were not supposed to be used for treatment outside RCTs. Meanwhile, the CFR at Diamond Princess was at 2.6%, so refrain from the idiocy "1 in 8 IFR estimates" as only completely uninformed people will fall for them.
  - janef0421 4y ago
    The reasonableness of a statement should be based on how well it is supported at the time it is made, not whether it is later demonstrated to be correct.
  - genewitch 4y ago
    To be fair, in the beginning the policy was to intubate and put on ventilator. This had a 97.2% chance of ending in death. So "1 in 8" was possibly true for the cases of people in the beginning who were both very susceptible to catching a respiratory disease in conjunction with 4 - or more - comorbidities.
  - 4y ago
H8crilA 4y ago
Here's an XKCD summary of the paper: https://xkcd.com/882/ https://xkcd.com/882/ It really is that simple.
- tgv 4y ago
  That's multiple comparisons. A common error, for sure, but not the only problem, by far.
  - H8crilA 4y ago
    N attempts at proving N different invalid hypotheses is not much different from N attempts at proving 1 invalid hypothesis (all at the same p values). Both will result in p*N incorrect validations.
user3939382 4y ago
Related, if you're interested in this topic: https://www.youtube.com/watch?v=z6IO2DZjOkY https://www.youtube.com/watch?v=z6IO2DZjOkY Basically, the corruption of medical science and practice.
elmolino89 4y ago
While using the software packages released last week may not be the greatest idea, I am leery of any research paper using 10 years old genome assembly, Ensembl annotation release in the 40ish (we are at #105 ) or clearly outdated program versions with X updates in the last 5 years. Also if Fedex/UPS were tracking packages with bordering on cavalier attitude observed in some labs, often we would be getting bags of guano ordered by some horticulturalist instead of a book of our choice. Or even an empty bag, since QC may be an afterthought and hard wet lab work may still produce unusable crappy data. On the other hand the brand new technologies are rather expensive, good quality human tissue samples hard to get so scrapping the bottom of the barrel trying to justify grant $$$ is unavoidable.
- stonemetal12 4y ago
  >Ensembl annotation release in the 40ish (we are at #105 ) Were there any major bug fixes that would impact their work in that time window? Maybe for them it is done instead of outdated.
  - elmolino89 4y ago
    Sure. Some genes were retired because there was not enough support. Earlier Ensembl versions used older genome assembly. Which means: some genes "jumped" from one chromosome to another, or got properly stitched residing before partially on floating contigs. Just to be clear: I am not saying that anything done in 2022 using hg19 is 100% wrong. Just that it is a bit like using a stretched shoelace 50cm (+/- 5cm) to measure your corridor when you have a decent tape measure in your pocket. There are microarrays used in Big Science projects with ~1/3 of probes not matching human transcripts from latest ENSEMBL. Since most of them map to the current genome who knows what is the meaning of the signal from such probes. Unannotated exons? Retained introns? But some probes do not even map to the genome => some silly splicing error(?) packed in a plasmid in the 90ies?
apienx 4y ago
Related: https://en.m.wikipedia.org/wiki/Half-life_of_knowledge https://en.m.wikipedia.org/wiki/Half-life_of_knowledge Psychology is the worst offender. Human behavior is quite hard to model. ;-)
- JacobThreeThree 4y ago
  The journal linked in the OP is apparently one of the better and more rigorous journals. For example: >Public Library of Science, PLOS ONE, was the only journal that called attention to the paper's potential ethical problems and consequently rejected it within 2 weeks. https://www.theguardian.com/higher-education-network/2013/oct/04/open-access-journals-fake-paper https://www.theguardian.com/higher-education-network/2013/oc...
slt2021 4y ago
Is this research finding ("Why most published research findings are false") also false? Which means published research findings are true? But then it will turn all research findings false, which is impossible I am confused
- est 4y ago
  you are confused with the concept of "most"
trention 4y ago
I am not completely sure whether most of Ioannidis' own research is "false" - but all of his covid takes (in whatever form) are.
janef0421 4y ago
That's totally normal. An individual study is unlikely to have enough statistical power to make a definite conclusion, and an individual line of evidence is insufficient to confirm a theory. No conclusion can be stated with any certainty until it is verified repeatedly, and no theory is fit for use until several consistent lines of evidence conform to its predictions. In fact, even calling these findings "false"is often inaccurate, as most researchers don't make strong claims.
epolanski 4y ago
That's not new, especially in natural sciences 90%+ of the experiments cannot be replicated.
- nequo 4y ago
  That sounds surprising to me. Do you have a source where I could read more?
  - justinpombrio 4y ago
    https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
    - nequo 4y ago
      The footnote provided[1] for this sentence in the intro does not actually mention the word "natural": Data strongly indicate that other natural, and social sciences are affected as well. It only writes that physicists and chemists are the most confident in their fields' results, and that a biochemistry graduate student is complaining about the effort it takes to replicate results. It also writes that 70+% of researchers have failed to replicate an experiment. But that is very far from saying that 70+% of all experiments don't replicate. And the figure given by parent is 90+%. The Prevalence section of the Wikipedia page talks about psychology, medicine, economics, and other social sciences. Medicine is arguably a natural science. But parent’s comment made me think it’s referring to results in physics or chemistry. [1] https://www.nature.com/articles/533452a https://www.nature.com/articles/533452a
- MichaelCollins 4y ago
  Do you mean in the social sciences? The natural sciences are physics, astronomy, chemistry, geology, and biology. These fields of inquiry lend themselves to empiricism. Mistakes/fraud happen, but it's a lot better than social science where empiricism is routinely rejected (straining the claim to even be science at all.)
29athrowaway 4y ago
Publish or perish and citation quantity driven science, that's why.

157 points by Jimmc414 4y ago

T3RMINATED 4y ago

realaleris149 4y ago

including this one?

planetsprite 4y ago
predictable joke
- Sakos 4y ago
  Is it a joke?

fastaguy88 4y ago

This was a very controversial paper when it was published, perhaps because of its incendiary title. But the paper is much more subtle than the title suggests. Basically the idea is that if you try to test phenomena that are completely unexpected, your prior odds are low, so even if you get a positive result, there is a good chance the result is incorrect. So there is a danger that by trying to ensure more correct results, scientists may avoid paradigm changing experiments, which would be a loss. Follow up analysis by Jager and Leek (2014) https://academic.oup.com/biostatistics/article/15/1/1/244509 https://academic.oup.com/biostatistics/article/15/1/1/244509 suggests the false discovery rate is closer to 14% than 50%.

themitigating 4y ago
That's a huge difference yet anti-intellectuals will contiune to post this paper until the end of time as proof that you can't trust science
- diognesofsinope 4y ago
  I mean, the failure of a lot of experiments to replicate is extraordinarily well documented... Not to mention the corollaries from the Wikipedia article (https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...): "In addition to the main result, Ioannidis lists six corollaries for factors that can influence the reliability of published research. Research findings in a scientific field are less likely to be true, 1. the smaller the studies conducted. 2. the smaller the effect sizes. 3. the greater the number and the lesser the selection of tested relationships. 4. the greater the flexibility in designs, definitions, outcomes, and analytical modes. 5. the greater the financial and other interests and prejudices. 6. the hotter the scientific field (with more scientific teams involved)." These are all reasonable criticisms from my own area of expertise: applied econometrics.
  - nonameiguess 4y ago
    Being skeptical of cutting edge research is a lot different than distrusting science. There are plenty of people out there denying special relativity, evolution by natural selection, or believing all of western medicine is invalid, based on extrapolation from stuff like this. I won't speak for all textbooks, but generally stuff you find in there should not be the same as what you find in journals, and is much more settled. Big caveat that that isn't necessarily true for younger sciences without long-established theory, say exercise physiology or social psychology, but something like a chemistry textbook is pretty damn trustworthy. And those are what people who aren't actually scientists should mostly be educating themselves with, not newspaper science reporting sections.
    - epolanski 4y ago
      Doubting a scientific paper results is different from distrusting science. Academia is a complex place and it's full of fake results to get publications. (I'm coauthor of several scientific papers and I know for a fact countless highly rated papers in my former field are not reproducible because they come from adjusted numbers).
- deleted 4y ago
glofish 4y ago
This followup paper seems to summarize p-values reported in 77K papers and uses the distribution of these p-values to compute the FDR. Alas everyone knows that the p-values reported in accepted (!) papers are questionable at best - any analysis that uses them is on shaky foundation. The misuse of p-values in science is well known and an endemic problem - so what do we learn from a re-analysis of made up numbers?
- Manu40 4y ago
  Not to be pedantic, but considering how many people I have run into who take all research studies as if they are holy gospel; you may want to reconsider saying "everyone knows". Just my 5 cents on the matter.
  - glofish 4y ago
    good point, what I really meant is by now everybody should know that p-values are questionable at best
    - genewitch 4y ago
      can we name one mainstream journalist, or any policy makers that know anything about reproducibility - or, more honestly, the appalling lack thereof - or "p-values are questionable at best"? There's been books, such as "how to lie with statistics" that should be mandatory reading, at least on entry to adulthood. I am one of maybe 3 people i know personally that knows how to actually parse a scientific paper, so when someone starts throwing a half dozen studies at me to back up their point, i can usually - on the scale of quarter hours - nitpick their understanding of the sources they used. This doesn't win anyone friends, and something i've said for nearly 3 decades: Religion and science like to point fingers, but religion seems more likely to bend and change than science. Every time i mention this or an analogue i'm shouted at that it's "actually the opposite". Shut up already, it's science. The science is in. This is science!
      - Manu40 4y ago
6gvONxR4sf7o 4y ago
I'd love to know the correlation between failure to replicate and public virality of findings. I wouldn't be surprised of the more exciting findings replicate at a different rate from average.
- H8crilA 4y ago
  That's kind of the point. More exciting statements are more often "proven" because they're more often attempted to be proven.
nextos 4y ago
A good way to address these issues is to frame all experiments as multilevel models. See [1] for a long discussion from Andrew Gelman et al on why this is advisable. Surprisingly, many people working in statistics still ignore the James-Stein theorem, which provides a theoretical justification for multilevel models. In layman terms, said theorem shows that if you are simultaneously estimating many random variables you should borrow information across variables [2]. Estimating them one by one is suboptimal and does not minimize the global mean squared error. Multilevel models "shrink" individual effect sizes by looking at the overall distribution of effect sizes and provide much more realistic estimates. [1] https://arxiv.org/abs/0907.2478 https://arxiv.org/abs/0907.2478 [2] https://en.wikipedia.org/wiki/Stein%27s_example https://en.wikipedia.org/wiki/Stein%27s_example
- kimukasetsu 4y ago
  This. Multilevel models relax the assumption of independent observations by specifying that the measures of repeated experimental units are dependent on each other. It's a way of telling your model that it has less information than it would have if all observations came from independent units. Therefore, standard errors of effects are usually larger. Otherwise, they are biased [1]. Since most researches are not aware of multilevel models, they design their experiments and aggregate their data to fit the independence assumption, which is rarely a good idea. Many are not even aware of modeling beyond hypothesis tests, and are unable or unwilling to adjust their analysis for confounding factors or non-sampling errors that arise due to experiment design flaws. Also, p-values should be deprecated, since a) nil hypothesis are strawmen at best and false by definition at worst [2] and b) they incentivize researchers to not think hard about effect sizes and uncertainty in their problems. [1] https://academic.oup.com/biomet/article/73/1/13/246001 https://academic.oup.com/biomet/article/73/1/13/246001 [2] http://www.stat.columbia.edu/~gelman/research/published/failure_of.pdf http://www.stat.columbia.edu/~gelman/research/published/fail...
  - nextos 4y ago
    Many articles in e.g. Nature Genetics are cheating by hacking their p-values. These hacks would be much harder to get away with if, as a starter, they were asked to use hierarchical models and continuous explanatory variables, whenever possible. The article from Andrew Gelman you cited explains this quite well. In general the review articles and books he has co-authored are incredibly helpful to learn how to avoid common issues that plague statistical inference. We need to shift away from null hypotheses and p-values towards generative models, model selection and effect sizes. It leads to much more robust inference.
statheros 4y ago
So the results of this paper are also false... ironic.
thedailymail 4y ago
>>Follow up analysis by Jager and Leek (2014) https://academic.oup.com/biostatistics/article/15/1/1/244509 https://academic.oup.com/biostatistics/article/15/1/1/244509 suggests the false discovery rate is closer to 14% than 50%. Maybe the title should be updated to "Why most published research findings, including this one, are false".
civilized 4y ago
The 2014 paper is just another theoretical analysis that gets a different answer. It's just one cute, provocative theory versus another.
KvanteKat 4y ago
For anyone who likes academic drama (or who is interested in the underlying methodological disagreements among academic statisticians), it is worth pointing out that Jager and Leek's 2014 paper was a discussion paper, and that Ionnidas was one of the people invited to write a response to be published alongside with the original paper. He did (https://doi.org/10.1093/biostatistics/kxt036 https://doi.org/10.1093/biostatistics/kxt036) and the response is extremely critical of Jaeger and Leeks' methodology and his contempt for the authors is not hard to read between the lines.
- KvanteKat 4y ago
  For anyone who wants to get really into the weeds, here are all the articles in the sequence of discussion papers: Main Paper (Jager and Leek): ttps://doi.org/10.1093/biostatistics/kxt007 Response papers: - Yoav Benjamini and Yotam Hechtlinger: https://doi.org/10.1093/biostatistics/kxt032 - David R. Cox: https://doi.org/10.1093/biostatistics/kxt033 - Andrew Gelman and Keith O'Rourke: https://doi.org/10.1093/biostatistics/kxt034 - Steven N. Goodman: https://doi.org/10.1093/biostatistics/kxt035 - John P. A. Ioannidis (the spicy response): https://doi.org/10.1093/biostatistics/kxt036 - Martijn J. Schuemie, Patrick B. Ryan, Marc A. Suchard, Zach Shahn, and David Madigan: https://doi.org/10.1093/biostatistics/kxt037 Jaeger and Leeks' rejoinder to the responses: https://doi.org/10.1093/biostatistics/kxt038 https://doi.org/10.1093/biostatistics/kxt038 edit: fixed some formatting and link to main paper

daze42 4y ago

There appears to be an error in the correction posted in August of this year. See if you can spot it.

usgroup 4y ago
This one? https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1004085 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... Lol yes, research finding Yes, true relationship No is missing a left bracket in the denominator.

dang 4y ago

Related: Most Published Research Findings Are False (2005) - https://news.ycombinator.com/item?id=19016399 https://news.ycombinator.com/item?id=19016399 - Jan 2019 (39 comments) Why Most Published Research Findings Are False (2005) - https://news.ycombinator.com/item?id=18106679 https://news.ycombinator.com/item?id=18106679 - Sept 2018 (40 comments) Most Published Research Findings Are False–But Little Replication Goes Long Way - https://news.ycombinator.com/item?id=9337355 https://news.ycombinator.com/item?id=9337355 - April 2015 (3 comments) Why Most Published Research Findings Are False - https://news.ycombinator.com/item?id=8340405 https://news.ycombinator.com/item?id=8340405 - Sept 2014 (2 comments) Why most published scientific research is probably false [video] - https://news.ycombinator.com/item?id=6661710 https://news.ycombinator.com/item?id=6661710 - Nov 2013 (53 comments) Most published research results are false - https://news.ycombinator.com/item?id=2207750 https://news.ycombinator.com/item?id=2207750 - Feb 2011 (15 comments) Why Most Published Research Findings Are False - https://news.ycombinator.com/item?id=1825007 https://news.ycombinator.com/item?id=1825007 - Oct 2010 (40 comments) Lies, Damned Lies, and Medical Science - https://news.ycombinator.com/item?id=1793838 https://news.ycombinator.com/item?id=1793838 - Oct 2010 (27 comments) Lies, Damned Lies, and Medical Science - https://news.ycombinator.com/item?id=1787055 https://news.ycombinator.com/item?id=1787055 - Oct 2010 (2 comments) Why Most Published Research Findings Are False (2005) - https://news.ycombinator.com/item?id=833879 https://news.ycombinator.com/item?id=833879 - Sept 2009 (2 comments)

hdesh 4y ago
Dang (pun intended), is there a trick to find all these similar articles?
- dang 4y ago
  I just use HN search (which zaik linked to) but I have a bunch of keyboard shortcuts to make it go faster - https://news.ycombinator.com/item?id=29370676 https://news.ycombinator.com/item?id=29370676.
- zaik 4y ago
  https://hn.algolia.com/ https://hn.algolia.com/

deleted 4y ago

[deleted]

Dawnyhf5 4y ago

hackandthink 4y ago

Ioannidis paper really impressed me back then. I was surprised when Ioannidis supported shady Covid research. Ideology can catch everyone. Beware. https://sciencebasedmedicine.org/what-the-heck-happened-to-john-ioannidis/ https://sciencebasedmedicine.org/what-the-heck-happened-to-j...

rossdavidh 4y ago
Ioannidis was the first major name to point out that the mortality rate for covid-19 was 0.25-0.5%, at a time when figures as high as 1 in 8 were being cited. Abundant evidence since then, including CDC's official estimates in the US, suggest he was correct. So, you know, beware, ideology can catch everyone. Even you.
- trention 4y ago
  The first estimates from Ioannidis from the flawed (in every possible way) California paper were less than 0.1% IFR. He later revised them to 0.16% IFR, not sure if subsequent "revision" (=admission of being wrong) was done afterwards. Bear in mind, that was his IFR estimate for March-April 2020, when a lot of treatment was being done wrong (=intubate early) and even steroids were not supposed to be used for treatment outside RCTs. Meanwhile, the CFR at Diamond Princess was at 2.6%, so refrain from the idiocy "1 in 8 IFR estimates" as only completely uninformed people will fall for them.
- janef0421 4y ago
  The reasonableness of a statement should be based on how well it is supported at the time it is made, not whether it is later demonstrated to be correct.
- genewitch 4y ago
  To be fair, in the beginning the policy was to intubate and put on ventilator. This had a 97.2% chance of ending in death. So "1 in 8" was possibly true for the cases of people in the beginning who were both very susceptible to catching a respiratory disease in conjunction with 4 - or more - comorbidities.
- 4y ago

H8crilA 4y ago

Here's an XKCD summary of the paper: https://xkcd.com/882/ https://xkcd.com/882/ It really is that simple.

tgv 4y ago
That's multiple comparisons. A common error, for sure, but not the only problem, by far.
- H8crilA 4y ago
  N attempts at proving N different invalid hypotheses is not much different from N attempts at proving 1 invalid hypothesis (all at the same p values). Both will result in p*N incorrect validations.

user3939382 4y ago

Related, if you're interested in this topic: https://www.youtube.com/watch?v=z6IO2DZjOkY https://www.youtube.com/watch?v=z6IO2DZjOkY Basically, the corruption of medical science and practice.

elmolino89 4y ago

While using the software packages released last week may not be the greatest idea, I am leery of any research paper using 10 years old genome assembly, Ensembl annotation release in the 40ish (we are at #105 ) or clearly outdated program versions with X updates in the last 5 years. Also if Fedex/UPS were tracking packages with bordering on cavalier attitude observed in some labs, often we would be getting bags of guano ordered by some horticulturalist instead of a book of our choice. Or even an empty bag, since QC may be an afterthought and hard wet lab work may still produce unusable crappy data. On the other hand the brand new technologies are rather expensive, good quality human tissue samples hard to get so scrapping the bottom of the barrel trying to justify grant $$$ is unavoidable.

stonemetal12 4y ago
>Ensembl annotation release in the 40ish (we are at #105 ) Were there any major bug fixes that would impact their work in that time window? Maybe for them it is done instead of outdated.
- elmolino89 4y ago
  Sure. Some genes were retired because there was not enough support. Earlier Ensembl versions used older genome assembly. Which means: some genes "jumped" from one chromosome to another, or got properly stitched residing before partially on floating contigs. Just to be clear: I am not saying that anything done in 2022 using hg19 is 100% wrong. Just that it is a bit like using a stretched shoelace 50cm (+/- 5cm) to measure your corridor when you have a decent tape measure in your pocket. There are microarrays used in Big Science projects with ~1/3 of probes not matching human transcripts from latest ENSEMBL. Since most of them map to the current genome who knows what is the meaning of the signal from such probes. Unannotated exons? Retained introns? But some probes do not even map to the genome => some silly splicing error(?) packed in a plasmid in the 90ies?

apienx 4y ago

Related: https://en.m.wikipedia.org/wiki/Half-life_of_knowledge https://en.m.wikipedia.org/wiki/Half-life_of_knowledge Psychology is the worst offender. Human behavior is quite hard to model. ;-)

JacobThreeThree 4y ago
The journal linked in the OP is apparently one of the better and more rigorous journals. For example: >Public Library of Science, PLOS ONE, was the only journal that called attention to the paper's potential ethical problems and consequently rejected it within 2 weeks. https://www.theguardian.com/higher-education-network/2013/oct/04/open-access-journals-fake-paper https://www.theguardian.com/higher-education-network/2013/oc...

slt2021 4y ago

Is this research finding ("Why most published research findings are false") also false? Which means published research findings are true? But then it will turn all research findings false, which is impossible I am confused

est 4y ago
you are confused with the concept of "most"

trention 4y ago

I am not completely sure whether most of Ioannidis' own research is "false" - but all of his covid takes (in whatever form) are.

janef0421 4y ago

That's totally normal. An individual study is unlikely to have enough statistical power to make a definite conclusion, and an individual line of evidence is insufficient to confirm a theory. No conclusion can be stated with any certainty until it is verified repeatedly, and no theory is fit for use until several consistent lines of evidence conform to its predictions. In fact, even calling these findings "false"is often inaccurate, as most researchers don't make strong claims.

epolanski 4y ago

That's not new, especially in natural sciences 90%+ of the experiments cannot be replicated.

nequo 4y ago
That sounds surprising to me. Do you have a source where I could read more?
- justinpombrio 4y ago
  https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
  - nequo 4y ago
    The footnote provided[1] for this sentence in the intro does not actually mention the word "natural": Data strongly indicate that other natural, and social sciences are affected as well. It only writes that physicists and chemists are the most confident in their fields' results, and that a biochemistry graduate student is complaining about the effort it takes to replicate results. It also writes that 70+% of researchers have failed to replicate an experiment. But that is very far from saying that 70+% of all experiments don't replicate. And the figure given by parent is 90+%. The Prevalence section of the Wikipedia page talks about psychology, medicine, economics, and other social sciences. Medicine is arguably a natural science. But parent’s comment made me think it’s referring to results in physics or chemistry. [1] https://www.nature.com/articles/533452a https://www.nature.com/articles/533452a
MichaelCollins 4y ago
Do you mean in the social sciences? The natural sciences are physics, astronomy, chemistry, geology, and biology. These fields of inquiry lend themselves to empiricism. Mistakes/fraud happen, but it's a lot better than social science where empiricism is routinely rejected (straining the claim to even be science at all.)

29athrowaway 4y ago

Publish or perish and citation quantity driven science, that's why.