7 ms·

This was a very controversial paper when it was published, perhaps because of its incendiary title. But the paper is much more subtle than the title suggests.

This was a very controversial paper when it was published, perhaps because of its incendiary title. But the paper is much more subtle than the title suggests. Basically the idea is that if you try to test phenomena that are completely unexpected, your prior odds are low, so even if you get a positive result, there is a good chance the result is incorrect. So there is a danger that by trying to ensure more correct results, scientists may avoid paradigm changing experiments, which would be a loss. Follow up analysis by Jager and Leek (2014) https://academic.oup.com/biostatistics/article/15/1/1/244509 https://academic.oup.com/biostatistics/article/15/1/1/244509 suggests the false discovery rate is closer to 14% than 50%.

themitigating 4y ago
That's a huge difference yet anti-intellectuals will contiune to post this paper until the end of time as proof that you can't trust science
- diognesofsinope 4y ago
  I mean, the failure of a lot of experiments to replicate is extraordinarily well documented... Not to mention the corollaries from the Wikipedia article (https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...): "In addition to the main result, Ioannidis lists six corollaries for factors that can influence the reliability of published research. Research findings in a scientific field are less likely to be true, 1. the smaller the studies conducted. 2. the smaller the effect sizes. 3. the greater the number and the lesser the selection of tested relationships. 4. the greater the flexibility in designs, definitions, outcomes, and analytical modes. 5. the greater the financial and other interests and prejudices. 6. the hotter the scientific field (with more scientific teams involved)." These are all reasonable criticisms from my own area of expertise: applied econometrics.
  - nonameiguess 4y ago
    Being skeptical of cutting edge research is a lot different than distrusting science. There are plenty of people out there denying special relativity, evolution by natural selection, or believing all of western medicine is invalid, based on extrapolation from stuff like this. I won't speak for all textbooks, but generally stuff you find in there should not be the same as what you find in journals, and is much more settled. Big caveat that that isn't necessarily true for younger sciences without long-established theory, say exercise physiology or social psychology, but something like a chemistry textbook is pretty damn trustworthy. And those are what people who aren't actually scientists should mostly be educating themselves with, not newspaper science reporting sections.
    - epolanski 4y ago
      Doubting a scientific paper results is different from distrusting science. Academia is a complex place and it's full of fake results to get publications. (I'm coauthor of several scientific papers and I know for a fact countless highly rated papers in my former field are not reproducible because they come from adjusted numbers).
- deleted 4y ago
  [deleted]
- yucky 4y ago
  The Replication Crisis is real, denying that is anti-intellectual. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
  - Retric 4y ago
    Calling something a crisis is arbitrary. So it both is and isn’t a crisis based on completely arbitrary metrics.
    - yucky 4y ago
      I encourage you to dig in to the sources posted. The majority of published research can't be and/or hasn't been replicated. That's a crisis by any definition, especially with the amount of idiots running around bleating about "the science is settled" or "trust the science" or whatever catch phrase.
      - Retric 4y ago
        The degree to which the science is settled varies wildly by topic and that’s ok. Nutrition is perceived to be filled with a lot of junk, but without dietary vitamin C you will die. That’s settled even inside a field filled with debate. Individual papers where never intended to be the final arbiters of truth, that’s not their role. If nobody thinks things are worth looking into again then stuff stays in a very nebulous state which is no worse than where things where before a paper was published.
        yucky 4y ago
        >If nobody thinks things are worth looking into again then stuff stays in a very nebulous state which is no worse than where things where before a paper was published. I don't think so, because of the aforementioned idiots who take a study that can't even be replicated but then cite the paper as evidence for something. If it can't be replicated, logic would say there is a very good chance it's inaccurate. Yet it's treated as being "settled science" which is worse. Also, many times it isn't that nobody thinks it's worth looking into again, it's that coming to "the wrong conclusion" on some things will end their careers so they'd rather play it safe and just keep sucking up grant money to pump out more of the same garbage.
- deleted 4y ago
  [deleted]
- serial_dev 4y ago
  Mentioning that not everything that's published in a peer reviewed publication is automatically 100% correct is not anti-intellectual. Mistakes can be made, data can be limited or misinterpreted, scientists can be corrupted. Theories, "common sense" can change: what we thought we'd know for sure have been proven wrong. If you are a scientist, you know that. Only because I don't automatically take every "scientific" finding at face value, it doesn't mean I don't trust science. I trust science, I just don't trust every single research, experiment, scientist, or journal. Actually that's in itself, in a way, science.
  - afpx 4y ago
    “Science” currently incentivizes the wrong things. How do you change the incentives (back)?
    - chithanh 4y ago
      I think it is a misconception that switching to the "right" incentives will solve problems. The very existence of incentives is not conducive for knowledge-based work. https://hbr.org/1993/09/why-incentive-plans-cannot-work https://hbr.org/1993/09/why-incentive-plans-cannot-work
  - hbn 4y ago
    Yup. Blindly "trusting the science" is inherently anti-science There is plenty of things that we took as scientific fact in the past that turned out to be false. Science is supposed to question existing notions and test/prove new theories. If you just assume that we're in a post-science era where we've got it all figured out, and our current theories are all correct, that's not science, it's dogmatic faith. Ignaz Semmelweis was destroyed by the medical community for his insane idea that doctors should wash their hands before performing medical procedures. He was attacked about to the point where he ended up having a nervous breakdown and was committed to an asylum where he was beaten to death by the guards. Serves him right for questioning the science!
    - dekhn 4y ago
- timr 4y ago
  It's also wrong. There have now been large-scale initiatives to reproduce papers, and they're getting higher failure rates than 14%. The statistics underlying all of this make a number of implicit assumptions that may or may not be true IRL (for example: that journals publish papers independent of the salaciousness of claims made within them). If Science picks out only the top-5% most-sensational claims for publication, then you can't assume that a 95% CI is a safe threshold. You've probably got to increase it to a much higher value to have any prayer of getting past the inherent bias in such a process.
  - genewitch 4y ago
    Wait until you start looking in to, say, psychology papers, vis a vis reproducibility. I think that a lot of these studies should be reviewed during the actual research and study part by at least a sociologist or cultural anthropologist, and a mathematician with expertise in probability. When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. For a recent example, the US White house is engaging in a 5 year study to determine if releasing sulfur dioxide into the stratosphere would accomplish enough "cooling" to offset the "climate crisis". the "climate crisis" is something that exists in computer models, few of the models agree, and if they do, it behooves us to look at when they started agreeing and find out why that change happened. a more personal example: I can't reproduce getting CaCO out of seawater, yet.
    - tgv 4y ago
      I think you've never done psych research. First of all, statisticians and methodologists are frequently included, also in the design phase, if it's not a simple adaptation of an existing study. Nonetheless, many papers contain design errors. Second, sociology and anthropology are among the least empirical of academic disciplines. What reason would you have to involve them? Third, many studies are of the type "my theory predicts X and my experiment shows p < 0.05". That's almost wrong by definition. Bayesian theory explains why. Fourth, many studies have conclusions/claims that cannot be inferred from the actual finding. > When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. It is important, but as long as academia is a "turn out papers for tenure" industry, it won't be fixed. Don't take the conclusions of a paper for true. They're meant for other academics.
- treeman79 4y ago
  It’s not that you can’t trust “science” it’s that people are fallible.
  - viridian 4y ago
    The big unaddressed issue is that conclusions based on empirical data can only be as good as the observations. If the data is biased, manipulated, "massaged", etc, there can be no good output from that poisoned tree, and other persons cannot contradict the results without attempting to replicate the entire research process beginning to end. With math, physics, applied math, computer science, etc, bad results can be usually (but not always) deconstructed by anyone with the merest tools of the rational; formal logic, application of various categorical laws, etc. Science as a process is useful, but its conclusions can only be as reliable as what its body of producers are incentivized to produce.
- kriops 4y ago
  There's a big difference between science and social science. Or science and medicine for that matter. They are different domains that require different models of reasoning, which come with their respective sets of limitations. There's a reason you don't see a crisis in e.g. physics, where their hypotheses are leaps and bounds more testable.
  - onos 4y ago
    This failure rate suggests a reason the social sciences don’t “lead anywhere”: If you try to build a theorem off of two earlier results, but these each only have probability p of being right, then the chance your new theorem is correct goes like p^2. For small p this means deductions based on combining results are very small.
  - onos 4y ago
    This failure rate suggests a reason the social sciences don’t “lead anywhere”: If you try to build a theorem off of two earlier results, but these each only have probability p of being right, then the chance your new theorem is correct goes like p^2. For small p this means the chance that a deduction is correct is very small.
- pfortuny 4y ago
  Have you read Ioaniddis’ reply? It is worth the effort.
glofish 4y ago
This followup paper seems to summarize p-values reported in 77K papers and uses the distribution of these p-values to compute the FDR. Alas everyone knows that the p-values reported in accepted (!) papers are questionable at best - any analysis that uses them is on shaky foundation. The misuse of p-values in science is well known and an endemic problem - so what do we learn from a re-analysis of made up numbers?
- Manu40 4y ago
  Not to be pedantic, but considering how many people I have run into who take all research studies as if they are holy gospel; you may want to reconsider saying "everyone knows". Just my 5 cents on the matter.
  - glofish 4y ago
    good point, what I really meant is by now everybody should know that p-values are questionable at best
    - genewitch 4y ago
      can we name one mainstream journalist, or any policy makers that know anything about reproducibility - or, more honestly, the appalling lack thereof - or "p-values are questionable at best"? There's been books, such as "how to lie with statistics" that should be mandatory reading, at least on entry to adulthood. I am one of maybe 3 people i know personally that knows how to actually parse a scientific paper, so when someone starts throwing a half dozen studies at me to back up their point, i can usually - on the scale of quarter hours - nitpick their understanding of the sources they used. This doesn't win anyone friends, and something i've said for nearly 3 decades: Religion and science like to point fingers, but religion seems more likely to bend and change than science. Every time i mention this or an analogue i'm shouted at that it's "actually the opposite". Shut up already, it's science. The science is in. This is science!
      - Manu40 4y ago
6gvONxR4sf7o 4y ago
I'd love to know the correlation between failure to replicate and public virality of findings. I wouldn't be surprised of the more exciting findings replicate at a different rate from average.
- H8crilA 4y ago
  That's kind of the point. More exciting statements are more often "proven" because they're more often attempted to be proven.
nextos 4y ago
A good way to address these issues is to frame all experiments as multilevel models. See [1] for a long discussion from Andrew Gelman et al on why this is advisable. Surprisingly, many people working in statistics still ignore the James-Stein theorem, which provides a theoretical justification for multilevel models. In layman terms, said theorem shows that if you are simultaneously estimating many random variables you should borrow information across variables [2]. Estimating them one by one is suboptimal and does not minimize the global mean squared error. Multilevel models "shrink" individual effect sizes by looking at the overall distribution of effect sizes and provide much more realistic estimates. [1] https://arxiv.org/abs/0907.2478 https://arxiv.org/abs/0907.2478 [2] https://en.wikipedia.org/wiki/Stein%27s_example https://en.wikipedia.org/wiki/Stein%27s_example
- kimukasetsu 4y ago
  This. Multilevel models relax the assumption of independent observations by specifying that the measures of repeated experimental units are dependent on each other. It's a way of telling your model that it has less information than it would have if all observations came from independent units. Therefore, standard errors of effects are usually larger. Otherwise, they are biased [1]. Since most researches are not aware of multilevel models, they design their experiments and aggregate their data to fit the independence assumption, which is rarely a good idea. Many are not even aware of modeling beyond hypothesis tests, and are unable or unwilling to adjust their analysis for confounding factors or non-sampling errors that arise due to experiment design flaws. Also, p-values should be deprecated, since a) nil hypothesis are strawmen at best and false by definition at worst [2] and b) they incentivize researchers to not think hard about effect sizes and uncertainty in their problems. [1] https://academic.oup.com/biomet/article/73/1/13/246001 https://academic.oup.com/biomet/article/73/1/13/246001 [2] http://www.stat.columbia.edu/~gelman/research/published/failure_of.pdf http://www.stat.columbia.edu/~gelman/research/published/fail...
  - nextos 4y ago
    Many articles in e.g. Nature Genetics are cheating by hacking their p-values. These hacks would be much harder to get away with if, as a starter, they were asked to use hierarchical models and continuous explanatory variables, whenever possible. The article from Andrew Gelman you cited explains this quite well. In general the review articles and books he has co-authored are incredibly helpful to learn how to avoid common issues that plague statistical inference. We need to shift away from null hypotheses and p-values towards generative models, model selection and effect sizes. It leads to much more robust inference.
statheros 4y ago
So the results of this paper are also false... ironic.
thedailymail 4y ago
>>Follow up analysis by Jager and Leek (2014) https://academic.oup.com/biostatistics/article/15/1/1/244509 https://academic.oup.com/biostatistics/article/15/1/1/244509 suggests the false discovery rate is closer to 14% than 50%. Maybe the title should be updated to "Why most published research findings, including this one, are false".
civilized 4y ago
The 2014 paper is just another theoretical analysis that gets a different answer. It's just one cute, provocative theory versus another.
KvanteKat 4y ago
For anyone who likes academic drama (or who is interested in the underlying methodological disagreements among academic statisticians), it is worth pointing out that Jager and Leek's 2014 paper was a discussion paper, and that Ionnidas was one of the people invited to write a response to be published alongside with the original paper. He did (https://doi.org/10.1093/biostatistics/kxt036 https://doi.org/10.1093/biostatistics/kxt036) and the response is extremely critical of Jaeger and Leeks' methodology and his contempt for the authors is not hard to read between the lines.
- KvanteKat 4y ago
  For anyone who wants to get really into the weeds, here are all the articles in the sequence of discussion papers: Main Paper (Jager and Leek): ttps://doi.org/10.1093/biostatistics/kxt007 Response papers: - Yoav Benjamini and Yotam Hechtlinger: https://doi.org/10.1093/biostatistics/kxt032 - David R. Cox: https://doi.org/10.1093/biostatistics/kxt033 - Andrew Gelman and Keith O'Rourke: https://doi.org/10.1093/biostatistics/kxt034 - Steven N. Goodman: https://doi.org/10.1093/biostatistics/kxt035 - John P. A. Ioannidis (the spicy response): https://doi.org/10.1093/biostatistics/kxt036 - Martijn J. Schuemie, Patrick B. Ryan, Marc A. Suchard, Zach Shahn, and David Madigan: https://doi.org/10.1093/biostatistics/kxt037 Jaeger and Leeks' rejoinder to the responses: https://doi.org/10.1093/biostatistics/kxt038 https://doi.org/10.1093/biostatistics/kxt038 edit: fixed some formatting and link to main paper

themitigating 4y ago

That's a huge difference yet anti-intellectuals will contiune to post this paper until the end of time as proof that you can't trust science

diognesofsinope 4y ago
I mean, the failure of a lot of experiments to replicate is extraordinarily well documented... Not to mention the corollaries from the Wikipedia article (https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...): "In addition to the main result, Ioannidis lists six corollaries for factors that can influence the reliability of published research. Research findings in a scientific field are less likely to be true, 1. the smaller the studies conducted. 2. the smaller the effect sizes. 3. the greater the number and the lesser the selection of tested relationships. 4. the greater the flexibility in designs, definitions, outcomes, and analytical modes. 5. the greater the financial and other interests and prejudices. 6. the hotter the scientific field (with more scientific teams involved)." These are all reasonable criticisms from my own area of expertise: applied econometrics.
- nonameiguess 4y ago
  Being skeptical of cutting edge research is a lot different than distrusting science. There are plenty of people out there denying special relativity, evolution by natural selection, or believing all of western medicine is invalid, based on extrapolation from stuff like this. I won't speak for all textbooks, but generally stuff you find in there should not be the same as what you find in journals, and is much more settled. Big caveat that that isn't necessarily true for younger sciences without long-established theory, say exercise physiology or social psychology, but something like a chemistry textbook is pretty damn trustworthy. And those are what people who aren't actually scientists should mostly be educating themselves with, not newspaper science reporting sections.
  - epolanski 4y ago
    Doubting a scientific paper results is different from distrusting science. Academia is a complex place and it's full of fake results to get publications. (I'm coauthor of several scientific papers and I know for a fact countless highly rated papers in my former field are not reproducible because they come from adjusted numbers).
deleted 4y ago
[deleted]
yucky 4y ago
The Replication Crisis is real, denying that is anti-intellectual. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- Retric 4y ago
  Calling something a crisis is arbitrary. So it both is and isn’t a crisis based on completely arbitrary metrics.
  - yucky 4y ago
    I encourage you to dig in to the sources posted. The majority of published research can't be and/or hasn't been replicated. That's a crisis by any definition, especially with the amount of idiots running around bleating about "the science is settled" or "trust the science" or whatever catch phrase.
    - Retric 4y ago
      The degree to which the science is settled varies wildly by topic and that’s ok. Nutrition is perceived to be filled with a lot of junk, but without dietary vitamin C you will die. That’s settled even inside a field filled with debate. Individual papers where never intended to be the final arbiters of truth, that’s not their role. If nobody thinks things are worth looking into again then stuff stays in a very nebulous state which is no worse than where things where before a paper was published.
      - yucky 4y ago
        >If nobody thinks things are worth looking into again then stuff stays in a very nebulous state which is no worse than where things where before a paper was published. I don't think so, because of the aforementioned idiots who take a study that can't even be replicated but then cite the paper as evidence for something. If it can't be replicated, logic would say there is a very good chance it's inaccurate. Yet it's treated as being "settled science" which is worse. Also, many times it isn't that nobody thinks it's worth looking into again, it's that coming to "the wrong conclusion" on some things will end their careers so they'd rather play it safe and just keep sucking up grant money to pump out more of the same garbage.
deleted 4y ago
[deleted]
serial_dev 4y ago
Mentioning that not everything that's published in a peer reviewed publication is automatically 100% correct is not anti-intellectual. Mistakes can be made, data can be limited or misinterpreted, scientists can be corrupted. Theories, "common sense" can change: what we thought we'd know for sure have been proven wrong. If you are a scientist, you know that. Only because I don't automatically take every "scientific" finding at face value, it doesn't mean I don't trust science. I trust science, I just don't trust every single research, experiment, scientist, or journal. Actually that's in itself, in a way, science.
- afpx 4y ago
  “Science” currently incentivizes the wrong things. How do you change the incentives (back)?
  - chithanh 4y ago
    I think it is a misconception that switching to the "right" incentives will solve problems. The very existence of incentives is not conducive for knowledge-based work. https://hbr.org/1993/09/why-incentive-plans-cannot-work https://hbr.org/1993/09/why-incentive-plans-cannot-work
- hbn 4y ago
  Yup. Blindly "trusting the science" is inherently anti-science There is plenty of things that we took as scientific fact in the past that turned out to be false. Science is supposed to question existing notions and test/prove new theories. If you just assume that we're in a post-science era where we've got it all figured out, and our current theories are all correct, that's not science, it's dogmatic faith. Ignaz Semmelweis was destroyed by the medical community for his insane idea that doctors should wash their hands before performing medical procedures. He was attacked about to the point where he ended up having a nervous breakdown and was committed to an asylum where he was beaten to death by the guards. Serves him right for questioning the science!
  - dekhn 4y ago
timr 4y ago
It's also wrong. There have now been large-scale initiatives to reproduce papers, and they're getting higher failure rates than 14%. The statistics underlying all of this make a number of implicit assumptions that may or may not be true IRL (for example: that journals publish papers independent of the salaciousness of claims made within them). If Science picks out only the top-5% most-sensational claims for publication, then you can't assume that a 95% CI is a safe threshold. You've probably got to increase it to a much higher value to have any prayer of getting past the inherent bias in such a process.
- genewitch 4y ago
  Wait until you start looking in to, say, psychology papers, vis a vis reproducibility. I think that a lot of these studies should be reviewed during the actual research and study part by at least a sociologist or cultural anthropologist, and a mathematician with expertise in probability. When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. For a recent example, the US White house is engaging in a 5 year study to determine if releasing sulfur dioxide into the stratosphere would accomplish enough "cooling" to offset the "climate crisis". the "climate crisis" is something that exists in computer models, few of the models agree, and if they do, it behooves us to look at when they started agreeing and find out why that change happened. a more personal example: I can't reproduce getting CaCO out of seawater, yet.
  - tgv 4y ago
    I think you've never done psych research. First of all, statisticians and methodologists are frequently included, also in the design phase, if it's not a simple adaptation of an existing study. Nonetheless, many papers contain design errors. Second, sociology and anthropology are among the least empirical of academic disciplines. What reason would you have to involve them? Third, many studies are of the type "my theory predicts X and my experiment shows p < 0.05". That's almost wrong by definition. Bayesian theory explains why. Fourth, many studies have conclusions/claims that cannot be inferred from the actual finding. > When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. It is important, but as long as academia is a "turn out papers for tenure" industry, it won't be fixed. Don't take the conclusions of a paper for true. They're meant for other academics.
treeman79 4y ago
It’s not that you can’t trust “science” it’s that people are fallible.
- viridian 4y ago
  The big unaddressed issue is that conclusions based on empirical data can only be as good as the observations. If the data is biased, manipulated, "massaged", etc, there can be no good output from that poisoned tree, and other persons cannot contradict the results without attempting to replicate the entire research process beginning to end. With math, physics, applied math, computer science, etc, bad results can be usually (but not always) deconstructed by anyone with the merest tools of the rational; formal logic, application of various categorical laws, etc. Science as a process is useful, but its conclusions can only be as reliable as what its body of producers are incentivized to produce.
kriops 4y ago
There's a big difference between science and social science. Or science and medicine for that matter. They are different domains that require different models of reasoning, which come with their respective sets of limitations. There's a reason you don't see a crisis in e.g. physics, where their hypotheses are leaps and bounds more testable.
- onos 4y ago
  This failure rate suggests a reason the social sciences don’t “lead anywhere”: If you try to build a theorem off of two earlier results, but these each only have probability p of being right, then the chance your new theorem is correct goes like p^2. For small p this means deductions based on combining results are very small.
- onos 4y ago
  This failure rate suggests a reason the social sciences don’t “lead anywhere”: If you try to build a theorem off of two earlier results, but these each only have probability p of being right, then the chance your new theorem is correct goes like p^2. For small p this means the chance that a deduction is correct is very small.
pfortuny 4y ago
Have you read Ioaniddis’ reply? It is worth the effort.

glofish 4y ago

This followup paper seems to summarize p-values reported in 77K papers and uses the distribution of these p-values to compute the FDR. Alas everyone knows that the p-values reported in accepted (!) papers are questionable at best - any analysis that uses them is on shaky foundation. The misuse of p-values in science is well known and an endemic problem - so what do we learn from a re-analysis of made up numbers?

Manu40 4y ago
Not to be pedantic, but considering how many people I have run into who take all research studies as if they are holy gospel; you may want to reconsider saying "everyone knows". Just my 5 cents on the matter.
- glofish 4y ago
  good point, what I really meant is by now everybody should know that p-values are questionable at best
  - genewitch 4y ago
    can we name one mainstream journalist, or any policy makers that know anything about reproducibility - or, more honestly, the appalling lack thereof - or "p-values are questionable at best"? There's been books, such as "how to lie with statistics" that should be mandatory reading, at least on entry to adulthood. I am one of maybe 3 people i know personally that knows how to actually parse a scientific paper, so when someone starts throwing a half dozen studies at me to back up their point, i can usually - on the scale of quarter hours - nitpick their understanding of the sources they used. This doesn't win anyone friends, and something i've said for nearly 3 decades: Religion and science like to point fingers, but religion seems more likely to bend and change than science. Every time i mention this or an analogue i'm shouted at that it's "actually the opposite". Shut up already, it's science. The science is in. This is science!
    - Manu40 4y ago

6gvONxR4sf7o 4y ago

I'd love to know the correlation between failure to replicate and public virality of findings. I wouldn't be surprised of the more exciting findings replicate at a different rate from average.

H8crilA 4y ago
That's kind of the point. More exciting statements are more often "proven" because they're more often attempted to be proven.

nextos 4y ago

A good way to address these issues is to frame all experiments as multilevel models. See [1] for a long discussion from Andrew Gelman et al on why this is advisable. Surprisingly, many people working in statistics still ignore the James-Stein theorem, which provides a theoretical justification for multilevel models. In layman terms, said theorem shows that if you are simultaneously estimating many random variables you should borrow information across variables [2]. Estimating them one by one is suboptimal and does not minimize the global mean squared error. Multilevel models "shrink" individual effect sizes by looking at the overall distribution of effect sizes and provide much more realistic estimates. [1] https://arxiv.org/abs/0907.2478 https://arxiv.org/abs/0907.2478 [2] https://en.wikipedia.org/wiki/Stein%27s_example https://en.wikipedia.org/wiki/Stein%27s_example

kimukasetsu 4y ago
This. Multilevel models relax the assumption of independent observations by specifying that the measures of repeated experimental units are dependent on each other. It's a way of telling your model that it has less information than it would have if all observations came from independent units. Therefore, standard errors of effects are usually larger. Otherwise, they are biased [1]. Since most researches are not aware of multilevel models, they design their experiments and aggregate their data to fit the independence assumption, which is rarely a good idea. Many are not even aware of modeling beyond hypothesis tests, and are unable or unwilling to adjust their analysis for confounding factors or non-sampling errors that arise due to experiment design flaws. Also, p-values should be deprecated, since a) nil hypothesis are strawmen at best and false by definition at worst [2] and b) they incentivize researchers to not think hard about effect sizes and uncertainty in their problems. [1] https://academic.oup.com/biomet/article/73/1/13/246001 https://academic.oup.com/biomet/article/73/1/13/246001 [2] http://www.stat.columbia.edu/~gelman/research/published/failure_of.pdf http://www.stat.columbia.edu/~gelman/research/published/fail...
- nextos 4y ago
  Many articles in e.g. Nature Genetics are cheating by hacking their p-values. These hacks would be much harder to get away with if, as a starter, they were asked to use hierarchical models and continuous explanatory variables, whenever possible. The article from Andrew Gelman you cited explains this quite well. In general the review articles and books he has co-authored are incredibly helpful to learn how to avoid common issues that plague statistical inference. We need to shift away from null hypotheses and p-values towards generative models, model selection and effect sizes. It leads to much more robust inference.

statheros 4y ago

So the results of this paper are also false... ironic.

thedailymail 4y ago

>>Follow up analysis by Jager and Leek (2014) https://academic.oup.com/biostatistics/article/15/1/1/244509 https://academic.oup.com/biostatistics/article/15/1/1/244509 suggests the false discovery rate is closer to 14% than 50%. Maybe the title should be updated to "Why most published research findings, including this one, are false".

civilized 4y ago

The 2014 paper is just another theoretical analysis that gets a different answer. It's just one cute, provocative theory versus another.

KvanteKat 4y ago

For anyone who likes academic drama (or who is interested in the underlying methodological disagreements among academic statisticians), it is worth pointing out that Jager and Leek's 2014 paper was a discussion paper, and that Ionnidas was one of the people invited to write a response to be published alongside with the original paper. He did (https://doi.org/10.1093/biostatistics/kxt036 https://doi.org/10.1093/biostatistics/kxt036) and the response is extremely critical of Jaeger and Leeks' methodology and his contempt for the authors is not hard to read between the lines.

KvanteKat 4y ago
For anyone who wants to get really into the weeds, here are all the articles in the sequence of discussion papers: Main Paper (Jager and Leek): ttps://doi.org/10.1093/biostatistics/kxt007 Response papers: - Yoav Benjamini and Yotam Hechtlinger: https://doi.org/10.1093/biostatistics/kxt032 - David R. Cox: https://doi.org/10.1093/biostatistics/kxt033 - Andrew Gelman and Keith O'Rourke: https://doi.org/10.1093/biostatistics/kxt034 - Steven N. Goodman: https://doi.org/10.1093/biostatistics/kxt035 - John P. A. Ioannidis (the spicy response): https://doi.org/10.1093/biostatistics/kxt036 - Martijn J. Schuemie, Patrick B. Ryan, Marc A. Suchard, Zach Shahn, and David Madigan: https://doi.org/10.1093/biostatistics/kxt037 Jaeger and Leeks' rejoinder to the responses: https://doi.org/10.1093/biostatistics/kxt038 https://doi.org/10.1093/biostatistics/kxt038 edit: fixed some formatting and link to main paper