9 ms·

That's a huge difference yet anti-intellectuals will contiune to post this paper until the end of time as proof that you can't trust science

diognesofsinope 4y ago
I mean, the failure of a lot of experiments to replicate is extraordinarily well documented... Not to mention the corollaries from the Wikipedia article (https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...): "In addition to the main result, Ioannidis lists six corollaries for factors that can influence the reliability of published research. Research findings in a scientific field are less likely to be true, 1. the smaller the studies conducted. 2. the smaller the effect sizes. 3. the greater the number and the lesser the selection of tested relationships. 4. the greater the flexibility in designs, definitions, outcomes, and analytical modes. 5. the greater the financial and other interests and prejudices. 6. the hotter the scientific field (with more scientific teams involved)." These are all reasonable criticisms from my own area of expertise: applied econometrics.
- nonameiguess 4y ago
  Being skeptical of cutting edge research is a lot different than distrusting science. There are plenty of people out there denying special relativity, evolution by natural selection, or believing all of western medicine is invalid, based on extrapolation from stuff like this. I won't speak for all textbooks, but generally stuff you find in there should not be the same as what you find in journals, and is much more settled. Big caveat that that isn't necessarily true for younger sciences without long-established theory, say exercise physiology or social psychology, but something like a chemistry textbook is pretty damn trustworthy. And those are what people who aren't actually scientists should mostly be educating themselves with, not newspaper science reporting sections.
  - epolanski 4y ago
    Doubting a scientific paper results is different from distrusting science. Academia is a complex place and it's full of fake results to get publications. (I'm coauthor of several scientific papers and I know for a fact countless highly rated papers in my former field are not reproducible because they come from adjusted numbers).
deleted 4y ago
[deleted]
yucky 4y ago
The Replication Crisis is real, denying that is anti-intellectual. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- Retric 4y ago
  Calling something a crisis is arbitrary. So it both is and isn’t a crisis based on completely arbitrary metrics.
  - yucky 4y ago
    I encourage you to dig in to the sources posted. The majority of published research can't be and/or hasn't been replicated. That's a crisis by any definition, especially with the amount of idiots running around bleating about "the science is settled" or "trust the science" or whatever catch phrase.
    - Retric 4y ago
      The degree to which the science is settled varies wildly by topic and that’s ok. Nutrition is perceived to be filled with a lot of junk, but without dietary vitamin C you will die. That’s settled even inside a field filled with debate. Individual papers where never intended to be the final arbiters of truth, that’s not their role. If nobody thinks things are worth looking into again then stuff stays in a very nebulous state which is no worse than where things where before a paper was published.
      - yucky 4y ago
        >If nobody thinks things are worth looking into again then stuff stays in a very nebulous state which is no worse than where things where before a paper was published. I don't think so, because of the aforementioned idiots who take a study that can't even be replicated but then cite the paper as evidence for something. If it can't be replicated, logic would say there is a very good chance it's inaccurate. Yet it's treated as being "settled science" which is worse. Also, many times it isn't that nobody thinks it's worth looking into again, it's that coming to "the wrong conclusion" on some things will end their careers so they'd rather play it safe and just keep sucking up grant money to pump out more of the same garbage.
deleted 4y ago
[deleted]
serial_dev 4y ago
Mentioning that not everything that's published in a peer reviewed publication is automatically 100% correct is not anti-intellectual. Mistakes can be made, data can be limited or misinterpreted, scientists can be corrupted. Theories, "common sense" can change: what we thought we'd know for sure have been proven wrong. If you are a scientist, you know that. Only because I don't automatically take every "scientific" finding at face value, it doesn't mean I don't trust science. I trust science, I just don't trust every single research, experiment, scientist, or journal. Actually that's in itself, in a way, science.
- afpx 4y ago
  “Science” currently incentivizes the wrong things. How do you change the incentives (back)?
  - chithanh 4y ago
    I think it is a misconception that switching to the "right" incentives will solve problems. The very existence of incentives is not conducive for knowledge-based work. https://hbr.org/1993/09/why-incentive-plans-cannot-work https://hbr.org/1993/09/why-incentive-plans-cannot-work
- hbn 4y ago
  Yup. Blindly "trusting the science" is inherently anti-science There is plenty of things that we took as scientific fact in the past that turned out to be false. Science is supposed to question existing notions and test/prove new theories. If you just assume that we're in a post-science era where we've got it all figured out, and our current theories are all correct, that's not science, it's dogmatic faith. Ignaz Semmelweis was destroyed by the medical community for his insane idea that doctors should wash their hands before performing medical procedures. He was attacked about to the point where he ended up having a nervous breakdown and was committed to an asylum where he was beaten to death by the guards. Serves him right for questioning the science!
  - dekhn 4y ago
timr 4y ago
It's also wrong. There have now been large-scale initiatives to reproduce papers, and they're getting higher failure rates than 14%. The statistics underlying all of this make a number of implicit assumptions that may or may not be true IRL (for example: that journals publish papers independent of the salaciousness of claims made within them). If Science picks out only the top-5% most-sensational claims for publication, then you can't assume that a 95% CI is a safe threshold. You've probably got to increase it to a much higher value to have any prayer of getting past the inherent bias in such a process.
- genewitch 4y ago
  Wait until you start looking in to, say, psychology papers, vis a vis reproducibility. I think that a lot of these studies should be reviewed during the actual research and study part by at least a sociologist or cultural anthropologist, and a mathematician with expertise in probability. When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. For a recent example, the US White house is engaging in a 5 year study to determine if releasing sulfur dioxide into the stratosphere would accomplish enough "cooling" to offset the "climate crisis". the "climate crisis" is something that exists in computer models, few of the models agree, and if they do, it behooves us to look at when they started agreeing and find out why that change happened. a more personal example: I can't reproduce getting CaCO out of seawater, yet.
  - tgv 4y ago
    I think you've never done psych research. First of all, statisticians and methodologists are frequently included, also in the design phase, if it's not a simple adaptation of an existing study. Nonetheless, many papers contain design errors. Second, sociology and anthropology are among the least empirical of academic disciplines. What reason would you have to involve them? Third, many studies are of the type "my theory predicts X and my experiment shows p < 0.05". That's almost wrong by definition. Bayesian theory explains why. Fourth, many studies have conclusions/claims that cannot be inferred from the actual finding. > When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. It is important, but as long as academia is a "turn out papers for tenure" industry, it won't be fixed. Don't take the conclusions of a paper for true. They're meant for other academics.
treeman79 4y ago
It’s not that you can’t trust “science” it’s that people are fallible.
- viridian 4y ago
  The big unaddressed issue is that conclusions based on empirical data can only be as good as the observations. If the data is biased, manipulated, "massaged", etc, there can be no good output from that poisoned tree, and other persons cannot contradict the results without attempting to replicate the entire research process beginning to end. With math, physics, applied math, computer science, etc, bad results can be usually (but not always) deconstructed by anyone with the merest tools of the rational; formal logic, application of various categorical laws, etc. Science as a process is useful, but its conclusions can only be as reliable as what its body of producers are incentivized to produce.
kriops 4y ago
There's a big difference between science and social science. Or science and medicine for that matter. They are different domains that require different models of reasoning, which come with their respective sets of limitations. There's a reason you don't see a crisis in e.g. physics, where their hypotheses are leaps and bounds more testable.
- onos 4y ago
  This failure rate suggests a reason the social sciences don’t “lead anywhere”: If you try to build a theorem off of two earlier results, but these each only have probability p of being right, then the chance your new theorem is correct goes like p^2. For small p this means deductions based on combining results are very small.
- onos 4y ago
  This failure rate suggests a reason the social sciences don’t “lead anywhere”: If you try to build a theorem off of two earlier results, but these each only have probability p of being right, then the chance your new theorem is correct goes like p^2. For small p this means the chance that a deduction is correct is very small.
pfortuny 4y ago
Have you read Ioaniddis’ reply? It is worth the effort.

diognesofsinope 4y ago

I mean, the failure of a lot of experiments to replicate is extraordinarily well documented... Not to mention the corollaries from the Wikipedia article (https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...): "In addition to the main result, Ioannidis lists six corollaries for factors that can influence the reliability of published research. Research findings in a scientific field are less likely to be true, 1. the smaller the studies conducted. 2. the smaller the effect sizes. 3. the greater the number and the lesser the selection of tested relationships. 4. the greater the flexibility in designs, definitions, outcomes, and analytical modes. 5. the greater the financial and other interests and prejudices. 6. the hotter the scientific field (with more scientific teams involved)." These are all reasonable criticisms from my own area of expertise: applied econometrics.

nonameiguess 4y ago
Being skeptical of cutting edge research is a lot different than distrusting science. There are plenty of people out there denying special relativity, evolution by natural selection, or believing all of western medicine is invalid, based on extrapolation from stuff like this. I won't speak for all textbooks, but generally stuff you find in there should not be the same as what you find in journals, and is much more settled. Big caveat that that isn't necessarily true for younger sciences without long-established theory, say exercise physiology or social psychology, but something like a chemistry textbook is pretty damn trustworthy. And those are what people who aren't actually scientists should mostly be educating themselves with, not newspaper science reporting sections.
- epolanski 4y ago
  Doubting a scientific paper results is different from distrusting science. Academia is a complex place and it's full of fake results to get publications. (I'm coauthor of several scientific papers and I know for a fact countless highly rated papers in my former field are not reproducible because they come from adjusted numbers).

deleted 4y ago

[deleted]

yucky 4y ago

The Replication Crisis is real, denying that is anti-intellectual. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis

Retric 4y ago
Calling something a crisis is arbitrary. So it both is and isn’t a crisis based on completely arbitrary metrics.
- yucky 4y ago
  I encourage you to dig in to the sources posted. The majority of published research can't be and/or hasn't been replicated. That's a crisis by any definition, especially with the amount of idiots running around bleating about "the science is settled" or "trust the science" or whatever catch phrase.
  - Retric 4y ago
    The degree to which the science is settled varies wildly by topic and that’s ok. Nutrition is perceived to be filled with a lot of junk, but without dietary vitamin C you will die. That’s settled even inside a field filled with debate. Individual papers where never intended to be the final arbiters of truth, that’s not their role. If nobody thinks things are worth looking into again then stuff stays in a very nebulous state which is no worse than where things where before a paper was published.
    - yucky 4y ago
      >If nobody thinks things are worth looking into again then stuff stays in a very nebulous state which is no worse than where things where before a paper was published. I don't think so, because of the aforementioned idiots who take a study that can't even be replicated but then cite the paper as evidence for something. If it can't be replicated, logic would say there is a very good chance it's inaccurate. Yet it's treated as being "settled science" which is worse. Also, many times it isn't that nobody thinks it's worth looking into again, it's that coming to "the wrong conclusion" on some things will end their careers so they'd rather play it safe and just keep sucking up grant money to pump out more of the same garbage.

deleted 4y ago

[deleted]

serial_dev 4y ago

Mentioning that not everything that's published in a peer reviewed publication is automatically 100% correct is not anti-intellectual. Mistakes can be made, data can be limited or misinterpreted, scientists can be corrupted. Theories, "common sense" can change: what we thought we'd know for sure have been proven wrong. If you are a scientist, you know that. Only because I don't automatically take every "scientific" finding at face value, it doesn't mean I don't trust science. I trust science, I just don't trust every single research, experiment, scientist, or journal. Actually that's in itself, in a way, science.

afpx 4y ago
“Science” currently incentivizes the wrong things. How do you change the incentives (back)?
- chithanh 4y ago
  I think it is a misconception that switching to the "right" incentives will solve problems. The very existence of incentives is not conducive for knowledge-based work. https://hbr.org/1993/09/why-incentive-plans-cannot-work https://hbr.org/1993/09/why-incentive-plans-cannot-work
hbn 4y ago
Yup. Blindly "trusting the science" is inherently anti-science There is plenty of things that we took as scientific fact in the past that turned out to be false. Science is supposed to question existing notions and test/prove new theories. If you just assume that we're in a post-science era where we've got it all figured out, and our current theories are all correct, that's not science, it's dogmatic faith. Ignaz Semmelweis was destroyed by the medical community for his insane idea that doctors should wash their hands before performing medical procedures. He was attacked about to the point where he ended up having a nervous breakdown and was committed to an asylum where he was beaten to death by the guards. Serves him right for questioning the science!
- dekhn 4y ago

timr 4y ago

It's also wrong. There have now been large-scale initiatives to reproduce papers, and they're getting higher failure rates than 14%. The statistics underlying all of this make a number of implicit assumptions that may or may not be true IRL (for example: that journals publish papers independent of the salaciousness of claims made within them). If Science picks out only the top-5% most-sensational claims for publication, then you can't assume that a 95% CI is a safe threshold. You've probably got to increase it to a much higher value to have any prayer of getting past the inherent bias in such a process.

genewitch 4y ago
Wait until you start looking in to, say, psychology papers, vis a vis reproducibility. I think that a lot of these studies should be reviewed during the actual research and study part by at least a sociologist or cultural anthropologist, and a mathematician with expertise in probability. When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. For a recent example, the US White house is engaging in a 5 year study to determine if releasing sulfur dioxide into the stratosphere would accomplish enough "cooling" to offset the "climate crisis". the "climate crisis" is something that exists in computer models, few of the models agree, and if they do, it behooves us to look at when they started agreeing and find out why that change happened. a more personal example: I can't reproduce getting CaCO out of seawater, yet.
- tgv 4y ago
  I think you've never done psych research. First of all, statisticians and methodologists are frequently included, also in the design phase, if it's not a simple adaptation of an existing study. Nonetheless, many papers contain design errors. Second, sociology and anthropology are among the least empirical of academic disciplines. What reason would you have to involve them? Third, many studies are of the type "my theory predicts X and my experiment shows p < 0.05". That's almost wrong by definition. Bayesian theory explains why. Fourth, many studies have conclusions/claims that cannot be inferred from the actual finding. > When nearly every policy can be backed by "some papers published in <some journal>", stuff like this becomes important. It is important, but as long as academia is a "turn out papers for tenure" industry, it won't be fixed. Don't take the conclusions of a paper for true. They're meant for other academics.

treeman79 4y ago

It’s not that you can’t trust “science” it’s that people are fallible.

viridian 4y ago
The big unaddressed issue is that conclusions based on empirical data can only be as good as the observations. If the data is biased, manipulated, "massaged", etc, there can be no good output from that poisoned tree, and other persons cannot contradict the results without attempting to replicate the entire research process beginning to end. With math, physics, applied math, computer science, etc, bad results can be usually (but not always) deconstructed by anyone with the merest tools of the rational; formal logic, application of various categorical laws, etc. Science as a process is useful, but its conclusions can only be as reliable as what its body of producers are incentivized to produce.

kriops 4y ago

There's a big difference between science and social science. Or science and medicine for that matter. They are different domains that require different models of reasoning, which come with their respective sets of limitations. There's a reason you don't see a crisis in e.g. physics, where their hypotheses are leaps and bounds more testable.

onos 4y ago
This failure rate suggests a reason the social sciences don’t “lead anywhere”: If you try to build a theorem off of two earlier results, but these each only have probability p of being right, then the chance your new theorem is correct goes like p^2. For small p this means deductions based on combining results are very small.
onos 4y ago
This failure rate suggests a reason the social sciences don’t “lead anywhere”: If you try to build a theorem off of two earlier results, but these each only have probability p of being right, then the chance your new theorem is correct goes like p^2. For small p this means the chance that a deduction is correct is very small.

pfortuny 4y ago

Have you read Ioaniddis’ reply? It is worth the effort.