7 ms·

I don't think the blame can all be pinned on drug companies and academic journals, or whatever other third parties we can identify. They all have a role, but ma

I don't think the blame can all be pinned on drug companies and academic journals, or whatever other third parties we can identify. They all have a role, but many problems with biomedical science come down to individual researchers making poor decisions about their experimental design, statistical methods, and computational tools. I'm a statistician, and when I wrote my own book about bad statistics in science (see https://www.statisticsdonewrong.com/ https://www.statisticsdonewrong.com/), I made sure to reference studies which quantify how often errors occur in real published research. The rate is stunningly high. The average biomedical experiment is conducted with (a) a sample size which is far too small to detect an effect of the expected size, (b) a vague analysis plan which leads to exploratory analyses with high false positive rates, (c) frequent copy-and-paste errors and math mistakes in presenting important results, and (d) an overreliance on statistical methods to make up for poor experimental design. This means the average published paper is likely a false positive, likely an overestimate of the true effect if not, and is barely reproducible. Is this the fault of individual scientists? Partly, yes -- these problems have been pointed out for years in leading journals, but nobody takes action to do better research. It's also the fault of the grant funding systems which incentivize salami-slicing of results instead of doing one big, rigorous, well-designed study, and of journals which prefer dramatic but unreliable results over mundane but well-executed results. (Of course, the journal editors and reviewers are usually active scientists themselves.) I think the average researcher would like to "get it right", but has to focus on getting a career instead. Just a few papers on the problem of poor sample sizes in biomedicine: http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.2000797 http://journals.plos.org/plosbiology/article?id=10.1371/jour... http://rsos.royalsocietypublishing.org/content/4/2/160254 http://rsos.royalsocietypublishing.org/content/4/2/160254 http://www.nature.com/nrn/journal/v14/n5/full/nrn3475.html http://www.nature.com/nrn/journal/v14/n5/full/nrn3475.html

AlexCoventry 10y ago
I think the central issue is that in many subfields of biology, reproducibility is seen as infeasible or unnecessary, so even with the best of intentions researchers can go on fooling themselves about the reliability of their results for years. Plus of course, without the accountability afforded by reproducible results the bad money can end up driving out the good in funding decisions, and then the good intentions get driven out, too. Psychology is going through an epistemological crisis for because reproducibility only recently became a hot topic there, and psychology experiments are super-cheap and easy to replicate compared to typical benchwork. But there have been equally dismal results when there's been sufficient incentive to replicate in other fields. http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a... On the other hand, molecular biology and forward genetics have great traditions of reproducibility, and results in those fields tend to be pretty solid.
- nonbel 10y ago
  >"molecular biology and forward genetics have great traditions of reproducibility" Where is the evidence of this? There is plenty saying otherwise: http://www.sciencemag.org/news/2015/06/study-claims-28-billion-year-spent-irreproducible-biomedical-research http://www.sciencemag.org/news/2015/06/study-claims-28-billi... http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4270077/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4270077/ http://www.slate.com/articles/health_and_science/future_tense/2016/04/biomedicine_facing_a_worse_replication_crisis_than_the_one_plaguing_psychology.html http://www.slate.com/articles/health_and_science/future_tens... http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.htm... http://www.reuters.com/article/us-science-cancer-idUSBRE82R12P20120328 http://www.reuters.com/article/us-science-cancer-idUSBRE82R1... http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a... http://www.sciencemag.org/content/348/6242/1411 http://www.sciencemag.org/content/348/6242/1411 http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002165 http://journals.plos.org/plosbiology/article?id=10.1371/jour... http://www.nature.com/news/cancer-reproducibility-project-scales-back-ambitions-1.18938 http://www.nature.com/news/cancer-reproducibility-project-sc... http://www.sciencemag.org/news/2017/01/rigorous-replication-effort-succeeds-just-two-five-cancer-papers http://www.sciencemag.org/news/2017/01/rigorous-replication-...
  - AlexCoventry 10y ago
    I actually cited the seminal cancer-biology editorial in the GP. I suppose cancer biology is often a form of molecular biology, but I meant more fundamental research into molecular structure and biological function, and in-vitro methods for querying and modifying the structure. CRISPR is a good example.
    - nonbel 10y ago
      CRISPR is a great example. Please see my other post on that topic in this thread, and if you are interested I will go find the refs that led me to that position.
untilHellbanned 10y ago
I appreciate your comments but as the biologist hopefully you can understand that it's not like we are trying to keep our sample sizes small. We understand statistical power. It's simply not feasible logistically and financially to have the ideal numbers a lot of the time. Real life intervenes, so we do our best. For math/computer people which is probably many here in HN, I think they have a hard time with real-world messiness that is biomedical research. It's the same mentality of "oh I'll just apply my algorithms to this..." which works for them but not so much for us.
- capnrefsmmat 10y ago
  Given how infrequently we see power calculations in published papers, and the results of typical surveys of scientists, I don't think the typical scientist does understand statistical power or other statistical concepts. There are plenty of surveys (e.g. http://www.tandfonline.com/doi/abs/10.1080/17470218.2014.885986 http://www.tandfonline.com/doi/abs/10.1080/17470218.2014.885...) showing that scientists just pick sample sizes based on what's usually done in their field, rather than working out what's necessary. There are plenty of other surveys showing practicing scientists don't understand the concepts of significance and power. Some of the authors of those studies don't understand them either -- I've seen at least one study with questions with no correct answer... I know large sample sizes are often impractical, and you have to make do. But given that, results must be presented with all sorts of disclaimers, since results from underpowered studies are frequently wrong or exaggerated (https://www.ncbi.nlm.nih.gov/pubmed/18633328 https://www.ncbi.nlm.nih.gov/pubmed/18633328). That's not what happens -- scientists who should be aware that their studies have very little value as evidence instead present them as groundbreaking definitive results. I would much rather see slow, difficult, tentative studies instead of the high-speed spray of nearly meaningless results we see every day. But that will take a dramatic change in career and funding incentives.
  - toufka 10y ago
    >I would much rather see slow, difficult, tentative studies instead of the high-speed spray of nearly meaningless results we see every day. A major hurdle is the experiments do keep getting slower and slower. As a recently graduated biomedical grad student I spent literally 6.5 years of my life trying to get an experiment to work enough times to glean useful data out of it. I knew exactly how much power my data had (I had literally years to think and rethink about it). The number of stars that had to align to get equipment, protocols, controls, materials, animals, sleep-schedules, etc. all aligned to get data out of an incredibly complicated system meant that there is no fast data. Useful biological data is hard-won. And it's getting harder and harder (read, more expensive and labor-intensive). You could say, well, then just don't expect every scientist to do their own (independent) research (see the author lists on CERN publications...). That indeed would change incentives a lot in the biomedical fields - and I actually do think for the better. The last point is I think there's a big difference between how a practicing scientist sees a 'published paper' in their field, and how everyone else sees it. A published paper really is just a mark of what you did, and what happened, and maybe a basic interpretation. It really does not claim to be 'true' in a strong sense. And there are entirely reasonable situations where different papers come to opposite (justifiable) conclusions. This is itself weighted data to be used in the next motions of the field precisely because sample size and (more often than not) experimental complexity makes clarity hard to come by for a single paper.

AlexCoventry 10y ago

I think the central issue is that in many subfields of biology, reproducibility is seen as infeasible or unnecessary, so even with the best of intentions researchers can go on fooling themselves about the reliability of their results for years. Plus of course, without the accountability afforded by reproducible results the bad money can end up driving out the good in funding decisions, and then the good intentions get driven out, too. Psychology is going through an epistemological crisis for because reproducibility only recently became a hot topic there, and psychology experiments are super-cheap and easy to replicate compared to typical benchwork. But there have been equally dismal results when there's been sufficient incentive to replicate in other fields. http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a... On the other hand, molecular biology and forward genetics have great traditions of reproducibility, and results in those fields tend to be pretty solid.

nonbel 10y ago
>"molecular biology and forward genetics have great traditions of reproducibility" Where is the evidence of this? There is plenty saying otherwise: http://www.sciencemag.org/news/2015/06/study-claims-28-billion-year-spent-irreproducible-biomedical-research http://www.sciencemag.org/news/2015/06/study-claims-28-billi... http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4270077/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4270077/ http://www.slate.com/articles/health_and_science/future_tense/2016/04/biomedicine_facing_a_worse_replication_crisis_than_the_one_plaguing_psychology.html http://www.slate.com/articles/health_and_science/future_tens... http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.htm... http://www.reuters.com/article/us-science-cancer-idUSBRE82R12P20120328 http://www.reuters.com/article/us-science-cancer-idUSBRE82R1... http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a... http://www.sciencemag.org/content/348/6242/1411 http://www.sciencemag.org/content/348/6242/1411 http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002165 http://journals.plos.org/plosbiology/article?id=10.1371/jour... http://www.nature.com/news/cancer-reproducibility-project-scales-back-ambitions-1.18938 http://www.nature.com/news/cancer-reproducibility-project-sc... http://www.sciencemag.org/news/2017/01/rigorous-replication-effort-succeeds-just-two-five-cancer-papers http://www.sciencemag.org/news/2017/01/rigorous-replication-...
- AlexCoventry 10y ago
  I actually cited the seminal cancer-biology editorial in the GP. I suppose cancer biology is often a form of molecular biology, but I meant more fundamental research into molecular structure and biological function, and in-vitro methods for querying and modifying the structure. CRISPR is a good example.
  - nonbel 10y ago
    CRISPR is a great example. Please see my other post on that topic in this thread, and if you are interested I will go find the refs that led me to that position.

untilHellbanned 10y ago

I appreciate your comments but as the biologist hopefully you can understand that it's not like we are trying to keep our sample sizes small. We understand statistical power. It's simply not feasible logistically and financially to have the ideal numbers a lot of the time. Real life intervenes, so we do our best. For math/computer people which is probably many here in HN, I think they have a hard time with real-world messiness that is biomedical research. It's the same mentality of "oh I'll just apply my algorithms to this..." which works for them but not so much for us.

capnrefsmmat 10y ago
Given how infrequently we see power calculations in published papers, and the results of typical surveys of scientists, I don't think the typical scientist does understand statistical power or other statistical concepts. There are plenty of surveys (e.g. http://www.tandfonline.com/doi/abs/10.1080/17470218.2014.885986 http://www.tandfonline.com/doi/abs/10.1080/17470218.2014.885...) showing that scientists just pick sample sizes based on what's usually done in their field, rather than working out what's necessary. There are plenty of other surveys showing practicing scientists don't understand the concepts of significance and power. Some of the authors of those studies don't understand them either -- I've seen at least one study with questions with no correct answer... I know large sample sizes are often impractical, and you have to make do. But given that, results must be presented with all sorts of disclaimers, since results from underpowered studies are frequently wrong or exaggerated (https://www.ncbi.nlm.nih.gov/pubmed/18633328 https://www.ncbi.nlm.nih.gov/pubmed/18633328). That's not what happens -- scientists who should be aware that their studies have very little value as evidence instead present them as groundbreaking definitive results. I would much rather see slow, difficult, tentative studies instead of the high-speed spray of nearly meaningless results we see every day. But that will take a dramatic change in career and funding incentives.
- toufka 10y ago
  >I would much rather see slow, difficult, tentative studies instead of the high-speed spray of nearly meaningless results we see every day. A major hurdle is the experiments do keep getting slower and slower. As a recently graduated biomedical grad student I spent literally 6.5 years of my life trying to get an experiment to work enough times to glean useful data out of it. I knew exactly how much power my data had (I had literally years to think and rethink about it). The number of stars that had to align to get equipment, protocols, controls, materials, animals, sleep-schedules, etc. all aligned to get data out of an incredibly complicated system meant that there is no fast data. Useful biological data is hard-won. And it's getting harder and harder (read, more expensive and labor-intensive). You could say, well, then just don't expect every scientist to do their own (independent) research (see the author lists on CERN publications...). That indeed would change incentives a lot in the biomedical fields - and I actually do think for the better. The last point is I think there's a big difference between how a practicing scientist sees a 'published paper' in their field, and how everyone else sees it. A published paper really is just a mark of what you did, and what happened, and maybe a basic interpretation. It really does not claim to be 'true' in a strong sense. And there are entirely reasonable situations where different papers come to opposite (justifiable) conclusions. This is itself weighted data to be used in the next motions of the field precisely because sample size and (more often than not) experimental complexity makes clarity hard to come by for a single paper.