7 ms·

As a biomedical scientist at a solid US university, I'm getting pretty fed up with all the attacks on my field's credibility. It's similar to education where ev

As a biomedical scientist at a solid US university, I'm getting pretty fed up with all the attacks on my field's credibility. It's similar to education where everyone moans about poor quality. While i cant speak to the average teacher I can honestly say the average researcher tries very hard to get it right. Remember most people don't become biomedical scientists unless they worked hard for many, many years in school and beyond for often quite terrible wages. So you can hate the game (drug companies, academic journals), but please think through any hate you might have on the player.

capnrefsmmat 10y ago
I don't think the blame can all be pinned on drug companies and academic journals, or whatever other third parties we can identify. They all have a role, but many problems with biomedical science come down to individual researchers making poor decisions about their experimental design, statistical methods, and computational tools. I'm a statistician, and when I wrote my own book about bad statistics in science (see https://www.statisticsdonewrong.com/ https://www.statisticsdonewrong.com/), I made sure to reference studies which quantify how often errors occur in real published research. The rate is stunningly high. The average biomedical experiment is conducted with (a) a sample size which is far too small to detect an effect of the expected size, (b) a vague analysis plan which leads to exploratory analyses with high false positive rates, (c) frequent copy-and-paste errors and math mistakes in presenting important results, and (d) an overreliance on statistical methods to make up for poor experimental design. This means the average published paper is likely a false positive, likely an overestimate of the true effect if not, and is barely reproducible. Is this the fault of individual scientists? Partly, yes -- these problems have been pointed out for years in leading journals, but nobody takes action to do better research. It's also the fault of the grant funding systems which incentivize salami-slicing of results instead of doing one big, rigorous, well-designed study, and of journals which prefer dramatic but unreliable results over mundane but well-executed results. (Of course, the journal editors and reviewers are usually active scientists themselves.) I think the average researcher would like to "get it right", but has to focus on getting a career instead. Just a few papers on the problem of poor sample sizes in biomedicine: http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.2000797 http://journals.plos.org/plosbiology/article?id=10.1371/jour... http://rsos.royalsocietypublishing.org/content/4/2/160254 http://rsos.royalsocietypublishing.org/content/4/2/160254 http://www.nature.com/nrn/journal/v14/n5/full/nrn3475.html http://www.nature.com/nrn/journal/v14/n5/full/nrn3475.html
- AlexCoventry 10y ago
  I think the central issue is that in many subfields of biology, reproducibility is seen as infeasible or unnecessary, so even with the best of intentions researchers can go on fooling themselves about the reliability of their results for years. Plus of course, without the accountability afforded by reproducible results the bad money can end up driving out the good in funding decisions, and then the good intentions get driven out, too. Psychology is going through an epistemological crisis for because reproducibility only recently became a hot topic there, and psychology experiments are super-cheap and easy to replicate compared to typical benchwork. But there have been equally dismal results when there's been sufficient incentive to replicate in other fields. http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a... On the other hand, molecular biology and forward genetics have great traditions of reproducibility, and results in those fields tend to be pretty solid.
  - nonbel 10y ago
    >"molecular biology and forward genetics have great traditions of reproducibility" Where is the evidence of this? There is plenty saying otherwise: http://www.sciencemag.org/news/2015/06/study-claims-28-billion-year-spent-irreproducible-biomedical-research http://www.sciencemag.org/news/2015/06/study-claims-28-billi... http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4270077/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4270077/ http://www.slate.com/articles/health_and_science/future_tense/2016/04/biomedicine_facing_a_worse_replication_crisis_than_the_one_plaguing_psychology.html http://www.slate.com/articles/health_and_science/future_tens... http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.htm... http://www.reuters.com/article/us-science-cancer-idUSBRE82R12P20120328 http://www.reuters.com/article/us-science-cancer-idUSBRE82R1... http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a... http://www.sciencemag.org/content/348/6242/1411 http://www.sciencemag.org/content/348/6242/1411 http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002165 http://journals.plos.org/plosbiology/article?id=10.1371/jour... http://www.nature.com/news/cancer-reproducibility-project-scales-back-ambitions-1.18938 http://www.nature.com/news/cancer-reproducibility-project-sc... http://www.sciencemag.org/news/2017/01/rigorous-replication-effort-succeeds-just-two-five-cancer-papers http://www.sciencemag.org/news/2017/01/rigorous-replication-...
nonbel 10y ago
I was trained in biomed and found it completely unsatisfactory. Simply put, I was trained to do pseudoscience (NHST, where you test a strawman "null" hypothesis rather than your own hypothesis and commit an egregious affirming the consequent error). There are also various other institutionalized satellite issues like the extreme difficulty of publishing replication results. Consider these questions: 1) Can you define a p-value and articulate why you would perform a significance test? 2) How many replication studies have you published? 3) How often do you perform your experiments blinded to intervention? 4) Is there any quantitative prediction that has been made by anyone in your sub-field of research ever? I'm not even asking for an accurate one, just the publication of a point or small-interval prediction.
- toufka 10y ago
  There are other ways to do biomedical science than p-hacking at large data sets. Some of the best science I've seen were experiments where the result was guaranteed to be one of a few different possibilities - and therefore any concrete result was interesting. That's mostly attributable to good experimental design. Cell A does X under condition 1. Cell A does Y under condition 2. What is the protein/pathway/atom/molecule responsible for triggering condition Y under condition 2 (or conversely triggering X under condition 1). Pick your level of inquiry, discover the answer, remove the answer to test for necessity, and add the answer back to a null background to test for sufficiency. If all are true, you have no found something useful and true. Statistics was only a tool to help you accomplish the above and actually had little to no bearing on the knowledge being created. Good experimental design could create situations where the culminating experiment either presented an indicator or did not, confirming the questioned hypothesis true or false. Those kinds of factual discoveries were the biomedical science I was taught to probe for, where whatever the answer was it would be significant, interesting, and useful. And there is no way to successfully conduct the above experiments in such a way that statistics even can be of much relevance to the scientific question itself. Either you discover a mechanism or you do not, and logically it can only be one of a very few options.

capnrefsmmat 10y ago

I don't think the blame can all be pinned on drug companies and academic journals, or whatever other third parties we can identify. They all have a role, but many problems with biomedical science come down to individual researchers making poor decisions about their experimental design, statistical methods, and computational tools. I'm a statistician, and when I wrote my own book about bad statistics in science (see https://www.statisticsdonewrong.com/ https://www.statisticsdonewrong.com/), I made sure to reference studies which quantify how often errors occur in real published research. The rate is stunningly high. The average biomedical experiment is conducted with (a) a sample size which is far too small to detect an effect of the expected size, (b) a vague analysis plan which leads to exploratory analyses with high false positive rates, (c) frequent copy-and-paste errors and math mistakes in presenting important results, and (d) an overreliance on statistical methods to make up for poor experimental design. This means the average published paper is likely a false positive, likely an overestimate of the true effect if not, and is barely reproducible. Is this the fault of individual scientists? Partly, yes -- these problems have been pointed out for years in leading journals, but nobody takes action to do better research. It's also the fault of the grant funding systems which incentivize salami-slicing of results instead of doing one big, rigorous, well-designed study, and of journals which prefer dramatic but unreliable results over mundane but well-executed results. (Of course, the journal editors and reviewers are usually active scientists themselves.) I think the average researcher would like to "get it right", but has to focus on getting a career instead. Just a few papers on the problem of poor sample sizes in biomedicine: http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.2000797 http://journals.plos.org/plosbiology/article?id=10.1371/jour... http://rsos.royalsocietypublishing.org/content/4/2/160254 http://rsos.royalsocietypublishing.org/content/4/2/160254 http://www.nature.com/nrn/journal/v14/n5/full/nrn3475.html http://www.nature.com/nrn/journal/v14/n5/full/nrn3475.html

AlexCoventry 10y ago
I think the central issue is that in many subfields of biology, reproducibility is seen as infeasible or unnecessary, so even with the best of intentions researchers can go on fooling themselves about the reliability of their results for years. Plus of course, without the accountability afforded by reproducible results the bad money can end up driving out the good in funding decisions, and then the good intentions get driven out, too. Psychology is going through an epistemological crisis for because reproducibility only recently became a hot topic there, and psychology experiments are super-cheap and easy to replicate compared to typical benchwork. But there have been equally dismal results when there's been sufficient incentive to replicate in other fields. http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a... On the other hand, molecular biology and forward genetics have great traditions of reproducibility, and results in those fields tend to be pretty solid.
- nonbel 10y ago
  >"molecular biology and forward genetics have great traditions of reproducibility" Where is the evidence of this? There is plenty saying otherwise: http://www.sciencemag.org/news/2015/06/study-claims-28-billion-year-spent-irreproducible-biomedical-research http://www.sciencemag.org/news/2015/06/study-claims-28-billi... http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4270077/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4270077/ http://www.slate.com/articles/health_and_science/future_tense/2016/04/biomedicine_facing_a_worse_replication_crisis_than_the_one_plaguing_psychology.html http://www.slate.com/articles/health_and_science/future_tens... http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.htm... http://www.reuters.com/article/us-science-cancer-idUSBRE82R12P20120328 http://www.reuters.com/article/us-science-cancer-idUSBRE82R1... http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a... http://www.sciencemag.org/content/348/6242/1411 http://www.sciencemag.org/content/348/6242/1411 http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002165 http://journals.plos.org/plosbiology/article?id=10.1371/jour... http://www.nature.com/news/cancer-reproducibility-project-scales-back-ambitions-1.18938 http://www.nature.com/news/cancer-reproducibility-project-sc... http://www.sciencemag.org/news/2017/01/rigorous-replication-effort-succeeds-just-two-five-cancer-papers http://www.sciencemag.org/news/2017/01/rigorous-replication-...

nonbel 10y ago

I was trained in biomed and found it completely unsatisfactory. Simply put, I was trained to do pseudoscience (NHST, where you test a strawman "null" hypothesis rather than your own hypothesis and commit an egregious affirming the consequent error). There are also various other institutionalized satellite issues like the extreme difficulty of publishing replication results. Consider these questions: 1) Can you define a p-value and articulate why you would perform a significance test? 2) How many replication studies have you published? 3) How often do you perform your experiments blinded to intervention? 4) Is there any quantitative prediction that has been made by anyone in your sub-field of research ever? I'm not even asking for an accurate one, just the publication of a point or small-interval prediction.

toufka 10y ago
There are other ways to do biomedical science than p-hacking at large data sets. Some of the best science I've seen were experiments where the result was guaranteed to be one of a few different possibilities - and therefore any concrete result was interesting. That's mostly attributable to good experimental design. Cell A does X under condition 1. Cell A does Y under condition 2. What is the protein/pathway/atom/molecule responsible for triggering condition Y under condition 2 (or conversely triggering X under condition 1). Pick your level of inquiry, discover the answer, remove the answer to test for necessity, and add the answer back to a null background to test for sufficiency. If all are true, you have no found something useful and true. Statistics was only a tool to help you accomplish the above and actually had little to no bearing on the knowledge being created. Good experimental design could create situations where the culminating experiment either presented an indicator or did not, confirming the questioned hypothesis true or false. Those kinds of factual discoveries were the biomedical science I was taught to probe for, where whatever the answer was it would be significant, interesting, and useful. And there is no way to successfully conduct the above experiments in such a way that statistics even can be of much relevance to the scientific question itself. Either you discover a mechanism or you do not, and logically it can only be one of a very few options.