12 ms·

More than half of high-impact cancer lab studies could not be replicated

194 points by kevin_hu 5y ago

robbedpeter 5y ago
In mice? Also, the lab animal telomere issue has corrupted almost all animal studies in the last 50 years, but there doesn't seem to be any effort to correct course, or even to acknowledge the problem.
- ramraj07 5y ago
  What lab animal telomere issue? Never heard of that in my decade of working in a lab.
  - hall0ween 5y ago
    Also curious to know.
  - Semaphor 5y ago
    This seems relevant both for the links and the rebuttal (?): https://biology.stackexchange.com/questions/90609/why-might-long-telomeres-be-selected-for-in-laboratory-mice https://biology.stackexchange.com/questions/90609/why-might-...
  - alextheparrot 5y ago
    Probably regurgitating it from the Weinstein stuff. https://uncancelclub.com/bret-weinstein-1/ https://uncancelclub.com/bret-weinstein-1/
  - ashwal 5y ago
    They're probably referring to: https://pubmed.ncbi.nlm.nih.gov/11909679/ https://pubmed.ncbi.nlm.nih.gov/11909679/ I think this is one symptom of a much greater issue: the inbred mice strains we use are disastrously weird (arising from the initial population artificially bred to make research easier). For the sake of controlling for genetics, we've chosen to make lab research translation drastically less effective.
  - KenoFischer 5y ago
    I'm assuming OP is referring to this: https://oncobites.blog/2020/07/29/modeling-aging-and-cancer-are-lab-mice-different-from-their-wild-cousins/ https://oncobites.blog/2020/07/29/modeling-aging-and-cancer-...
  - robbedpeter 5y ago
    https://www.nobelprize.org/prizes/medicine/2009/illustrated-information/#:~:text=The%20Nobel%20Assembly%20at%20Karolinska,telomeres%20and%20the%20enzyme%20telomerase https://www.nobelprize.org/prizes/medicine/2009/illustrated-... 2009 Nobel prize given for this work. Lab animals, due to the way they are bred, are essentially susceptible to cancer and health issues not seen in normal wildtype populations. Shortened telomeres are the primary danger, increasing mutation rate and cell senescence. Essentially, all data involving the expectation of normal cellular behavior is subject to question - signaling from lab animals is corrupted.
    - pazimzadeh 5y ago
      Do you have a source because as I understand it the length of the telomeres is reset each generation and so generally not transmissible to the next generation. Apparently leukocyte telomere length is heritable though, is that what you are talking about?
      - robbedpeter 5y ago
        https://pubmed.ncbi.nlm.nih.gov/11909679/ https://pubmed.ncbi.nlm.nih.gov/11909679/ https://www.gwern.net/docs/longevity/2002-weinstein.pdf https://www.gwern.net/docs/longevity/2002-weinstein.pdf
        ramraj07 5y ago
        What exactly are you interpreting out of this paper now ? How does this invalidate cancer studies in mice? Most cancer studies in mice are xenografts of human cancer cells so any relevance of mice telomeres is completely lost anyway. Also this obsession with telomeres being the fundamental key for all our biological problems seems so conspiratorial. Like you think our cells which do unimaginable things can't keep their DNA ends frothy? Like that's what is gonna spell their doom? Or you consider telomeres are just a symptom of a cells longevity decided by other factors ?
    - AceJohnny2 5y ago
      The Nobel prize is about discovery of the telomeres, not about anything about lab populations.
smegsicle 5y ago
Cancer as entropy- the plot thickens
- lelanthran 5y ago
  > Cancer as entropy- the plot thickens Isn't cancer basically the result of entropy? With enough imperfect cell-divisions, sooner or later one of them is going to have a mutation that causes unbound cell-division (i.e. growth).
deleted 5y ago
[deleted]
Victerius 5y ago
I wrote a brief comment in a previous thread about some of the causes of this replicability crisis: https://news.ycombinator.com/item?id=29478946 https://news.ycombinator.com/item?id=29478946 Tl;dr misaligned incentives in science, the academic job market, and the funding system, the publish or perish dogma, and an overproduction of PhDs.
- smegsicle 5y ago
  What's your connection to the industry, that you recommend fewer scientists and greater centralization of research?
ashwal 5y ago
There's many papers showing similar. (My personal favorite is https://www.science.org/doi/10.1126/scitranslmed.aaw8412 https://www.science.org/doi/10.1126/scitranslmed.aaw8412 where they removed the claimed target for various cancer therapies and many still worked) If you've ever been within a mile of a research lab this shouldn't be remotely shocking. Typical research is done rapidly, sloppily, and in a context where getting the "right" answer is the only incentive. The replication crisis that hit psych is only being held back by (IMO) that bio is harder to replicate due to tooling, protocols, reagents, cell lines, etc. In complex systems you have the degrees of freedom to be wrong at a scale that folks still do not appreciate.
- kergonath 5y ago
  > If you've ever been within a mile of a research lab this shouldn't be remotely shocking. Typical research is done rapidly, sloppily, and in a context where getting the "right" answer is the only incentive. Let me preface that by saying that 50% sounds like an awful lot, even bearing in mind the following. On top of that, these studies focus on highly complicated phenomena, where almost anything could play a role. It is common that one factor that is not necessarily accounted for in a study turns out to be important, even in well-run studies. Flukes happen as well. Failure to replicate per se is not a problem, if the original study was done well and honestly. It is still some data points that could be re interpreted later in the light of other studies. Now, the fact that there is so little incentive to publish confirmation studies is a big problem, because it means that these non-reproducible works are not as challenged as they should be. Remember that one study is meaningless. Even with tiny rates of false positive or negative, erroneous conclusions are bound to happen at some point. A result is not significant unless it has been observed independently by different research groups.
  - hdjjhhvvhga 5y ago
    > the fact that there is so little incentive to publish confirmation studies is a big problem I don't quite get this point: this is cancer research, co in theory all related institutions should be interested in replicating breakthrough studies, right? And failure to replicate it, time after time, should be a cause of concern, shouldn't it?
time_to_smile 5y ago
It's funny how early 20th century science was filled with mind blowing discoveries and realizations that fundamentally changed how we viewed the universe and to some extent even life itself, while the early 21st century is filled with a surprisingly number of boring work churned out by overworked grad students, that turns out to be, in unsettling quantities, largely bad, irreproducible science. People love to blame bad statistics, but it's much more obvious that the issue is the "publish or perish" culture and a perpetual race to manufacture credibility in academia. If you're curious you can trace some of this back to the 70s when peer review as we know it today was essentially born, precisely as a means of proving credibility during a period of academic funding crunches. As much as it's easy to point a finger at academic I increasingly find the same thing in industry. Within my own career the number of companies that are doing nothing more than finding elaborate ways to burn investor funding (without even realizing they are participating in this) has grown to consume almost the entire tech industry. At this point the most shocking news isn't fraud, it would be to find someone out their down real, meaningful work.
- wilkommen 5y ago
  I agree with this. I’ve been coming to similar conclusions myself.
- tekstar 5y ago
  If you have not, you should read The Idea Factory. https://www.amazon.com/Idea-Factory-Great-American-Innovation/dp/0143122797 https://www.amazon.com/Idea-Factory-Great-American-Innovatio... We owe most of our "modern world" to the hard science done at Bell Labs in the 1900s, before the corporation was singularly focused on quarterly profits.
- malwarebytess 5y ago
  I really do believe that the attempt to industrialize scientific research has failed. Science cannot be done at scale. It comes from singularly focused projects lead by pure inquiry. I don't regret my PhD because I enjoyed achieving it, but I see it as a waste of time. There's an analogy in Software Engineering. Programmers are akin to Artisans or Craftsmen, but the industry wants and needs them to be something more like little factories. I don't think that approach makes good software, even if it's a good product. Just as I don't think the system we have created in academia doesn't produce good research, even though it makes good careers.
farseer 5y ago
More than half of high-impact cancer lab studies can be replicated?
- sebow 5y ago
  More than half of all studies can be replicated?
deleted 5y ago
[deleted]
epistasis 5y ago
> Additional problems surfaced when labs began experiments, such as tumor cells that did not behave as expected in a baseline study. Given the various difficulties of culturing cell lines, this is not surprising. What is infuriating to me is unresponsive author queries about data and protocols. That should result in some sort of ding to the paper on the journal's website.
a-dub 5y ago
i've said it before and i'll say it again, this isn't going to improve until there are more grant dollars and recognition for replication than there is for novel discovery. replication is boring, yet critical, and we should be compensating highly skilled scientists for doing it.
- glial 5y ago
  100% agree. The NIH could replicating studies a requirement of funding for graduate or postdoctoral training. In addition to the scientific and societal benefit, it would just be good training.
  - a-dub 5y ago
    your first task in lab will be to reproduce our competitor's work! in seriousness though, there's enough money and trained scientists looking for work... moreover i think there's a whole personality type that is prevalent in science that would be very happy to replicate and reproduce results...
  - tsbischof 5y ago
    I would be glad to have funding explicitly set towards appropriate and open documentation of the actual protocols, equipment, and materials, plus a combined effort to enhance these reporting requirements by the journal. There has to be a balance between the one-off interesting new findings and the large-scale work, but clinical trials definitely fall under the latter category.
- bmitc 5y ago
  That sounds like spending money on trying to replicate results that can’t be replicated doesn’t it? Isn’t that what this study was doing? Doesn’t it make more sense to revisit how the existing funding is spent? I have very little insight into this area of research, so I’d just have to guess everyone’s in a hurry and reaching for results or published papers.
  - a-dub 5y ago
timr 5y ago
I'm shocked that close to half can be replicated. It's not a matter of academic misconduct -- biology is just hard. There are a lot of ways to fool yourself. If you doubt, just take look at the number of drug candidates that make it out of the lab (it's a LOT lower than 50%). The journalism side of Science magazine is far too credulous. Most science is wrong. Even when it is published in Science (perhaps, especially when it is published in Science).
- throwawaylinux 5y ago
  All scientific fields are hard though. The way they are advanced is with rigorous methods. I'm not shocked either though. Not because biology is hard, but because academia is corrupt.
  - soco 5y ago
    Hanlon's razor: never attribute to malice that which is adequately explained by stupidity. Not that academics were stupid, but making mistakes usually obvious in hindsight. And yeah there would be some malice as well, as everywhere, including here, because everybody's a human until those rigorous methods weed humanity out of the processes (way to go).
    - native_samples 5y ago
      Many of the studies couldn't be replicated because the original authors didn't provide sufficient details in their papers, and then refused to reply when asked for those details. Whether this is malice or not is kind of irrelevant - they have failed to meet the most basic requirements expected of researchers. They took the money but did not deliver the expected work artifacts.
  - ansgri 5y ago
    Not that it's corrupt, but incentive structure is suboptimal to say the least, especially for government-funded work. There was a recent blog post / HN thread [1] about groups never admitting failure, it's very relevant here. The funding agency has to have a process for vetting applications, and what's more reasonable than the ability to manage their own work, i.e. propose a plan and successfully execute on it? Combine it with slow cadence of funding and you have a list of promised results for next 3+ years which you must achieve somehow or you'll never see a grant from that agency again. And they give out a lot of grants, so reviewing reports in detail to be able to distinguish "this result we promised was not achieved because we are incompetent" from "this result is negative, and we need to change direction based on new research" is just infeasible. In commercial research the situation is much better in my experience, probably because your "funding agency" is more narrowly focused and is typically interested in getting useful results. [1] https://news.ycombinator.com/item?id=29488641 https://news.ycombinator.com/item?id=29488641
jjtheblunt 5y ago
Perhaps these unrepeatable studies will be obviated by personalized targeted treatments, given the ubiquity of Illumina (etc) sequencers and designer antigens (BioNTech and Moderna, for example)?
- derbOac 5y ago
  There's reasons to suspect problems could be worse not better with personalized inference. Defining replication gets tricky when your N is 1, and your population is one system, perhaps at a particular moment in time.
em500 5y ago
Remindes me of the Vox article with the classic line "Everything we eat both causes and prevents cancer". https://www.vox.com/2015/3/23/8264355/research-study-hype https://www.vox.com/2015/3/23/8264355/research-study-hype
acenes 5y ago
Does anyone know of such a systematic replication effort in other fields of science? I wonder if the conclusion would be much different.
- nverno 5y ago
  There are a few, one of the popular ones on this site, https://www.science.org/doi/10.1126/sciadv.abd1705 https://www.science.org/doi/10.1126/sciadv.abd1705 shows the results are pretty similar in different fields [[a search for "replication" on hackernews will bring up some others]]
77pt77 5y ago
At least it's better than psychology...
micro_cam 5y ago
I worked in cancer genomics (and applications of ml to that) and have a couple of papers to my name. (Software methods so repeatable in the sense you can rerun them but that doesn't mean the data will mean anything). Every ML practitioner should spend some time working with genetic data just to realize how weird things can get when you have millions of sparse features for a few thousand cases and all sorts of batch effects. Like to do it well you need to control for the lightbulbs in the microarray machine you were using or the sequencing center used, minor version of the sequencing technology and reagents order the subjects were sequenced in, who ran the machine, geographic origin of subjects etc. And then when you go out and try to tie your data into the literature you need to be aware of all sorts of self reinforcing bias in which genes get published on. Amazing and artful applications of mathematics have been developed to address all of these and the care bioinformatic papers address things like cross validation with makes ML look like a joke. Beyond that its even common to go out and get more data or validate with different technology before publishing but it is still easy to get things wrong. If we truly want reproducible research we need to address the batch effects with less noisy sequencing machines and an assembly line like approach to generating orders of magnitude more data as cheaply as possible including new data on new subjects to verify studies after the fact.
goto11 5y ago
Good thing we don't have empirical research in software development. This means we avoid a replication crisis.
- bhk 5y ago
  Hah! This paper describes a deep-dive into papers on database research that made quantitative claims, and about half of the investigated papers had problems with replications of some of their claims. The Repeatability Experiment of SIGMOD 2008 https://pages.saclay.inria.fr/ioana.manolescu/PAPERS/SIGRecRepeatability.pdf https://pages.saclay.inria.fr/ioana.manolescu/PAPERS/SIGRecR...
- Leherenn 5y ago
  Benchmarking is often close enough.
jmnicolas 5y ago
Dear HNers, I hope that you will remember this article (or all the previous one) the next time you will invoke science like a religious dogma in a discussion. If you can't reproduce it, it's not science. I believe in science, but I also know scientists are humans and thus fallible. We live in a post-truth world, I have no certainty about anything anymore except what I can check by myself, which is not much.
- jiggawatts 5y ago
  Worse, scientific organisations are institutions. I trust individual scientists to be reasonably honest, but I trust the bureaucratic organisations they belong to as far as I can throw them...
  - the_only_law 5y ago
    I really would like to do research in a specific field, but I’m just so worried about the bureaucracy and horror stories of academia. I mean I’ve had to operate in bureaucratic environments, but it just seems so antithetical to what my actual goals would be. Just the amount of bureaucracy I’ve encountered trying to get into universities has rubbed me wrong. Sadly I couldn’t remotely afford the equipment to do things on my own (nor do I think I could handle the initial learning workload these days without some structure).
    - tsbischof 5y ago
      The equipment can be affordable depending on how you structure things. For example, there are opportunities like the Molecular Foundry user program [1] which provide grants for the costs of running experiments on expensive machinery. The main trick is having a specific enough question to answer that you can define how the grant would help you do so. There are also community initiatives like BosLab [2] which are more similar to maker spaces in structure. [1] https://foundry.lbl.gov/user-program/get-started-here/ https://foundry.lbl.gov/user-program/get-started-here/ [2] https://www.boslab.org/ https://www.boslab.org/
- Pyramus 5y ago
wesleywt 5y ago
As someone who took pains to make my work reproducible, ultimately leading to my lab continually expanding on the project as well as leading to a patent, I can tell you. Yeah, many researchers are full of shit. But it only takes a few to push the science forward.
planck01 5y ago
There is only one real solution in the paper quality crisis. High quality journals need not only to peer review papers but also do the independent replication or verification before publication. This is expensive and time consuming, but better and cheaper than the current house of cards that is science.
rob_c 5y ago
slow clap... Unfortunately the argument of "too expensive" or "too difficult" fails to fly or appeal to me when astro/particle/nuclear physics always split their people/experiments/resources into multiple teams, eventually to compare and contrast results. "Big-pharma", often supporting this research has enough money/resources to waste it seems that they will pursue any research to test it rather than verify it outright. And this culture has fed back into the mindset of most biologists these days it seems... This has to be one of the biggest drains/in-efficiencies in the field and is driven by naive corporate policy/greed... such a waste.
ricardobayes 5y ago
I briefly worked for a startup that made cancer patients genom sequence data and looked up relevant medicines, and trials in pubmed. They sold this as personalized cancer treatment for tens and hundreds of dollars. One key takeaway for me from that experience was that I should not smoke and should not eat processed meat, if I want to live a long life. Neither small cell lung cancer or gastric cancers are very treatable or fun.

194 points by kevin_hu 5y ago

In mice? Also, the lab animal telomere issue has corrupted almost all animal studies in the last 50 years, but there doesn't seem to be any effort to correct course, or even to acknowledge the problem.

ramraj07 5y ago
What lab animal telomere issue? Never heard of that in my decade of working in a lab.
- hall0ween 5y ago
  Also curious to know.
- Semaphor 5y ago
  This seems relevant both for the links and the rebuttal (?): https://biology.stackexchange.com/questions/90609/why-might-long-telomeres-be-selected-for-in-laboratory-mice https://biology.stackexchange.com/questions/90609/why-might-...
- alextheparrot 5y ago
  Probably regurgitating it from the Weinstein stuff. https://uncancelclub.com/bret-weinstein-1/ https://uncancelclub.com/bret-weinstein-1/
- ashwal 5y ago
  They're probably referring to: https://pubmed.ncbi.nlm.nih.gov/11909679/ https://pubmed.ncbi.nlm.nih.gov/11909679/ I think this is one symptom of a much greater issue: the inbred mice strains we use are disastrously weird (arising from the initial population artificially bred to make research easier). For the sake of controlling for genetics, we've chosen to make lab research translation drastically less effective.
- KenoFischer 5y ago
  I'm assuming OP is referring to this: https://oncobites.blog/2020/07/29/modeling-aging-and-cancer-are-lab-mice-different-from-their-wild-cousins/ https://oncobites.blog/2020/07/29/modeling-aging-and-cancer-...
- robbedpeter 5y ago
  https://www.nobelprize.org/prizes/medicine/2009/illustrated-information/#:~:text=The%20Nobel%20Assembly%20at%20Karolinska,telomeres%20and%20the%20enzyme%20telomerase https://www.nobelprize.org/prizes/medicine/2009/illustrated-... 2009 Nobel prize given for this work. Lab animals, due to the way they are bred, are essentially susceptible to cancer and health issues not seen in normal wildtype populations. Shortened telomeres are the primary danger, increasing mutation rate and cell senescence. Essentially, all data involving the expectation of normal cellular behavior is subject to question - signaling from lab animals is corrupted.
  - pazimzadeh 5y ago
    Do you have a source because as I understand it the length of the telomeres is reset each generation and so generally not transmissible to the next generation. Apparently leukocyte telomere length is heritable though, is that what you are talking about?
    - robbedpeter 5y ago
      https://pubmed.ncbi.nlm.nih.gov/11909679/ https://pubmed.ncbi.nlm.nih.gov/11909679/ https://www.gwern.net/docs/longevity/2002-weinstein.pdf https://www.gwern.net/docs/longevity/2002-weinstein.pdf
      - ramraj07 5y ago
        What exactly are you interpreting out of this paper now ? How does this invalidate cancer studies in mice? Most cancer studies in mice are xenografts of human cancer cells so any relevance of mice telomeres is completely lost anyway. Also this obsession with telomeres being the fundamental key for all our biological problems seems so conspiratorial. Like you think our cells which do unimaginable things can't keep their DNA ends frothy? Like that's what is gonna spell their doom? Or you consider telomeres are just a symptom of a cells longevity decided by other factors ?
  - AceJohnny2 5y ago
    The Nobel prize is about discovery of the telomeres, not about anything about lab populations.

smegsicle 5y ago

Cancer as entropy- the plot thickens

lelanthran 5y ago
> Cancer as entropy- the plot thickens Isn't cancer basically the result of entropy? With enough imperfect cell-divisions, sooner or later one of them is going to have a mutation that causes unbound cell-division (i.e. growth).

deleted 5y ago

[deleted]

Victerius 5y ago

I wrote a brief comment in a previous thread about some of the causes of this replicability crisis: https://news.ycombinator.com/item?id=29478946 https://news.ycombinator.com/item?id=29478946 Tl;dr misaligned incentives in science, the academic job market, and the funding system, the publish or perish dogma, and an overproduction of PhDs.

smegsicle 5y ago
What's your connection to the industry, that you recommend fewer scientists and greater centralization of research?

ashwal 5y ago

There's many papers showing similar. (My personal favorite is https://www.science.org/doi/10.1126/scitranslmed.aaw8412 https://www.science.org/doi/10.1126/scitranslmed.aaw8412 where they removed the claimed target for various cancer therapies and many still worked) If you've ever been within a mile of a research lab this shouldn't be remotely shocking. Typical research is done rapidly, sloppily, and in a context where getting the "right" answer is the only incentive. The replication crisis that hit psych is only being held back by (IMO) that bio is harder to replicate due to tooling, protocols, reagents, cell lines, etc. In complex systems you have the degrees of freedom to be wrong at a scale that folks still do not appreciate.

kergonath 5y ago
> If you've ever been within a mile of a research lab this shouldn't be remotely shocking. Typical research is done rapidly, sloppily, and in a context where getting the "right" answer is the only incentive. Let me preface that by saying that 50% sounds like an awful lot, even bearing in mind the following. On top of that, these studies focus on highly complicated phenomena, where almost anything could play a role. It is common that one factor that is not necessarily accounted for in a study turns out to be important, even in well-run studies. Flukes happen as well. Failure to replicate per se is not a problem, if the original study was done well and honestly. It is still some data points that could be re interpreted later in the light of other studies. Now, the fact that there is so little incentive to publish confirmation studies is a big problem, because it means that these non-reproducible works are not as challenged as they should be. Remember that one study is meaningless. Even with tiny rates of false positive or negative, erroneous conclusions are bound to happen at some point. A result is not significant unless it has been observed independently by different research groups.
- hdjjhhvvhga 5y ago
  > the fact that there is so little incentive to publish confirmation studies is a big problem I don't quite get this point: this is cancer research, co in theory all related institutions should be interested in replicating breakthrough studies, right? And failure to replicate it, time after time, should be a cause of concern, shouldn't it?

time_to_smile 5y ago

It's funny how early 20th century science was filled with mind blowing discoveries and realizations that fundamentally changed how we viewed the universe and to some extent even life itself, while the early 21st century is filled with a surprisingly number of boring work churned out by overworked grad students, that turns out to be, in unsettling quantities, largely bad, irreproducible science. People love to blame bad statistics, but it's much more obvious that the issue is the "publish or perish" culture and a perpetual race to manufacture credibility in academia. If you're curious you can trace some of this back to the 70s when peer review as we know it today was essentially born, precisely as a means of proving credibility during a period of academic funding crunches. As much as it's easy to point a finger at academic I increasingly find the same thing in industry. Within my own career the number of companies that are doing nothing more than finding elaborate ways to burn investor funding (without even realizing they are participating in this) has grown to consume almost the entire tech industry. At this point the most shocking news isn't fraud, it would be to find someone out their down real, meaningful work.

wilkommen 5y ago
I agree with this. I’ve been coming to similar conclusions myself.
tekstar 5y ago
If you have not, you should read The Idea Factory. https://www.amazon.com/Idea-Factory-Great-American-Innovation/dp/0143122797 https://www.amazon.com/Idea-Factory-Great-American-Innovatio... We owe most of our "modern world" to the hard science done at Bell Labs in the 1900s, before the corporation was singularly focused on quarterly profits.
malwarebytess 5y ago
I really do believe that the attempt to industrialize scientific research has failed. Science cannot be done at scale. It comes from singularly focused projects lead by pure inquiry. I don't regret my PhD because I enjoyed achieving it, but I see it as a waste of time. There's an analogy in Software Engineering. Programmers are akin to Artisans or Craftsmen, but the industry wants and needs them to be something more like little factories. I don't think that approach makes good software, even if it's a good product. Just as I don't think the system we have created in academia doesn't produce good research, even though it makes good careers.

farseer 5y ago

More than half of high-impact cancer lab studies can be replicated?

sebow 5y ago
More than half of all studies can be replicated?

deleted 5y ago

[deleted]

epistasis 5y ago

> Additional problems surfaced when labs began experiments, such as tumor cells that did not behave as expected in a baseline study. Given the various difficulties of culturing cell lines, this is not surprising. What is infuriating to me is unresponsive author queries about data and protocols. That should result in some sort of ding to the paper on the journal's website.

a-dub 5y ago

i've said it before and i'll say it again, this isn't going to improve until there are more grant dollars and recognition for replication than there is for novel discovery. replication is boring, yet critical, and we should be compensating highly skilled scientists for doing it.

glial 5y ago
100% agree. The NIH could replicating studies a requirement of funding for graduate or postdoctoral training. In addition to the scientific and societal benefit, it would just be good training.
- a-dub 5y ago
  your first task in lab will be to reproduce our competitor's work! in seriousness though, there's enough money and trained scientists looking for work... moreover i think there's a whole personality type that is prevalent in science that would be very happy to replicate and reproduce results...
- tsbischof 5y ago
  I would be glad to have funding explicitly set towards appropriate and open documentation of the actual protocols, equipment, and materials, plus a combined effort to enhance these reporting requirements by the journal. There has to be a balance between the one-off interesting new findings and the large-scale work, but clinical trials definitely fall under the latter category.
bmitc 5y ago
That sounds like spending money on trying to replicate results that can’t be replicated doesn’t it? Isn’t that what this study was doing? Doesn’t it make more sense to revisit how the existing funding is spent? I have very little insight into this area of research, so I’d just have to guess everyone’s in a hurry and reaching for results or published papers.
- a-dub 5y ago

timr 5y ago

I'm shocked that close to half can be replicated. It's not a matter of academic misconduct -- biology is just hard. There are a lot of ways to fool yourself. If you doubt, just take look at the number of drug candidates that make it out of the lab (it's a LOT lower than 50%). The journalism side of Science magazine is far too credulous. Most science is wrong. Even when it is published in Science (perhaps, especially when it is published in Science).

throwawaylinux 5y ago
All scientific fields are hard though. The way they are advanced is with rigorous methods. I'm not shocked either though. Not because biology is hard, but because academia is corrupt.
- soco 5y ago
  Hanlon's razor: never attribute to malice that which is adequately explained by stupidity. Not that academics were stupid, but making mistakes usually obvious in hindsight. And yeah there would be some malice as well, as everywhere, including here, because everybody's a human until those rigorous methods weed humanity out of the processes (way to go).
  - native_samples 5y ago
    Many of the studies couldn't be replicated because the original authors didn't provide sufficient details in their papers, and then refused to reply when asked for those details. Whether this is malice or not is kind of irrelevant - they have failed to meet the most basic requirements expected of researchers. They took the money but did not deliver the expected work artifacts.
- ansgri 5y ago
  Not that it's corrupt, but incentive structure is suboptimal to say the least, especially for government-funded work. There was a recent blog post / HN thread [1] about groups never admitting failure, it's very relevant here. The funding agency has to have a process for vetting applications, and what's more reasonable than the ability to manage their own work, i.e. propose a plan and successfully execute on it? Combine it with slow cadence of funding and you have a list of promised results for next 3+ years which you must achieve somehow or you'll never see a grant from that agency again. And they give out a lot of grants, so reviewing reports in detail to be able to distinguish "this result we promised was not achieved because we are incompetent" from "this result is negative, and we need to change direction based on new research" is just infeasible. In commercial research the situation is much better in my experience, probably because your "funding agency" is more narrowly focused and is typically interested in getting useful results. [1] https://news.ycombinator.com/item?id=29488641 https://news.ycombinator.com/item?id=29488641

jjtheblunt 5y ago

Perhaps these unrepeatable studies will be obviated by personalized targeted treatments, given the ubiquity of Illumina (etc) sequencers and designer antigens (BioNTech and Moderna, for example)?

derbOac 5y ago
There's reasons to suspect problems could be worse not better with personalized inference. Defining replication gets tricky when your N is 1, and your population is one system, perhaps at a particular moment in time.

em500 5y ago

Remindes me of the Vox article with the classic line "Everything we eat both causes and prevents cancer". https://www.vox.com/2015/3/23/8264355/research-study-hype https://www.vox.com/2015/3/23/8264355/research-study-hype

acenes 5y ago

Does anyone know of such a systematic replication effort in other fields of science? I wonder if the conclusion would be much different.

nverno 5y ago
There are a few, one of the popular ones on this site, https://www.science.org/doi/10.1126/sciadv.abd1705 https://www.science.org/doi/10.1126/sciadv.abd1705 shows the results are pretty similar in different fields [[a search for "replication" on hackernews will bring up some others]]

77pt77 5y ago

At least it's better than psychology...

micro_cam 5y ago

I worked in cancer genomics (and applications of ml to that) and have a couple of papers to my name. (Software methods so repeatable in the sense you can rerun them but that doesn't mean the data will mean anything). Every ML practitioner should spend some time working with genetic data just to realize how weird things can get when you have millions of sparse features for a few thousand cases and all sorts of batch effects. Like to do it well you need to control for the lightbulbs in the microarray machine you were using or the sequencing center used, minor version of the sequencing technology and reagents order the subjects were sequenced in, who ran the machine, geographic origin of subjects etc. And then when you go out and try to tie your data into the literature you need to be aware of all sorts of self reinforcing bias in which genes get published on. Amazing and artful applications of mathematics have been developed to address all of these and the care bioinformatic papers address things like cross validation with makes ML look like a joke. Beyond that its even common to go out and get more data or validate with different technology before publishing but it is still easy to get things wrong. If we truly want reproducible research we need to address the batch effects with less noisy sequencing machines and an assembly line like approach to generating orders of magnitude more data as cheaply as possible including new data on new subjects to verify studies after the fact.

goto11 5y ago

Good thing we don't have empirical research in software development. This means we avoid a replication crisis.

bhk 5y ago
Hah! This paper describes a deep-dive into papers on database research that made quantitative claims, and about half of the investigated papers had problems with replications of some of their claims. The Repeatability Experiment of SIGMOD 2008 https://pages.saclay.inria.fr/ioana.manolescu/PAPERS/SIGRecRepeatability.pdf https://pages.saclay.inria.fr/ioana.manolescu/PAPERS/SIGRecR...
Leherenn 5y ago
Benchmarking is often close enough.

jmnicolas 5y ago

Dear HNers, I hope that you will remember this article (or all the previous one) the next time you will invoke science like a religious dogma in a discussion. If you can't reproduce it, it's not science. I believe in science, but I also know scientists are humans and thus fallible. We live in a post-truth world, I have no certainty about anything anymore except what I can check by myself, which is not much.

jiggawatts 5y ago
Worse, scientific organisations are institutions. I trust individual scientists to be reasonably honest, but I trust the bureaucratic organisations they belong to as far as I can throw them...
- the_only_law 5y ago
  I really would like to do research in a specific field, but I’m just so worried about the bureaucracy and horror stories of academia. I mean I’ve had to operate in bureaucratic environments, but it just seems so antithetical to what my actual goals would be. Just the amount of bureaucracy I’ve encountered trying to get into universities has rubbed me wrong. Sadly I couldn’t remotely afford the equipment to do things on my own (nor do I think I could handle the initial learning workload these days without some structure).
  - tsbischof 5y ago
    The equipment can be affordable depending on how you structure things. For example, there are opportunities like the Molecular Foundry user program [1] which provide grants for the costs of running experiments on expensive machinery. The main trick is having a specific enough question to answer that you can define how the grant would help you do so. There are also community initiatives like BosLab [2] which are more similar to maker spaces in structure. [1] https://foundry.lbl.gov/user-program/get-started-here/ https://foundry.lbl.gov/user-program/get-started-here/ [2] https://www.boslab.org/ https://www.boslab.org/
Pyramus 5y ago

wesleywt 5y ago

As someone who took pains to make my work reproducible, ultimately leading to my lab continually expanding on the project as well as leading to a patent, I can tell you. Yeah, many researchers are full of shit. But it only takes a few to push the science forward.

planck01 5y ago

There is only one real solution in the paper quality crisis. High quality journals need not only to peer review papers but also do the independent replication or verification before publication. This is expensive and time consuming, but better and cheaper than the current house of cards that is science.

rob_c 5y ago

slow clap... Unfortunately the argument of "too expensive" or "too difficult" fails to fly or appeal to me when astro/particle/nuclear physics always split their people/experiments/resources into multiple teams, eventually to compare and contrast results. "Big-pharma", often supporting this research has enough money/resources to waste it seems that they will pursue any research to test it rather than verify it outright. And this culture has fed back into the mindset of most biologists these days it seems... This has to be one of the biggest drains/in-efficiencies in the field and is driven by naive corporate policy/greed... such a waste.

ricardobayes 5y ago

I briefly worked for a startup that made cancer patients genom sequence data and looked up relevant medicines, and trials in pubmed. They sold this as personalized cancer treatment for tens and hundreds of dollars. One key takeaway for me from that experience was that I should not smoke and should not eat processed meat, if I want to live a long life. Neither small cell lung cancer or gastric cancers are very treatable or fun.