14 ms·

Just 11% of 53 cancer research papers were reproducible

292 points by vog 14y ago

RyanMcGreal 14y ago
The analysis in question: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a...
- micro_cam 14y ago
  This is a really insightful article that contains a number of points that qualify the result and present lessons for any one interested in data science. A few points that stand out to me: !) The 11% number comes not from a randomized sample of cancer papers but from attempts made by AmGen to reproduce results that might be useful in drug development. This means that they made good effort to reproduce these results but there is also a strong selection bias that needs to be acknowledged. 2) They point out that using survival time as the measurement complicates things. I've done a lot of machine learning and statistics on cancer and medical data and, in my experience, this can not be overemphasized. and There are loads of confounding factors that contribute to survival time. I expect that big break throughs will come as we develop rigorous ways for measuring the behavior of a tumor (does it metastasize? how does it feed itself?) and use those as the targets of our regression. (Currently these thins are measured by human visual inspection if at all.) 3) They point out that the studies that were repeatable were the ones that were careful about using blinded controls, and eliminating investigator bias. This is basic stuff but easy to overlook. In terms applicable to a startup, a data scientist needs to me motivated to vet existing and proposed practices and identify flawed ones as much or more then they are motivated to maximize gain.
  - pcrh 14y ago
    With regards to 1), it would be ironic if this study was itself not statistically valid...
mchusma 14y ago
This is a great issue to highlight. However, I'm unconvinced we need to optimize around creating research that can be reproducible, rather that the process of science generates the best possible results. Both the scientific and lay public should read this (appropriately) as: one study does not prove a result, there is an 89% chance that a single research paper is simply wrong. This mentality seems to be the bigger gap, although generating high quality research is a big part of the equation (mostly I think for the wasting of resources to validate, process, and reproduce claims).
- ttrreeww 14y ago
  Cold fusion was reproducible. What are you suggesting we do here?
wheaties 14y ago
This is one of the reasons I went into math and software. In math there is only black and white, provably true and patently wrong. Still, part of the reason this is so is that to show any results everything must be laid on the table. CS and Math papers are therefore open by design and the world has benefited. I hope the rest of science will move in this direction.
- rajeevk 14y ago
  CS has similar problem, specially in AI fields. Researches often claim the accuracy rate more than 95%, but in reality that is much lesser (in fact less than 10% in most of the claims).
  - gordaco 14y ago
    Having worked in computer vision and robotics, I can attest to this.
    - jacobparker 14y ago
      Yes. Of the fields I've implemented algorithms from papers in, computer vision seems to be particularly adverse to discussing the (often critical) downsides of their algorithms. Often its something like "Camera calibration isn't exactly perfect? Well this scene reconstruction technique simply wont work."
  - 5kg 14y ago
    Yes, that's why I no longer consider pursuing PhD in CS. It's a waste of life to reproduce non-reproducible papers that is full of buzzword. Hopefully, more journals and confs will follow the Open Access and Reproducible Research model like IPOL[1]. > Each article contains a text describing an algorithm andsource code, with an online demonstration facility and an archive of online experiments. The text and source code are peer-reviewed and the demonstration is controlled. [1]http://www.ipol.im/ http://www.ipol.im/
- jamessb 14y ago
  There are 'proofs' that contain errors, just as there are science papers that reach incorrect conclusions. Also, it's not necessarily easier to verify a proof than it is to repeat an experiment (or perform additional supporting experiments), especially for complex proofs which very few people know enough to completely understand (eg. of Fermat's last theorem, or the Poincare conjecture).
ilikejam 14y ago
An article claiming that only 11% of findings could be reproduced doesn't cite its sources, thus rendering the findings of the article unreproducible. Nice work!
- insickness 14y ago
  Exactly. This should be the top comment. Are we just supposed to assume their work was true based on trust? Perhaps this 'study' was actually just a fleecing meant to show that we take too many studies to be true based on simple trust.
  - Ensorceled 14y ago
    It should not be the top comment. The original article is an explanation of WHY this was done. If a comment is posted explaining why the study was a bad idea or should not have been published despite this explanation THAT should be the top comment.
- Ensorceled 14y ago
  Nature wrote an editorial justifying the situation, you can read it here http://www.nature.com/nature/journal/v485/n7396/full/485041e.html http://www.nature.com/nature/journal/v485/n7396/full/485041e...
  - return0 14y ago
    > "those authors required them to sign an agreement that they would not disclose their findings about specific papers" I don't know why this doesn't cause a huge stir in the scientific community. Seems like everyone is fine with people sweeping their negative data under the rug. Shameful is a very mild word for that.
    - Ensorceled 14y ago
      Agreed, but this study was specifically designed to highlight the problem not solve it. And the results would be useless, or non-existent, if they restricted themselves to scientists that would co-operate fully with the process and the possible "shaming" afterward.
viggity 14y ago
After hearing similar things about psychology papers, this is rather disconcerting. This is why I am a climate change skeptic, I don't know whether it is happening due to CO2 or not, but I am confident that science can't be confident when they can't do control experiments. You can't control for any variable when it comes to the climate, let alone all the reasonable ones. We have infinitely more capabilities to control variables when researching cancer than we do with the climate of something as massive and complex as the earths climate, and it turns out our confidence is cancer research may kind of suck. one interested in the subject should check out feynman's discussion on the psychological effect on scientific research. Millikan use bad assumptions for his oil drop experiment to determine the charge of an electron, but nobody would publish results that differed too much. http://en.m.wikipedia.org/wiki/Oil_drop_experiment http://en.m.wikipedia.org/wiki/Oil_drop_experiment
- JoachimSchipper 14y ago
  You took an almost entirely unrelated story and made it about your political hobbyhorse. Please keep politics off HN.
  - viggity 14y ago
    The article was not about cancer. The whole point of the discussion is about reproducibility in scientific research. I was merely applying the same discussion and concept to a different branch of science. HN is a place for nerds to discuss interesting things. Things like science and it's applications. As far as I'm concerned, that is what I did.
  - CamperBob2 14y ago
    And you just took an entire field off the table for discussion on the grounds that it looks like a "political hobbyhorse." Good job.
- campnic 14y ago
  I realized that by replying i'm probably encouraging you to continue posting this non-sense. I agree with Joachim below, this is barely related to the article.
kailuowang 14y ago
Scientific publications have a history of having papers contradictory to each other for a long time. In medical research, findings are often based on statics results from limited samples. From that perspective, it will be a big surprise if they are more accurate than public polls.
drorweiss 14y ago
So maybe we need rename Life Sciences to Life Arts?
languagehacker 14y ago
What the hell? That's 5.83 papers. How do you reproduce .83 of something?
- drharris 14y ago
  Um, 6 papers out of 53, significant digits give you 11%. Not that difficult.
  - ndr 14y ago
    I guess saying "11% of 53 papers" was cooler than saying "6 out of 53". Meh.
    - drharris 14y ago
      Yeah, my guess is they used that form to imply 11% of all cancer research is unreproducible. But in ironic fashion, they can't show their own data and thus we can make no broad claims about it.
codeulike 14y ago
Most published scientific research papers are wrong, according to a new analysis. http://www.newscientist.com/article/dn7915-most-scientific-papers-are-probably-wrong.html http://www.newscientist.com/article/dn7915-most-scientific-p... Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 http://dx.doi.org/10.1371/journal.pmed.0020124 http://dx.doi.org/10.1371/journal.pmed.0020124
- kailuowang 14y ago
  Is this analysis a published scientific research?
  - return0 14y ago
    Joking aside, the authors of these studies have highlighted the low statistical power of research in a number of fields (see also http://www.nature.com/nrn/journal/v14/n5/abs/nrn3475.html http://www.nature.com/nrn/journal/v14/n5/abs/nrn3475.html), which is a serious issue that often leads to over-interpretation. It's as if researches willing to publish more are rushing hasty studies out the door. It's a good thing that people can measure these things and ring the alarm
  - gwern 14y ago
    Depends. By and large, Ioannidis's paper is pointing out the basic consequences of the alpha and beta parameters in Neyman-Pearson hypothesis testing, so there's a very good argument that this paper is merely statistics, and statistics is generally considered to be mathematics, and mathematics not science but its own thing.
- deleted 14y ago
  [deleted]
gordaco 14y ago
Wow, that's sad. Are we really so blind when it comes to cancer research? I recall that oncology journals usually have a ludicrously high impact index (as ludicrously as 5 digit IIRC); that means there are a lot of citations, which is an indicator that there is a lot of research going on. And with a lot of research going on, well, you can expect a lot of false positives. So, I'm wondering, could this be a case of cherry-picking or some kind of selection bias? It wouldn't be difficult to select a lot of bogus-sounding research and test it.
- Anderkent 14y ago
  Medical research in general is dubious at best. See for example http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal... There's even a startup trying to work around it: http://www.metamed.com/ http://www.metamed.com/. They look for papers related for a given condition on request, trying to find ones that are actually well done and promising.
- Alex3917 14y ago
  "And with a lot of research going on, well, you can expect a lot of false positives." Why would you expect a lot of false positives? Aren't results supposed to be reproducible at least 95% of the time?
  - gordaco 14y ago
    I was thinking about absolute numbers, so 5% of a lot can still be a lot.
- deleted 14y ago
  [deleted]
- atrus 14y ago
  But who is going to pay to reproduce that research? What if that research took years to do?
  - jerf 14y ago
api 14y ago
After decades of publish-or-perish sweatshop science I'm sure the great shining archive of scientific truth is sort of like an inbox with no spam filter.
- evoloution 14y ago
  Such a powerful sentence. Lately science looks like olympic games without anti-doping control...
  - api 14y ago
    I'm becoming pretty convinced that the human future depends on the disruption and obsolescence of bureaucracies.
    - spikels 14y ago
      I think you are on to something. When science became so big that it first needed then eventually was run by "managers" I bet it lost much of its effectiveness. But how do we change this? For starters there is a lot of money involved for the institutions conducting this (apparently shoddy) research.
      - kanzure 14y ago
        > When science became so big that it first needed then eventually was run by "managers" I bet it lost much of its effectiveness. You can just do projects on your own, you know. There's nothing about science that absolutely requires an institution. http://groups.google.com/group/diybio http://groups.google.com/group/diybio
campnic 14y ago
I have been in discussions about this with one of my friends working in academic materials research. Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools. I'm talking opening their code in notepad, 'versioning' files by sending around zip files with numbers manually added to the end of the file name, etc. This doesn't even begin to scratch the surface of the 'reproducible results' problem. Often times, the software I've seen is 'rough' to be kind. Most times its not even possible to get the software running (missing some really specific library or some changes to a dependency which haven't been distributed) or its built for a super specific environment and makes huge assumptions on what can 'be assumed about the system.' This same software produces results which end up being published in journals. If any of these places had money to spend, I think there could be a valuable business in teaching science types how to better manage their software. Its really unfortunate that outside of a few core libraries (numpy, etc.) the default method is for each researcher to rebuild the components they need. I'm surprised about only 11% of results being reproducible. It seems lower then I'd expect. I agree we don't want to optimize for reproducibility, but obviously there is some problem here that needs to be addressed.
- phireal 14y ago
  I think that's what the folks at Software Carpentry [0] are trying to do. I went on one of their courses, and you're taught the basics of writing good software, version control and databases (SQLite). I've frequently recommended it to fellow scientists. [0] http://software-carpentry.org/ http://software-carpentry.org/
  - campnic 14y ago
    This is great! Thanks for sharing.
- bigiain 14y ago
  Have you seen: http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/ "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it into running; and it's not pretty code. Academic code is about "proof of concept." These rough edges make academics reluctant to release their software. But, that doesn't mean they shouldn't. Most open source licenses (1) require source and modifications to be shared with binaries, and (2) absolve authors of legal liability. An open source license for academics has additional needs: (1) it should require that source and modifications used to validate scientific claims be released with those claims; and (2) more importantly, it should absolve authors of shame, embarrassment and ridicule for ugly code."
pesenti 14y ago
The problem is not that research is not reproducible. Researchers are humans and full of biases. The problem is too much emphasis is put on novelty vs. reproducibility. It took a long time for software engineers to recognize the value of testing/validating vs. creating new code. In the same way the research world needs to put new emphasis on reproducing/validating results.
- return0 14y ago
  The problem is that too much emphasis is put on prestige and reputation than on actual impact of the work. Coming from a physics background I always felt that life scientists tend to be overly audacious in their claims, am I alone in this?
  - pesenti 14y ago
    Is it only in life sciences? Remember the cold fusion and the faster than light particle fiascos?
baldfat 14y ago
Pediatric Cancer gets only 4% of Federal funding and less than 1% of Private Drug company research. Also American Cancer Society less than 5 cents on the dollar goes to research. Pediatric Cancer is the leading cause of death for 15 and under. So to see this funding for cancer get thrown away by fraud is extremely upsetting.
- refurb 14y ago
  Those are some pretty misleading statistics. First off, pediatric cancer is NOT the leading cause of death for those 15 and under[1]. It's up there, but "unintentional injury" takes twice as many lives. Second of all, not many children die (1200 per year), so even if something is the leading cause of death, it doesn't mean it happens very often. Cancer kills 10 times as many people between the ages of 35-44. If you add up all adult cancers it's probably 20-30x the number of deaths. Also, just because money goes to adult cancer, doesn't mean it can't help. Most drug are developed for adults first and then tested in children. It's not like people are ignoring pediatric cancer. Before you get so upset you should really sit down and think these things through. [1]http://www.cdc.gov/injury/wisqars/pdf/10LCID_All_Deaths_By_Age_Group_2010-a.pdf http://www.cdc.gov/injury/wisqars/pdf/10LCID_All_Deaths_By_A...
  - Alex3917 14y ago
    "Most drug are developed for adults first and then tested in children." It was also be unethical to develop drugs for children without first testing them on adults, since children can't give informed consent.
brudgers 14y ago
Perhaps we are looking for the wrong measure of success. This is not inclined plane ball rolling in fifth period Physics with Mr. Johannes. Reproducibility in cutting edge experiment is success. It means that what is being measured - even if the wrong thing - is within the control of the experimenter. It is absurd, but practical, to publish experimental results with the implication that they are reproducible based on a peer review of the results. "Well they look reproducible to me," when uttered by a peer reviewer is the current standard. All we are seeing is that the a priori reasoning of experts is not a substitute for empirical investigation of scientific claims. Let us not forget that a journal which finds no worthy articles this quarter has only two options, publish unworthy ones or publish no articles. I know how I react when a magazine subscription does not come.
- atrus 14y ago
  I wonder if it would be possible to produce a journal that only released an issue when it has, say, 10 articles? No monthly/quarterly schedule, you could have an issue release next week, or next year.
  - brudgers 14y ago
    As a digital document it is easy. Because printing requires substantial coordination, it is less practical. The subscription model common to the scientific journal industry makes it contractually problematic. I wonder what portion of academic research is equivalent to content farming.
zipdog 14y ago
What is more troubling is not that so few of these results results are reproducible, but that it appears almost no-one is trying to reproduce the results of earlier studies. The ability of the scientists who wrote the paper to even get access to the resources necessary to try and reproduce the results is limited. I'm reminded of Feymann's Cargo Cult Science: "I was shocked to hear of an experiment done at the big accelerator at the National Accelerator Laboratory, where a person used deuterium. In order to compare his heavy hydrogen results to what might happen with light hydrogen he had to use data from someone else's experiment on light hydrogen, which was done on different apparatus. When asked why, he said it was because he couldn't get time on the program (because there's so little time and it's such expensive apparatus) to do the experiment with light hydrogen on this apparatus because there wouldn't be any new result. And so the men in charge of programs at NAL are so anxious for new results, in order to get more money to keep the thing going for public relations purposes, they are destroying--possibly--the value of the experiments themselves, which is the whole purpose of the thing. It is often hard for the experimenters there to complete their work as their scientific integrity demands." For anyone who hasn't read it, the whole thing is excellent http://www.lhup.edu/~DSIMANEK/cargocul.htm http://www.lhup.edu/~DSIMANEK/cargocul.htm
- varjag 14y ago
  In practical terms, reproducing someone else's work most of the time boils down to redoing someone else's PhD thesis. Which is both not very interesting and doesn't help getting your own research done.
  - rdfi 14y ago
    I agree that reproducing someone else's experiment won't help you get your PhD closer to completion (because you should be doing your own experiments and publishing no matter what, if you expect to land that lecturing position, i.e. publish or perish). Reproducibility is one of the main principles of the scientific method (those papers shouldn't even been accepted if they don't contain enough information about how to reproduce the experiments they describe) If an experiment is so complex that it can be compared to redoing someone else's thesis it can be either that the thesis is very simple or the experiment is so complex that it probably proves nothing.
psaintla 14y ago
There are many reasons so many papers are not reproducible. 1.) The prospective data collection methods used in a lot of studies are deeply flawed and worse aren't documented. 2.) The retrospective data used in many studies is poorly validated and the quality is rarely a concern for most universities. 3.) The people who publish these papers are very intelligent, most far more intelligent than I am, but you'd be surprised how many of them don't have a very good handle on basic statistics. Not because they aren't smart enough to learn but mostly because they have no interest in it. 4.) There is a lot of pressure put on researchers to publish papers even if they aren't ready and as a result mistakes are made, there really needs to be less of an emphasis on the quantity of papers that are published and more emphasis on quality. For anyone who might be interested in such things, if you ever find yourself locked in a dungeon with nothing but a computer loaded with every journal article ever published you can find articles involving similar cohorts from the exact same universities that have different results.
tokenadult 14y ago
This editorial commentary and the article on which it is based are part of an ongoing effort to improve the quality of scientific publication in a number of disciplines. The Retraction Watch group blog http://retractionwatch.wordpress.com/ http://retractionwatch.wordpress.com/ by two experienced science journalists picks up many--but not all--of the cases of peer-reviewed research papers being retracted later from science journals. Psychology as a discipline has been especially stung by papers that cannot be reproduced and indeed in many cases have simply been made up. http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-audacious-academic-fraud.html?pagewanted=all&_r=0 http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-... That has prompted statistically astute psychologists such as Jelte Wicherts http://wicherts.socsci.uva.nl/ http://wicherts.socsci.uva.nl/ and Uri Simonsohn http://opim.wharton.upenn.edu/~uws/ http://opim.wharton.upenn.edu/~uws/ to call for better general research standards that can be practiced as checklists by researchers and journal editors so that errors are prevented. Jelte Wicherts writing in Frontiers of Computational Neuroscience (an open-access journal) provides a set of general suggestions Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020 http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full http://www.frontiersin.org/Computational_Neuroscience/10.338... on how to make the peer-review process in scientific publishing more reliable. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence. "With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses." Uri Simonsohn provides an abstract (which links to a full, free download of a funny, thought-provoking paper) http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2160588 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2160588 with a "twenty-one word solution" to some of the practices most likely to make psychology research papers unreliable. He has a whole site devoted to avoiding "p-hacking," http://www.p-curve.com/ http://www.p-curve.com/ an all too common practice in science that can be detected by statistical tests. He also has a paper posted just a few days ago http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2259879 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2259879 on evaluating replication results (the issue discussed in the commentary submitted to open this thread) with more specific tips on that issue. "Abstract: "When does a replication attempt fail? The most common standard is: when it obtains p>.05. I begin here by evaluating this standard in the context of three published replication attempts, involving investigations of the embodiment of morality, the endowment effect, and weather effects on life satisfaction, concluding the standard has unacceptable problems. I then describe similarly unacceptable problems associated with standards that rely on effect-size comparisons between original and replication results. Finally, I propose a new standard: Replication attempts fail when their results indicate that the effect, if it exists at all, is too small to have been detected by the original study. This new standard (1) circumvents the problems associated with existing standards, (2) arrives at intuitively compelling interpretations of existing replication results, and (3) suggests a simple sample size requirement for replication attempts: 2.5 times the original sample." The writers of scientific papers have a responsibility to do better. And the readers of scientific papers that haven't been replicated (or, worse, press releases about findings that haven't even been published yet) also have a responsibility not to be too credulous. That's why my all-time favorite link to share in comments on HN is the essay "Warning Signs in Experimental Design and Interpretation" by Peter Norvig, LISP hacker and director of research at Google, on how to interpret scientific research. http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html Check each submission to Hacker News you read for how many of the important issues in interpreting research are NOT discussed in the submission.
pcrh 14y ago
I work in biomedical research, and this finding has been discussed quite broadly. Most researchers don't believe it. Exactly reproducing novel findings usually requires a significant investment into the underlying procedures, which most places will not undertake. Reproduction instead usually occurs as part of an extension of the initial findings. The complexity of biology means that the first findings are frequently not reproduced exactly the same way as the original, but this does not detract from the "direction" of the initial findings. For example, the first finding might be that protein A promotes tumor growth by modifying protein B. An extension of these findings might be Protein A sometimes modifies protein B, and when it does tumor growth is stimulated, but it mostly does not modify protein B, it instead modifies protein C, which suppresses tumor growth. In this case, were the first results replicated? Yes, and no. This is how most biomedical research proceeds...
- rdfi 14y ago
  Really? Meaning that either there's another unaccounted variable(s) that controls the effect of A in B and C, meaning that we cannot conclude anything from the experiment. Yes and no is not acceptable...
  - epistasis 14y ago
    There's 20,000 broad classes of variables (genes), and on average 12 variations of each of those variable (alternative splicing, post-translational modification, cellular localization, other variables we haven't yet discovered ). This series of two experiments and likely 2 years found associations between three proteins, which is a hook into system of great complexity, a way to start looking for more. It is not "reproducible" yet it is incredibly valuable.
  - pcrh 14y ago
    It is exactly as you suggest, additional variables. Bear in mind that a single human cell has ~25,000 genes, and therefore at least that many potential degrees of freedom, even before you include external variables. It is very difficult to control for all of them.
  - 14y ago
jballanc 14y ago
I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on... Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being completely worthless, snake-oil type stuff, but this was not the case at all. Rather, individuals were working on an area about which almost nothing was known, and (more importantly) for which the key central organizing laws were not yet revealed (for Alchemy: the atomic theory of matter and chemical bonding, for Cell Signaling: how individual proteins and small molecules interact). Furthermore, the goals were lofty and almost certainly unattainable (for Alchemy: turn base metals into gold, for Cell Signaling: cure cancer), driving people to do "rush" work. It's not that the results are completely invalid, or that the experiments are useless. It's just that everyone feels like they're so close to a solution (Alchemists were way off with Phlogiston, and I'll bet Cell Signaling researchers are similarly clueless as to what really matters) that no one takes the time to step back and synthesize the results in an attempt to understand the forest from the trees. Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev. From experience, the fact that 89% of these "cancer research" papers are not reproducible almost definitely has less to do with fraudulent data and so much more to do with crazy complex experimental setups that end up probing a half-dozen experimental variables all at once (without the researchers even grasping that this is going on). Yeah, publish or perish sucks, but what sucks more is the death of basic science research. The Alchemists eventually became Chemists because they refocused on core principles (atomic theory, bonding) and forgot about the lofty goals (turn lead into gold)... ...but try telling any politician that they should decrease cancer research funding and refocus on genetics, structural biology, and evolution research. I'd love to know how they respond.
- jpdoctor 14y ago
  > It's not that the results are completely invalid Especially if only 11% of them are reproducible, they're only 89% invalid. I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.
nogoodnik 14y ago
The most interesting thing about all this is the fact that the researchers were required by paper authors to sign non-disclosure agreements and agree not to identify papers with non-reproducible results. This says a lot about the confidence authors have in their papers, doesn't it? Sorry but this isn't science anymore.
- ttrreeww 14y ago
  It's good old capitalism making money from "treating" sickness. The longer the better. A lifetime of "treatment" means a lifetime of profits. Who cares about the cure.
pessimizer 14y ago
There's an awful lot of money in cancer (research funding/patents), an awful lot of diet/lifestyle handwaving in cancer research conclusions (that aren't about treatments), and very little progress in making cancer more survivable. Just those facts would lead me to assume that most of the published research is bad. The great part is that 11% of the papers are reproducible, so the science isn't at a standstill.
- ttrreeww 14y ago
  Cancer is one of those sickness where you either cure it, or you die. Problem is, for profit companies seeks "treatment" to maximize profit, they don't seek the cure. Hence no progress on cancer.
- Mz 14y ago
  That isn't limited to cancer. Western medicine is driven by a lot of bad mental and social models. Doctors only make money if you are sick. That strikes me as a seriously flawed model right there.
japhyr 14y ago
This is one of the reasons I am happy to see what is going on with the Center for Open Science [0]. The goal of the project is to open up the entire research process, not just create open access to published results. They aim to make open tools supporting the entire research process as well, through the Open Science Framework [1]. One of the benefits of this approach is to provide peer review at each step of the research process. There is also an emphasis on encouraging validation of prior results, rather than having everyone focus on creating new research. This is the focus of the Reproducibility Project [2]. The Center is just getting started, and I sure hope it takes off. [0] - http://centerforopenscience.org/ http://centerforopenscience.org/ [1] - http://openscienceframework.org/ http://openscienceframework.org/ [2] - http://openscienceframework.org/project/EZcUj/wiki/home http://openscienceframework.org/project/EZcUj/wiki/home
brandon_wirtz 14y ago
This has been going around along with the Chemo Therapy doesn't work thread. The problem with people like the author is they think all cancer is the same. They try to apply information about Hodgkins to non-hodgkins. They think "Breast cancer" is a disease, not a symptom. There over 40 different cancer causes for breast cancer. And the treatments that work are primarily based on the cause not the visible symptom. This makes studies hard. Most people never learn the cause of their cancer. We are not fortunate enough to always have a single known cause that says "yes you worked in a nuclear waste plant for 6 years" and know that was the cause. Chemo for example in Hodgkin's Lymphoma increase your 5 year survival chances by 45%. But if you have Smoking related Lung Cancer it is less than 2% difference. Nature.com rarely produces articles that are informed. They push an agenda of Holistic medicine at the expense of scientific research. I am all for non-traditional medicine, but I don't discount the advances from University and Clinical research.
kdazzle 14y ago
It would have been 13% of papers reproducible if this blasted paper hadn't ruined things and included itself
teeja 14y ago
The tradition in science publishing has been to leave out the nitty-gritty details. Most specialists will know (or quickly deduce) most available methods (unless the results are a radical improvement). As for the details, that's what conferences and telephones and joining the department and gossip are for.
bjterry 14y ago
It seems like a big win for science could be had by centralizing some aspects of research within academic and research institutions. There should be well-funded software czars and statistics czars at each university that facilitate the efforts of the individual researchers to try to reduce the amount of shoddy science that gets published. Researchers are really terrible at this, and presumably slower than dedicated people even who are working on many different projects, so the division of labor should make the whole apparatus vastly more efficient, able to write and win more grants, etc. Of course, many researchers would fight against this because if you can't publish shoddy research, you can't publish as much research, but I would think there should be some way to get there from here.
narrator 14y ago
I read a lot of medical research papers and I'd say about 1 in 10 is about cancer. They usually go something like this: We found this receptor on a cancer cell and it caused it to grow or shrink. We therefore propose making a drug to inhibit or act as an agonist for this receptor site which also affects an enormous amounts of completely un-releated stuff around the body in order to fight cancer. Truth is, they may have just gotten a funny little mutant of a cancer cell that happens to heavily express this one receptor as part of its unique mutated biology. For all we know a lot of cancer research has been about exploring the peculiarities of the genome of Henrietta Lacks.
pg 14y ago
I feel obliged to point out that these other journals are merely copying the real pioneer in this field: http://jir.com http://jir.com.
- irollboozers 14y ago
  Negative results vary from field to field. In math or physics, a conclusive negative can be very important in deciding what to study next or disproving established theory. In biology, a negative usually just means "I haven't found what I'm looking for yet".
  - omnisci 14y ago
    And that is the exact problem with certain researchers. I just had to explain this to my undergrad who's project resulted in negative results (his 1'st study). I congratulated him, as he discovered that the treatment he gave did NOT effect the genes he looked into. He didn't understand, so I had to explain that his study (with the correct controls), provided information that is useful to science. Negative data = data. Data = good :)
elizabethiorns 14y ago
We are attempting to solve this by identifying and rewarding reproducible research (www.reproducibilityinitiative.org). However, it has been incredibly difficult to get funding to conduct the validation studies due to the obsession of funding "novel" results, rather than funding the replication studies required to identify reproducible research. Funders need to step up to solve this problem.
rosser 14y ago
Having read through these comments for the last several minutes, I'm left with the overwhelming conclusion that programmers, on the whole, have about as much to contribute to a discussion on the biological sciences as they do one on economics.
eavc 14y ago
There's nuance here that's going to be totally lost on the general public as this becomes an anti-science talking point.
CatMtKing 14y ago
Does reproducible in this context mean: The paper did not document their steps sufficiently, so the authors were unable to reproduce the experiment. -- or -- The paper relied on circumstantial evidence that was not well-understood enough to be reproduced. -- or -- The authors were able to perform the experiment as documented in the paper, but were unable to reproduce the results. or something else?
arrowgunz 14y ago
Why not just say 6 papers!
ttrreeww 14y ago
NaturalNews brought this back up recently.
ttrreeww 14y ago
Also, this is why evidence base medicine is flawed. What we want is science based medicine.
JDDunn9 14y ago
Perhaps part of this is statistical errors and selection bias. Most studies are done with a 95% confidence interval, but only studies that reject the null hypothesis are published. If we assume only 20% of all experiments reject the null hypothesis, then we should expect ~25% of those to be fasle positives. Journals might also be biased to include more sensationalistic results, which are more likely to be statistical errors. My guess is that if you included all test results, that reject the null hypothesis or not, and are published or not, then the reproducibility rate would be closer to 90%-95%.
- gwern 14y ago
  > My guess is that if you included all test results, that reject the null hypothesis or not, and are published or not, then the reproducibility rate would be closer to 90%-95%. No; you need to bring in power, not just assume an alpha of 0.05 and a base-rate of real results of 20%. (Consider the recent neuroscience paper estimating the experiments average a power of like 0.3...)
codex 14y ago
This can be explained without bringing in accusations of fraud or incompetence [1]. Basically, if you have a bazillion experiments running and only publish the results which are significant, you still have a huge predictability problem, because most of those statistically significant results will be due to chance alone. For example, assume that only 1% of cancer experiments produce something valid and interesting, and that standard for statistical significance is the industry standard 95%, or 1 in 20. If 100,000 hypothesis are tested, you'll get 100,000 * 1% = 1000 valid results and 100,000 * 5% = 5000 results from chance alone. In this example, only 16% of the published results will be reproducible, because only 1000 / 6000 are valid. [1] http://sciencehouse.wordpress.com/2009/05/08/why-most-published-results-are-false/ http://sciencehouse.wordpress.com/2009/05/08/why-most-publis...
- dkural 14y ago
  this basic fact is understood by scientists and children alike. it is still either incompetence (lack of middle school statistics) or fraud (ignoring the above fact).
smsm42 14y ago
So ironic that the paper about these findings is not reproducible itself because the initial data set is not available for legal reasons.
zmmmmm 14y ago
Cancer is such a weird beast that I'm actually not as alarmed by this as it might seem reasonable to be. Only in the last year or so people have realized that how heterogeneous cancer is. A patient biopsied 10 times from different parts of the cancer turns out to have 10 different (but related) cancers. While such major findings are still occurring about the underlying nature of cancer, it is not surprising that studies are hard to produce; they don't even know which parameters need to be controlled yet to make them reproducible.
podperson 14y ago
A scientific writer should state this result as 6 of 53, not 11%.
podperson 14y ago
This seems like an awesome example of Sturgeon's Law: 90% of everything is crap.
dkural 14y ago
I found a company to precisely fix this problem. Every dataset, every analysis, every computation fully transparent, reproducible, accessible. You can even embed it in your blog. For instance check out https://igor.sbgenomics.com/lab/public/pipelines/ https://igor.sbgenomics.com/lab/public/pipelines/ if someone uses one of those, and share their task, there is full transparency + instant replication.

292 points by vog 14y ago

The analysis in question: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html http://www.nature.com/nature/journal/v483/n7391/full/483531a...

micro_cam 14y ago
This is a really insightful article that contains a number of points that qualify the result and present lessons for any one interested in data science. A few points that stand out to me: !) The 11% number comes not from a randomized sample of cancer papers but from attempts made by AmGen to reproduce results that might be useful in drug development. This means that they made good effort to reproduce these results but there is also a strong selection bias that needs to be acknowledged. 2) They point out that using survival time as the measurement complicates things. I've done a lot of machine learning and statistics on cancer and medical data and, in my experience, this can not be overemphasized. and There are loads of confounding factors that contribute to survival time. I expect that big break throughs will come as we develop rigorous ways for measuring the behavior of a tumor (does it metastasize? how does it feed itself?) and use those as the targets of our regression. (Currently these thins are measured by human visual inspection if at all.) 3) They point out that the studies that were repeatable were the ones that were careful about using blinded controls, and eliminating investigator bias. This is basic stuff but easy to overlook. In terms applicable to a startup, a data scientist needs to me motivated to vet existing and proposed practices and identify flawed ones as much or more then they are motivated to maximize gain.
- pcrh 14y ago
  With regards to 1), it would be ironic if this study was itself not statistically valid...

mchusma 14y ago

This is a great issue to highlight. However, I'm unconvinced we need to optimize around creating research that can be reproducible, rather that the process of science generates the best possible results. Both the scientific and lay public should read this (appropriately) as: one study does not prove a result, there is an 89% chance that a single research paper is simply wrong. This mentality seems to be the bigger gap, although generating high quality research is a big part of the equation (mostly I think for the wasting of resources to validate, process, and reproduce claims).

ttrreeww 14y ago
Cold fusion was reproducible. What are you suggesting we do here?

wheaties 14y ago

This is one of the reasons I went into math and software. In math there is only black and white, provably true and patently wrong. Still, part of the reason this is so is that to show any results everything must be laid on the table. CS and Math papers are therefore open by design and the world has benefited. I hope the rest of science will move in this direction.

rajeevk 14y ago
CS has similar problem, specially in AI fields. Researches often claim the accuracy rate more than 95%, but in reality that is much lesser (in fact less than 10% in most of the claims).
- gordaco 14y ago
  Having worked in computer vision and robotics, I can attest to this.
  - jacobparker 14y ago
    Yes. Of the fields I've implemented algorithms from papers in, computer vision seems to be particularly adverse to discussing the (often critical) downsides of their algorithms. Often its something like "Camera calibration isn't exactly perfect? Well this scene reconstruction technique simply wont work."
- 5kg 14y ago
  Yes, that's why I no longer consider pursuing PhD in CS. It's a waste of life to reproduce non-reproducible papers that is full of buzzword. Hopefully, more journals and confs will follow the Open Access and Reproducible Research model like IPOL[1]. > Each article contains a text describing an algorithm andsource code, with an online demonstration facility and an archive of online experiments. The text and source code are peer-reviewed and the demonstration is controlled. [1]http://www.ipol.im/ http://www.ipol.im/
jamessb 14y ago
There are 'proofs' that contain errors, just as there are science papers that reach incorrect conclusions. Also, it's not necessarily easier to verify a proof than it is to repeat an experiment (or perform additional supporting experiments), especially for complex proofs which very few people know enough to completely understand (eg. of Fermat's last theorem, or the Poincare conjecture).

ilikejam 14y ago

An article claiming that only 11% of findings could be reproduced doesn't cite its sources, thus rendering the findings of the article unreproducible. Nice work!

insickness 14y ago
Exactly. This should be the top comment. Are we just supposed to assume their work was true based on trust? Perhaps this 'study' was actually just a fleecing meant to show that we take too many studies to be true based on simple trust.
- Ensorceled 14y ago
  It should not be the top comment. The original article is an explanation of WHY this was done. If a comment is posted explaining why the study was a bad idea or should not have been published despite this explanation THAT should be the top comment.
Ensorceled 14y ago
Nature wrote an editorial justifying the situation, you can read it here http://www.nature.com/nature/journal/v485/n7396/full/485041e.html http://www.nature.com/nature/journal/v485/n7396/full/485041e...
- return0 14y ago
  > "those authors required them to sign an agreement that they would not disclose their findings about specific papers" I don't know why this doesn't cause a huge stir in the scientific community. Seems like everyone is fine with people sweeping their negative data under the rug. Shameful is a very mild word for that.
  - Ensorceled 14y ago
    Agreed, but this study was specifically designed to highlight the problem not solve it. And the results would be useless, or non-existent, if they restricted themselves to scientists that would co-operate fully with the process and the possible "shaming" afterward.

viggity 14y ago

After hearing similar things about psychology papers, this is rather disconcerting. This is why I am a climate change skeptic, I don't know whether it is happening due to CO2 or not, but I am confident that science can't be confident when they can't do control experiments. You can't control for any variable when it comes to the climate, let alone all the reasonable ones. We have infinitely more capabilities to control variables when researching cancer than we do with the climate of something as massive and complex as the earths climate, and it turns out our confidence is cancer research may kind of suck. one interested in the subject should check out feynman's discussion on the psychological effect on scientific research. Millikan use bad assumptions for his oil drop experiment to determine the charge of an electron, but nobody would publish results that differed too much. http://en.m.wikipedia.org/wiki/Oil_drop_experiment http://en.m.wikipedia.org/wiki/Oil_drop_experiment

JoachimSchipper 14y ago
You took an almost entirely unrelated story and made it about your political hobbyhorse. Please keep politics off HN.
- viggity 14y ago
  The article was not about cancer. The whole point of the discussion is about reproducibility in scientific research. I was merely applying the same discussion and concept to a different branch of science. HN is a place for nerds to discuss interesting things. Things like science and it's applications. As far as I'm concerned, that is what I did.
- CamperBob2 14y ago
  And you just took an entire field off the table for discussion on the grounds that it looks like a "political hobbyhorse." Good job.
campnic 14y ago
I realized that by replying i'm probably encouraging you to continue posting this non-sense. I agree with Joachim below, this is barely related to the article.

kailuowang 14y ago

Scientific publications have a history of having papers contradictory to each other for a long time. In medical research, findings are often based on statics results from limited samples. From that perspective, it will be a big surprise if they are more accurate than public polls.

drorweiss 14y ago

So maybe we need rename Life Sciences to Life Arts?

languagehacker 14y ago

What the hell? That's 5.83 papers. How do you reproduce .83 of something?

drharris 14y ago
Um, 6 papers out of 53, significant digits give you 11%. Not that difficult.
- ndr 14y ago
  I guess saying "11% of 53 papers" was cooler than saying "6 out of 53". Meh.
  - drharris 14y ago
    Yeah, my guess is they used that form to imply 11% of all cancer research is unreproducible. But in ironic fashion, they can't show their own data and thus we can make no broad claims about it.

codeulike 14y ago

Most published scientific research papers are wrong, according to a new analysis. http://www.newscientist.com/article/dn7915-most-scientific-papers-are-probably-wrong.html http://www.newscientist.com/article/dn7915-most-scientific-p... Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 http://dx.doi.org/10.1371/journal.pmed.0020124 http://dx.doi.org/10.1371/journal.pmed.0020124

kailuowang 14y ago
Is this analysis a published scientific research?
- return0 14y ago
  Joking aside, the authors of these studies have highlighted the low statistical power of research in a number of fields (see also http://www.nature.com/nrn/journal/v14/n5/abs/nrn3475.html http://www.nature.com/nrn/journal/v14/n5/abs/nrn3475.html), which is a serious issue that often leads to over-interpretation. It's as if researches willing to publish more are rushing hasty studies out the door. It's a good thing that people can measure these things and ring the alarm
- gwern 14y ago
  Depends. By and large, Ioannidis's paper is pointing out the basic consequences of the alpha and beta parameters in Neyman-Pearson hypothesis testing, so there's a very good argument that this paper is merely statistics, and statistics is generally considered to be mathematics, and mathematics not science but its own thing.
deleted 14y ago
[deleted]

gordaco 14y ago

Wow, that's sad. Are we really so blind when it comes to cancer research? I recall that oncology journals usually have a ludicrously high impact index (as ludicrously as 5 digit IIRC); that means there are a lot of citations, which is an indicator that there is a lot of research going on. And with a lot of research going on, well, you can expect a lot of false positives. So, I'm wondering, could this be a case of cherry-picking or some kind of selection bias? It wouldn't be difficult to select a lot of bogus-sounding research and test it.

Anderkent 14y ago
Medical research in general is dubious at best. See for example http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal... There's even a startup trying to work around it: http://www.metamed.com/ http://www.metamed.com/. They look for papers related for a given condition on request, trying to find ones that are actually well done and promising.
Alex3917 14y ago
"And with a lot of research going on, well, you can expect a lot of false positives." Why would you expect a lot of false positives? Aren't results supposed to be reproducible at least 95% of the time?
- gordaco 14y ago
  I was thinking about absolute numbers, so 5% of a lot can still be a lot.
deleted 14y ago
[deleted]
atrus 14y ago
But who is going to pay to reproduce that research? What if that research took years to do?
- jerf 14y ago

api 14y ago

After decades of publish-or-perish sweatshop science I'm sure the great shining archive of scientific truth is sort of like an inbox with no spam filter.

evoloution 14y ago
Such a powerful sentence. Lately science looks like olympic games without anti-doping control...
- api 14y ago
  I'm becoming pretty convinced that the human future depends on the disruption and obsolescence of bureaucracies.
  - spikels 14y ago
    I think you are on to something. When science became so big that it first needed then eventually was run by "managers" I bet it lost much of its effectiveness. But how do we change this? For starters there is a lot of money involved for the institutions conducting this (apparently shoddy) research.
    - kanzure 14y ago
      > When science became so big that it first needed then eventually was run by "managers" I bet it lost much of its effectiveness. You can just do projects on your own, you know. There's nothing about science that absolutely requires an institution. http://groups.google.com/group/diybio http://groups.google.com/group/diybio

campnic 14y ago

I have been in discussions about this with one of my friends working in academic materials research. Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools. I'm talking opening their code in notepad, 'versioning' files by sending around zip files with numbers manually added to the end of the file name, etc. This doesn't even begin to scratch the surface of the 'reproducible results' problem. Often times, the software I've seen is 'rough' to be kind. Most times its not even possible to get the software running (missing some really specific library or some changes to a dependency which haven't been distributed) or its built for a super specific environment and makes huge assumptions on what can 'be assumed about the system.' This same software produces results which end up being published in journals. If any of these places had money to spend, I think there could be a valuable business in teaching science types how to better manage their software. Its really unfortunate that outside of a few core libraries (numpy, etc.) the default method is for each researcher to rebuild the components they need. I'm surprised about only 11% of results being reproducible. It seems lower then I'd expect. I agree we don't want to optimize for reproducibility, but obviously there is some problem here that needs to be addressed.

phireal 14y ago
I think that's what the folks at Software Carpentry [0] are trying to do. I went on one of their courses, and you're taught the basics of writing good software, version control and databases (SQLite). I've frequently recommended it to fellow scientists. [0] http://software-carpentry.org/ http://software-carpentry.org/
- campnic 14y ago
  This is great! Thanks for sharing.
bigiain 14y ago
Have you seen: http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/ "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it into running; and it's not pretty code. Academic code is about "proof of concept." These rough edges make academics reluctant to release their software. But, that doesn't mean they shouldn't. Most open source licenses (1) require source and modifications to be shared with binaries, and (2) absolve authors of legal liability. An open source license for academics has additional needs: (1) it should require that source and modifications used to validate scientific claims be released with those claims; and (2) more importantly, it should absolve authors of shame, embarrassment and ridicule for ugly code."

pesenti 14y ago

The problem is not that research is not reproducible. Researchers are humans and full of biases. The problem is too much emphasis is put on novelty vs. reproducibility. It took a long time for software engineers to recognize the value of testing/validating vs. creating new code. In the same way the research world needs to put new emphasis on reproducing/validating results.

return0 14y ago
The problem is that too much emphasis is put on prestige and reputation than on actual impact of the work. Coming from a physics background I always felt that life scientists tend to be overly audacious in their claims, am I alone in this?
- pesenti 14y ago
  Is it only in life sciences? Remember the cold fusion and the faster than light particle fiascos?

baldfat 14y ago

Pediatric Cancer gets only 4% of Federal funding and less than 1% of Private Drug company research. Also American Cancer Society less than 5 cents on the dollar goes to research. Pediatric Cancer is the leading cause of death for 15 and under. So to see this funding for cancer get thrown away by fraud is extremely upsetting.

refurb 14y ago
Those are some pretty misleading statistics. First off, pediatric cancer is NOT the leading cause of death for those 15 and under[1]. It's up there, but "unintentional injury" takes twice as many lives. Second of all, not many children die (1200 per year), so even if something is the leading cause of death, it doesn't mean it happens very often. Cancer kills 10 times as many people between the ages of 35-44. If you add up all adult cancers it's probably 20-30x the number of deaths. Also, just because money goes to adult cancer, doesn't mean it can't help. Most drug are developed for adults first and then tested in children. It's not like people are ignoring pediatric cancer. Before you get so upset you should really sit down and think these things through. [1]http://www.cdc.gov/injury/wisqars/pdf/10LCID_All_Deaths_By_Age_Group_2010-a.pdf http://www.cdc.gov/injury/wisqars/pdf/10LCID_All_Deaths_By_A...
- Alex3917 14y ago
  "Most drug are developed for adults first and then tested in children." It was also be unethical to develop drugs for children without first testing them on adults, since children can't give informed consent.

brudgers 14y ago

Perhaps we are looking for the wrong measure of success. This is not inclined plane ball rolling in fifth period Physics with Mr. Johannes. Reproducibility in cutting edge experiment is success. It means that what is being measured - even if the wrong thing - is within the control of the experimenter. It is absurd, but practical, to publish experimental results with the implication that they are reproducible based on a peer review of the results. "Well they look reproducible to me," when uttered by a peer reviewer is the current standard. All we are seeing is that the a priori reasoning of experts is not a substitute for empirical investigation of scientific claims. Let us not forget that a journal which finds no worthy articles this quarter has only two options, publish unworthy ones or publish no articles. I know how I react when a magazine subscription does not come.

atrus 14y ago
I wonder if it would be possible to produce a journal that only released an issue when it has, say, 10 articles? No monthly/quarterly schedule, you could have an issue release next week, or next year.
- brudgers 14y ago
  As a digital document it is easy. Because printing requires substantial coordination, it is less practical. The subscription model common to the scientific journal industry makes it contractually problematic. I wonder what portion of academic research is equivalent to content farming.

zipdog 14y ago

What is more troubling is not that so few of these results results are reproducible, but that it appears almost no-one is trying to reproduce the results of earlier studies. The ability of the scientists who wrote the paper to even get access to the resources necessary to try and reproduce the results is limited. I'm reminded of Feymann's Cargo Cult Science: "I was shocked to hear of an experiment done at the big accelerator at the National Accelerator Laboratory, where a person used deuterium. In order to compare his heavy hydrogen results to what might happen with light hydrogen he had to use data from someone else's experiment on light hydrogen, which was done on different apparatus. When asked why, he said it was because he couldn't get time on the program (because there's so little time and it's such expensive apparatus) to do the experiment with light hydrogen on this apparatus because there wouldn't be any new result. And so the men in charge of programs at NAL are so anxious for new results, in order to get more money to keep the thing going for public relations purposes, they are destroying--possibly--the value of the experiments themselves, which is the whole purpose of the thing. It is often hard for the experimenters there to complete their work as their scientific integrity demands." For anyone who hasn't read it, the whole thing is excellent http://www.lhup.edu/~DSIMANEK/cargocul.htm http://www.lhup.edu/~DSIMANEK/cargocul.htm

varjag 14y ago
In practical terms, reproducing someone else's work most of the time boils down to redoing someone else's PhD thesis. Which is both not very interesting and doesn't help getting your own research done.
- rdfi 14y ago
  I agree that reproducing someone else's experiment won't help you get your PhD closer to completion (because you should be doing your own experiments and publishing no matter what, if you expect to land that lecturing position, i.e. publish or perish). Reproducibility is one of the main principles of the scientific method (those papers shouldn't even been accepted if they don't contain enough information about how to reproduce the experiments they describe) If an experiment is so complex that it can be compared to redoing someone else's thesis it can be either that the thesis is very simple or the experiment is so complex that it probably proves nothing.

psaintla 14y ago

There are many reasons so many papers are not reproducible. 1.) The prospective data collection methods used in a lot of studies are deeply flawed and worse aren't documented. 2.) The retrospective data used in many studies is poorly validated and the quality is rarely a concern for most universities. 3.) The people who publish these papers are very intelligent, most far more intelligent than I am, but you'd be surprised how many of them don't have a very good handle on basic statistics. Not because they aren't smart enough to learn but mostly because they have no interest in it. 4.) There is a lot of pressure put on researchers to publish papers even if they aren't ready and as a result mistakes are made, there really needs to be less of an emphasis on the quantity of papers that are published and more emphasis on quality. For anyone who might be interested in such things, if you ever find yourself locked in a dungeon with nothing but a computer loaded with every journal article ever published you can find articles involving similar cohorts from the exact same universities that have different results.

tokenadult 14y ago

This editorial commentary and the article on which it is based are part of an ongoing effort to improve the quality of scientific publication in a number of disciplines. The Retraction Watch group blog http://retractionwatch.wordpress.com/ http://retractionwatch.wordpress.com/ by two experienced science journalists picks up many--but not all--of the cases of peer-reviewed research papers being retracted later from science journals. Psychology as a discipline has been especially stung by papers that cannot be reproduced and indeed in many cases have simply been made up. http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-audacious-academic-fraud.html?pagewanted=all&_r=0 http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-... That has prompted statistically astute psychologists such as Jelte Wicherts http://wicherts.socsci.uva.nl/ http://wicherts.socsci.uva.nl/ and Uri Simonsohn http://opim.wharton.upenn.edu/~uws/ http://opim.wharton.upenn.edu/~uws/ to call for better general research standards that can be practiced as checklists by researchers and journal editors so that errors are prevented. Jelte Wicherts writing in Frontiers of Computational Neuroscience (an open-access journal) provides a set of general suggestions Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020 http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full http://www.frontiersin.org/Computational_Neuroscience/10.338... on how to make the peer-review process in scientific publishing more reliable. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence. "With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses." Uri Simonsohn provides an abstract (which links to a full, free download of a funny, thought-provoking paper) http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2160588 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2160588 with a "twenty-one word solution" to some of the practices most likely to make psychology research papers unreliable. He has a whole site devoted to avoiding "p-hacking," http://www.p-curve.com/ http://www.p-curve.com/ an all too common practice in science that can be detected by statistical tests. He also has a paper posted just a few days ago http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2259879 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2259879 on evaluating replication results (the issue discussed in the commentary submitted to open this thread) with more specific tips on that issue. "Abstract: "When does a replication attempt fail? The most common standard is: when it obtains p>.05. I begin here by evaluating this standard in the context of three published replication attempts, involving investigations of the embodiment of morality, the endowment effect, and weather effects on life satisfaction, concluding the standard has unacceptable problems. I then describe similarly unacceptable problems associated with standards that rely on effect-size comparisons between original and replication results. Finally, I propose a new standard: Replication attempts fail when their results indicate that the effect, if it exists at all, is too small to have been detected by the original study. This new standard (1) circumvents the problems associated with existing standards, (2) arrives at intuitively compelling interpretations of existing replication results, and (3) suggests a simple sample size requirement for replication attempts: 2.5 times the original sample." The writers of scientific papers have a responsibility to do better. And the readers of scientific papers that haven't been replicated (or, worse, press releases about findings that haven't even been published yet) also have a responsibility not to be too credulous. That's why my all-time favorite link to share in comments on HN is the essay "Warning Signs in Experimental Design and Interpretation" by Peter Norvig, LISP hacker and director of research at Google, on how to interpret scientific research. http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html Check each submission to Hacker News you read for how many of the important issues in interpreting research are NOT discussed in the submission.

pcrh 14y ago

I work in biomedical research, and this finding has been discussed quite broadly. Most researchers don't believe it. Exactly reproducing novel findings usually requires a significant investment into the underlying procedures, which most places will not undertake. Reproduction instead usually occurs as part of an extension of the initial findings. The complexity of biology means that the first findings are frequently not reproduced exactly the same way as the original, but this does not detract from the "direction" of the initial findings. For example, the first finding might be that protein A promotes tumor growth by modifying protein B. An extension of these findings might be Protein A sometimes modifies protein B, and when it does tumor growth is stimulated, but it mostly does not modify protein B, it instead modifies protein C, which suppresses tumor growth. In this case, were the first results replicated? Yes, and no. This is how most biomedical research proceeds...

rdfi 14y ago
Really? Meaning that either there's another unaccounted variable(s) that controls the effect of A in B and C, meaning that we cannot conclude anything from the experiment. Yes and no is not acceptable...
- epistasis 14y ago
  There's 20,000 broad classes of variables (genes), and on average 12 variations of each of those variable (alternative splicing, post-translational modification, cellular localization, other variables we haven't yet discovered ). This series of two experiments and likely 2 years found associations between three proteins, which is a hook into system of great complexity, a way to start looking for more. It is not "reproducible" yet it is incredibly valuable.
- pcrh 14y ago
  It is exactly as you suggest, additional variables. Bear in mind that a single human cell has ~25,000 genes, and therefore at least that many potential degrees of freedom, even before you include external variables. It is very difficult to control for all of them.
- 14y ago

jballanc 14y ago

I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on... Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being completely worthless, snake-oil type stuff, but this was not the case at all. Rather, individuals were working on an area about which almost nothing was known, and (more importantly) for which the key central organizing laws were not yet revealed (for Alchemy: the atomic theory of matter and chemical bonding, for Cell Signaling: how individual proteins and small molecules interact). Furthermore, the goals were lofty and almost certainly unattainable (for Alchemy: turn base metals into gold, for Cell Signaling: cure cancer), driving people to do "rush" work. It's not that the results are completely invalid, or that the experiments are useless. It's just that everyone feels like they're so close to a solution (Alchemists were way off with Phlogiston, and I'll bet Cell Signaling researchers are similarly clueless as to what really matters) that no one takes the time to step back and synthesize the results in an attempt to understand the forest from the trees. Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev. From experience, the fact that 89% of these "cancer research" papers are not reproducible almost definitely has less to do with fraudulent data and so much more to do with crazy complex experimental setups that end up probing a half-dozen experimental variables all at once (without the researchers even grasping that this is going on). Yeah, publish or perish sucks, but what sucks more is the death of basic science research. The Alchemists eventually became Chemists because they refocused on core principles (atomic theory, bonding) and forgot about the lofty goals (turn lead into gold)... ...but try telling any politician that they should decrease cancer research funding and refocus on genetics, structural biology, and evolution research. I'd love to know how they respond.

jpdoctor 14y ago
> It's not that the results are completely invalid Especially if only 11% of them are reproducible, they're only 89% invalid. I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.

nogoodnik 14y ago

The most interesting thing about all this is the fact that the researchers were required by paper authors to sign non-disclosure agreements and agree not to identify papers with non-reproducible results. This says a lot about the confidence authors have in their papers, doesn't it? Sorry but this isn't science anymore.

ttrreeww 14y ago
It's good old capitalism making money from "treating" sickness. The longer the better. A lifetime of "treatment" means a lifetime of profits. Who cares about the cure.

pessimizer 14y ago

There's an awful lot of money in cancer (research funding/patents), an awful lot of diet/lifestyle handwaving in cancer research conclusions (that aren't about treatments), and very little progress in making cancer more survivable. Just those facts would lead me to assume that most of the published research is bad. The great part is that 11% of the papers are reproducible, so the science isn't at a standstill.

ttrreeww 14y ago
Cancer is one of those sickness where you either cure it, or you die. Problem is, for profit companies seeks "treatment" to maximize profit, they don't seek the cure. Hence no progress on cancer.
Mz 14y ago
That isn't limited to cancer. Western medicine is driven by a lot of bad mental and social models. Doctors only make money if you are sick. That strikes me as a seriously flawed model right there.

japhyr 14y ago

This is one of the reasons I am happy to see what is going on with the Center for Open Science [0]. The goal of the project is to open up the entire research process, not just create open access to published results. They aim to make open tools supporting the entire research process as well, through the Open Science Framework [1]. One of the benefits of this approach is to provide peer review at each step of the research process. There is also an emphasis on encouraging validation of prior results, rather than having everyone focus on creating new research. This is the focus of the Reproducibility Project [2]. The Center is just getting started, and I sure hope it takes off. [0] - http://centerforopenscience.org/ http://centerforopenscience.org/ [1] - http://openscienceframework.org/ http://openscienceframework.org/ [2] - http://openscienceframework.org/project/EZcUj/wiki/home http://openscienceframework.org/project/EZcUj/wiki/home

brandon_wirtz 14y ago

This has been going around along with the Chemo Therapy doesn't work thread. The problem with people like the author is they think all cancer is the same. They try to apply information about Hodgkins to non-hodgkins. They think "Breast cancer" is a disease, not a symptom. There over 40 different cancer causes for breast cancer. And the treatments that work are primarily based on the cause not the visible symptom. This makes studies hard. Most people never learn the cause of their cancer. We are not fortunate enough to always have a single known cause that says "yes you worked in a nuclear waste plant for 6 years" and know that was the cause. Chemo for example in Hodgkin's Lymphoma increase your 5 year survival chances by 45%. But if you have Smoking related Lung Cancer it is less than 2% difference. Nature.com rarely produces articles that are informed. They push an agenda of Holistic medicine at the expense of scientific research. I am all for non-traditional medicine, but I don't discount the advances from University and Clinical research.

kdazzle 14y ago

It would have been 13% of papers reproducible if this blasted paper hadn't ruined things and included itself

teeja 14y ago

The tradition in science publishing has been to leave out the nitty-gritty details. Most specialists will know (or quickly deduce) most available methods (unless the results are a radical improvement). As for the details, that's what conferences and telephones and joining the department and gossip are for.

bjterry 14y ago

It seems like a big win for science could be had by centralizing some aspects of research within academic and research institutions. There should be well-funded software czars and statistics czars at each university that facilitate the efforts of the individual researchers to try to reduce the amount of shoddy science that gets published. Researchers are really terrible at this, and presumably slower than dedicated people even who are working on many different projects, so the division of labor should make the whole apparatus vastly more efficient, able to write and win more grants, etc. Of course, many researchers would fight against this because if you can't publish shoddy research, you can't publish as much research, but I would think there should be some way to get there from here.

narrator 14y ago

I read a lot of medical research papers and I'd say about 1 in 10 is about cancer. They usually go something like this: We found this receptor on a cancer cell and it caused it to grow or shrink. We therefore propose making a drug to inhibit or act as an agonist for this receptor site which also affects an enormous amounts of completely un-releated stuff around the body in order to fight cancer. Truth is, they may have just gotten a funny little mutant of a cancer cell that happens to heavily express this one receptor as part of its unique mutated biology. For all we know a lot of cancer research has been about exploring the peculiarities of the genome of Henrietta Lacks.

pg 14y ago

I feel obliged to point out that these other journals are merely copying the real pioneer in this field: http://jir.com http://jir.com.

irollboozers 14y ago
Negative results vary from field to field. In math or physics, a conclusive negative can be very important in deciding what to study next or disproving established theory. In biology, a negative usually just means "I haven't found what I'm looking for yet".
- omnisci 14y ago
  And that is the exact problem with certain researchers. I just had to explain this to my undergrad who's project resulted in negative results (his 1'st study). I congratulated him, as he discovered that the treatment he gave did NOT effect the genes he looked into. He didn't understand, so I had to explain that his study (with the correct controls), provided information that is useful to science. Negative data = data. Data = good :)

elizabethiorns 14y ago

We are attempting to solve this by identifying and rewarding reproducible research (www.reproducibilityinitiative.org). However, it has been incredibly difficult to get funding to conduct the validation studies due to the obsession of funding "novel" results, rather than funding the replication studies required to identify reproducible research. Funders need to step up to solve this problem.

rosser 14y ago

Having read through these comments for the last several minutes, I'm left with the overwhelming conclusion that programmers, on the whole, have about as much to contribute to a discussion on the biological sciences as they do one on economics.

eavc 14y ago

There's nuance here that's going to be totally lost on the general public as this becomes an anti-science talking point.

CatMtKing 14y ago

Does reproducible in this context mean: The paper did not document their steps sufficiently, so the authors were unable to reproduce the experiment. -- or -- The paper relied on circumstantial evidence that was not well-understood enough to be reproduced. -- or -- The authors were able to perform the experiment as documented in the paper, but were unable to reproduce the results. or something else?

arrowgunz 14y ago

Why not just say 6 papers!

ttrreeww 14y ago

NaturalNews brought this back up recently.

ttrreeww 14y ago

Also, this is why evidence base medicine is flawed. What we want is science based medicine.

JDDunn9 14y ago

Perhaps part of this is statistical errors and selection bias. Most studies are done with a 95% confidence interval, but only studies that reject the null hypothesis are published. If we assume only 20% of all experiments reject the null hypothesis, then we should expect ~25% of those to be fasle positives. Journals might also be biased to include more sensationalistic results, which are more likely to be statistical errors. My guess is that if you included all test results, that reject the null hypothesis or not, and are published or not, then the reproducibility rate would be closer to 90%-95%.

gwern 14y ago
> My guess is that if you included all test results, that reject the null hypothesis or not, and are published or not, then the reproducibility rate would be closer to 90%-95%. No; you need to bring in power, not just assume an alpha of 0.05 and a base-rate of real results of 20%. (Consider the recent neuroscience paper estimating the experiments average a power of like 0.3...)

codex 14y ago

This can be explained without bringing in accusations of fraud or incompetence [1]. Basically, if you have a bazillion experiments running and only publish the results which are significant, you still have a huge predictability problem, because most of those statistically significant results will be due to chance alone. For example, assume that only 1% of cancer experiments produce something valid and interesting, and that standard for statistical significance is the industry standard 95%, or 1 in 20. If 100,000 hypothesis are tested, you'll get 100,000 * 1% = 1000 valid results and 100,000 * 5% = 5000 results from chance alone. In this example, only 16% of the published results will be reproducible, because only 1000 / 6000 are valid. [1] http://sciencehouse.wordpress.com/2009/05/08/why-most-published-results-are-false/ http://sciencehouse.wordpress.com/2009/05/08/why-most-publis...

dkural 14y ago
this basic fact is understood by scientists and children alike. it is still either incompetence (lack of middle school statistics) or fraud (ignoring the above fact).

smsm42 14y ago

So ironic that the paper about these findings is not reproducible itself because the initial data set is not available for legal reasons.

zmmmmm 14y ago

Cancer is such a weird beast that I'm actually not as alarmed by this as it might seem reasonable to be. Only in the last year or so people have realized that how heterogeneous cancer is. A patient biopsied 10 times from different parts of the cancer turns out to have 10 different (but related) cancers. While such major findings are still occurring about the underlying nature of cancer, it is not surprising that studies are hard to produce; they don't even know which parameters need to be controlled yet to make them reproducible.

podperson 14y ago

A scientific writer should state this result as 6 of 53, not 11%.

podperson 14y ago

This seems like an awesome example of Sturgeon's Law: 90% of everything is crap.

dkural 14y ago

I found a company to precisely fix this problem. Every dataset, every analysis, every computation fully transparent, reproducible, accessible. You can even embed it in your blog. For instance check out https://igor.sbgenomics.com/lab/public/pipelines/ https://igor.sbgenomics.com/lab/public/pipelines/ if someone uses one of those, and share their task, there is full transparency + instant replication.