15 ms·

Plan to replicate 50 high-impact cancer papers shrinks to just 18

253 points by nonbel 8y ago

nonbel 8y ago
I was mentioning this in another thread and came across the new info I hadn't heard about yet. Here are some previous articles about this: March 2012: https://www.nature.com/articles/483531a https://www.nature.com/articles/483531a June 2015: http://www.sciencemag.org/content/348/6242/1411 http://www.sciencemag.org/content/348/6242/1411 Dec 2015: https://www.nature.com/news/cancer-reproducibility-project-scales-back-ambitions-1.18938 https://www.nature.com/news/cancer-reproducibility-project-s... Jan 2017: https://www.nature.com/news/cancer-reproducibility-project-releases-first-results-1.21304 https://www.nature.com/news/cancer-reproducibility-project-r...
Palomides 8y ago
>says total costs for the 18 completed studies averaged about $60,000, including two high-priced that seems really cheap to me based on my brief experience with lab research. replicating 18 studies is a great achievement.
- jonlucc 8y ago
  It's not terrible for in vitro studies, but that would be a stretch for in vivo studies at a CRO, especially for oncology, where lots of studies require tumor grafts. It's not my area, but I think those mice can run $1000 and up per animal. The cheap, more common strains are ~$25 each. I haven't looked at the papers they're trying to replicate, but in my lab, it's not unusual to have 60 mice in a study, so you can see how just getting animals into your facility can be expensive.
  - dingo_bat 8y ago
    $1000 per mouse? Seems really expensive.
    - amelius 8y ago
      Annual cost of small mammal ownership: $300 Source: https://xkcd.com/980/huge/#x=-1166&y=-2798&z=5 https://xkcd.com/980/huge/#x=-1166&y=-2798&z=5 Add to that expenses for the experiment, and $1000 seems not far fetched.
      - mattkrause 8y ago
        That’s for a pet. While labs have huge economies of scale—-I worked somewhere with a quarter million mice—-the mice aren’t just random field mice. Some of the genetically engineered ones are hundreds of dollars a pair (and possibly more if custom, raised under unusual conditions, etc). Xenografts need skilled labor and very clean conditions for the immunocompromised mice. $1000 seems a little high, but not much, especially if the work isn’t being done by an underpaid grad student.
    - jonlucc 8y ago
      It is; that's my point. As mattkrause notes below, that number is for mice with human tumor xenografts. You have to harvest a human patient's tumor, blend it up, and inject it into immunocompromised mice. If you buy them from the vendor who has to buy their tumors, pay their labor, and make a profit, they're not cheap. If you have ready access to clinical tumors and cheap labor, the price drops precipitously to ~$70. By the way, you can poke around some of the animal vendor's sites. They mostly have their pricing on their websites along with a bunch of other information. Four really big ones are Taconic, Jackson Labs, Envigo, and Charles River.
- deleted 8y ago
  [deleted]
Alex3917 8y ago
> organizers realized they needed more information and materials from the original authors; Then the study already isn't replicable by definition, so why waste time asking the original authors? Just mark it down as 'not replicable' and move on.
- jonlucc 8y ago
  I'm not so sure this is the right way to go about it. They give the example of cell densities, but there are also tons of things that just are never reported in in vivo papers, including the light cycle in animal rooms (mice and rats are nocturnal, so some labs make their rooms dark during the day), the preparation of treatment compounds, information about animal ages, handling procedures, etc. In addition, many papers have length limits, so you can only include so much.
  - Alex3917 8y ago
    Sure, all of which make the research non replicable. That stuff is all important, you can't just discount it because it's the norm not to include it.
    - TallGuyShort 8y ago
      It's important that it become the norm to include it, though. So rather than just moving on, they're trying to gather that information so that if they CAN reproduce it, they now know how to make sure the next experiment can be more reproducible.
    - flukus 8y ago
      They should be replicable despite a lot of that, if they aren't then then the original study is garbage.
      - Alex3917 8y ago
        Getting the same result doesn’t mean a study is replicable or a result is reproduceable. If you don’t use the same methodology it’s meaningless.
kayhi 8y ago
I wrote my concerns about replicating these studies a couple of years ago [1]. In short, most papers don't have enough details to allow for a third party to replicate them. Problem areas include detailed protocols and reagents used. If you don't what someone did exactly and what they used then replicating it is going to be very difficult. [1] https://www.linkedin.com/pulse/scientific-reproducibility-reagent-variability-sean-seaver/ https://www.linkedin.com/pulse/scientific-reproducibility-re...
- nonbel 8y ago
  That is exactly the problem. All this stuff is supposed to be in the paper, the paper is not a press release. And everyone understands that its hard to know exactly what info someone else will be missing, so it wont be perfect at first. That is why there is supposed to be a back and forth the first time a new method is used. Figuring out what needs to go into the methods is going to be an iterative process. As it is right now its a disaster though. Like they say in this article no one even knows the most basic info like cell density. The entire system is not set up to deal with replications at all because they haven't been doing them.
  - rjkennedy98 8y ago
    > All this stuff is supposed to be in the paper, the paper is not a press release. Welcome to the real world. Science doesn't work like it was described in your 7th grade science class.
    - pas 8y ago
      But it can be approximated. Repeatable experiments. Done by machines, properly accounted ingredients and so on. Physics does it. It should be perfectly doable with cell cultures too. Mice are of course a lot harder. But if we wouldn't wait decades to replicate, but could simply get in touch with the originators - and so do a collaborative effort in establishing the parameters.
    - nonbel 8y ago
      These sayings like "welcome to the real world", "there's a difference between theory and practice", thought-killers are just excuses for being lazy. Science is hard, really hard. People who think doing a good job is too hard should get out of science instead of polluting the literature with their statistically significant results or whatever. Unfortunately its been the opposite happening and the people who cant handle it have been training the new people how to get away with half-assing it. This has been going on for a long time now, at least since the 1940s in some areas...
Avshalom 8y ago
turns out we can't raise the money to replicate researchers who barely had the money to do what they did.
- nonbel 8y ago
  I'd honestly say if the study isn't worth replicating then don't fund it to begin with. Its like funding it enough to get some animals or cells into your lab but then not enough for gloves and pipets and computers.
  - Avshalom 8y ago
    That sounds great in abstract but every study is subject to that logic, including studies studying studies. Which means that every study should be matched by an infinite regression of studies, which is untenable to put it mildly and mathematically fucking ridiculous to put it conservatively.
    - nonbel 8y ago
      There is no infinite regression, what are you talking about? Its that at least two groups try to measure the same thing under the same conditions (as similar as you can get them).
      - Avshalom 8y ago
        how do you know that those two groups aren't collaborating for the same result? Or biased in the same direction of the result? you don't. You either add a 3rd party (4th... 5th...6th... Xth)or accept the conclusion. If you add a third party how do you know they are trust worthy: infinite regression.
        nonbel 8y ago
        Its not about perfection, its about taking the simple, common sense, and time-tested precautions that gave us the technological marvels we enjoy today. But of course the more independent, even adversarial, the two groups are the better. Like in this story it mentions the "critics" of this project claim other labs (their friends?) have already replicated these studies, apparently using the super secret protocol they couldn't explain to this group. I'd compare it to using sms 2FA. Is it perfect, no. But it is far, far better than no 2FA at all. And there's nothing stopping you from putting your banking app on the same phone you use for 2FA.
Gatsky 8y ago
Replicating cancer papers: This is one of those ideas which although pointless and impractical, is seemingly impossible to criticize. Now the leaders of this project are basically saying - we can't do difficult experiments, and can't replicate studies which have already been replicated by others. What have we learned exactly, except that experimental biology is difficult?
- drb91 8y ago
  What's the point of cancer papers at all if they're un-reproducable? Are we just sinking government money into artificially inflated bio portfolios?
- pietroglyph 8y ago
  > pointless and impractical These papers underlie a lot of modern research and treatments. If they aren't reproducible, the papers' results, and other papers that use their results come into question. This is a serious, documented problem [0]. > What have we learned exactly, except that experimental biology is difficult? The problem wasn't that the project couldn't do difficult experiments, it was that not enough information was provided by the original papers. That's what we learned here. [0]: https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
  - Gatsky 8y ago
    I disagree with your characterisation of the problem and the proposed solution. Cancer research is not an edifice constructed on foundational results. It is not like physics in this regard, where for example, accurate calculation of the gravitational constant is vitally important, and over time repeat measurements are generally more precise but not wildly different, or do not dispute the importance of the gravitational constant. Take one of the papers they reproduced - Transcriptional amplification in tumor cells with elevated c-Myc. This is an interesting result conducted in highly contrived experimental conditions. Nobody is going to base their career, a drug development program, treat a patient or even start a PhD based on this result alone. I say this is a translational cancer scientist and medical oncologist. The contribution of this paper is to our knowledge of the biology of Myc, which has a multitude of actions which are context dependent. Myc is studied in a variety of different ways using many different methods. If this paper were being repeated today, the technologies and techniques would be quite different. The result of this paper is not plugged into the central dogma of cancer biology, setting us down an erroneous path for the next thousand years. So to return to my original question, what have we learned in trying to replicate this study, that we didn't already know? The money would have been better spent on orthogonal validation/extension of the result using modern techniques - another name for this is 'science'. The replication crisis suggests that this routine extension/validation process is somehow less important than going back and repeating the original experiment, which is I think a complete misunderstanding. You also seem to be saying that we should ask researchers to document in excruciating detail all experimental conditions such that a pastry chef or meteorologist could walk into a lab and successfully reproduce the experiment - this is an impossibly high bar to set for scientists who are already working under very difficult conditions, and is not the solution.
StanislavPetrov 8y ago
They should have just become sociologists so they could have dispensed with any reality-based interference at all.
- titanix2 8y ago
  I don’t get the downvotes, there is some truth here. I even skimmed a doctoral thesis supposed to be in the field of linguistics where the author declared her work was also made as an activist. Too many papers made by too many "academics" are straight made up ideological essays with no grounding in reality nor in science.
  - deleted 8y ago
    [deleted]
  - StanislavPetrov 8y ago
    Those ideological "academics" are the ones downvoting.
    - mattkrause 8y ago
      Nope—-one came from me, and I’m a systems neuro/CS staff scientist. Social science is incredibly important. Psychology departments have contributed so much to our understanding of perception and attention, which in turn has paved the way for safe and effective user interfaces, better compression algorithms, and prosthetics. Sociology can tell you how to optimize anything involving groups of people, whether it’s effectively provisioning government services or running an aircraft crew. Linguistics has obvious applications to speech and text processing, education, and even medicine. All this and more, plus you get to learn about what it means to be human! And it’s cheap to boot! Saving one airplane funds a hell of a lot of cockpit management studies. Sure, there is some crappy social science, but there are also crappy programmmers. The blatant dismissal of whole fields is the worst kind of pseudointellectual snobbery. By all means, call out work that’s subpar, but if you can’t see how social science is both interesting and useful, it’s only because you’re not trying.
- 8y ago
rrock 8y ago
Wait, they thought they could replicate 50 papers at the cost of $25k each, and all in the span of one year? Not a chance. I don’t think these guys have any idea what they’re doing. I feel sorry for the funders.
- lsh 8y ago
  It's good to have reasonable goals and a budget, even if some of them aren't met. The funders absolutely know what they are doing and handicapping a replication effort because it went a few thousands over budget would be outrageous. There are still unknown quantities at play here, a large scale replication effort hasn't been attempted before and those efforts themselves - the extent of the effort required to replicate and the impact of failure - are also goals. Failure results, of which there have been several already now, can severely damage the reputation of prestigious journals and teams of scientists. They're not doing this because replication is popular, or cheap or because they thought it would be a fun afternoon lark.
- LyndsySimon 8y ago
  They thought that because they’d already done 100 in another field: https://osf.io/ezcuj/ https://osf.io/ezcuj/ I worked there for three years :)
  - lsh 8y ago
    thank you for the link
  - rrock 8y ago
    I'm going to guess that psychology research is far less expensive. To give you a sense of perspective here, $25k will get you something like 4 months of a full time employee at the "fresh out of college" experience level, including benefits and research consumables. That also assumes that you have all of the equipment in place. If all there doing is running a few western blots, because that's all they can do on this budget, well then ok. But I'll venture to guess that that's a far cry from reproducing the most important results in these papers.
buchanae 8y ago
I have been working on reproducing computational biology papers from the cancer field lately. I am very frustrated. When the inputs and outputs are machine-readable data, there's no excuse for not making your work reproducible, in my opinion. Often the problem is plain laziness and disorganization. One major problem is that there's not much real incentive to make your work reproducible. Money granting organizations favor researchers breaking new and exciting ground, not those rehashing an already published method. Publishers don't require reproducible methods, and reviewers don't have the time, desire, nor expertise to do an in-depth methods review. Wet lab experiments are 1-2 orders of magnitude more expensive and difficult to reproduce, that's true, but we're not even getting the basics right!
- buchanae 8y ago
  And another thing! Why should a project be done when it gets published?! What kind of software project would make one release and call it good forever? Nonsense.
  - fmap 8y ago
    In an ideal world you would be right, but that's not how the funding structures for academic research are set up. Think of a research lab as a company that gets paid per prototype and then has to market the concept for the next prototype in an infinite loop. If you can't package up what you're doing into a sequence of small prototypes then you're not getting paid.
    - ajuc 8y ago
      The requirement for publishing in a scientific journalshould be opensourcing all the code used. You can't expect the results to be reproductible 20 years later otherways.
      - skummetmaelk 8y ago
        20 year old code that has not been maintained most likely would not run on a modern system anyway, unless it is extremely simple.
gnusci 8y ago
This is a well known problem of scientific publications now-a-days: https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 https://www.nature.com/news/1-500-scientists-lift-the-lid-on... https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis Papers have become a target for success, so scientist need publications for better status and remuneration: http://www.sciencemag.org/news/2017/08/cash-bonuses-peer-reviewed-papers-go-global http://www.sciencemag.org/news/2017/08/cash-bonuses-peer-rev... A new scientific "mafia" is in place around the world: https://www.technologyreview.com/s/608266/the-truth-about-chinas-cash-for-publication-policy/ https://www.technologyreview.com/s/608266/the-truth-about-ch... https://retractionwatch.com/2017/08/10/paid-publish-not-just-china/ https://retractionwatch.com/2017/08/10/paid-publish-not-just... So papers have become a good business, no the way to disseminate outstanding research results.
- lsh 8y ago
  > So papers have become a good business, no the way to disseminate outstanding research results. That's awfully cynical and over-broad, but I agree to a point. Greedy and unscrupulous publishers are part of the problem, but so are lax or unprincipled scientists eager for prestige and a career-making publication in a top tier journal. It's an unfortunate chicken-and-egg cycle now with no easy way to cut it. Perhaps more emphasis on replication post-publication? Perhaps a reputation system for unethical publishers or scientists?
  - timr 8y ago
    "Greedy and unscrupulous publishers are part of the problem, but so are lax or unprincipled scientists eager for prestige and a career-making publication in a top tier journal." That's just incredibly unfair. There are some fields and methodologies where p-hacking and cherry-picking have been a problem, but the primary reason that papers aren't reproducible is just noise and basic statistics. As a scientist, you control for what you can think of, but there are often way too many variables to control completely, and it's probable that you miss some. Those variables come to light when someone else tries to work with your method and can't reproduce it locally. However, real scientists don't stop and accuse the original authors of being "unprincipled" -- nine-point-nine times out of ten, they work with the original authors to discover the discrepancies. It isn't surprising at all to actual, working scientists that most papers are impossible to reproduce from a clean room, using only the original paper. It's the expected state of affairs when you're working with imperfect, noisy techniques, and trying tease out subtle phenomena.
atdt 8y ago
Why, apart from expedience, do designers of experiments share an institutional affiliation with the experimenters themselves? I wonder what would happen if the two groups were far apart, and if communication between them restricted to the transmission of an experimental protocol and the reception of raw observations.
- nonbel 8y ago
  This is a really good idea. I bet it is not as hard as some of the "real scientists" claim.
lsh 8y ago
A direct link to the results: https://elifesciences.org/collections/9b1e83d1/reproducibility-project-cancer-biology https://elifesciences.org/collections/9b1e83d1/reproducibili... There have been several news items on Hacker News recently about academic publishing, reproducibility in general and pre-print servers and I would love it if eLife got some more attention, for better or worse, for the excellent work they are doing and helping to fund. To those biology researchers in the comments, please submit your work! even failed results! Even if it's software heavy. Editors are listening closely to your feedback. edit: disclosure, I work for eLife
- chmaynard 8y ago
  Does eLife have an RSS feed? I can't find one listed on your website.
  - rossmounce 8y ago
    Yes, various RSS feeds. Alert options explained here: https://elifesciences.org/alerts https://elifesciences.org/alerts early versions: https://elifesciences.org/rss/ahead.xml https://elifesciences.org/rss/ahead.xml final published versions: https://elifesciences.org/rss/recent.xml https://elifesciences.org/rss/recent.xml
- Q6T46nT668w6i3m 8y ago
  eLife is excellent.
anticensor 8y ago
It turned out remaining 32 are unworthy to replicate.
adiusmus 8y ago
Initiate Rant. Most science isn’t. Most of it is a set of nearly fanatical beliefs about reality only vaguely related to facts agreed through consensus by authority. Look at how strongly doctors fought against germ theory. They saw no reason to wash their hands for delivering a new born after having spent their morning cutting open cadavers. Their response was anything but scientific even in the face of easily tested claims and results. If research can’t be reproduced it’s just a story with interesting data. I doubt most scientists using statistics would be able to provide the alpha / p value / confidence interval etc for their hypothesis. When the bar is set so low should we be surprised by low quality? Rant complete. Initiate beer.
- southern_cross 8y ago
  Hand washing may still be a problem. I saw a blurb recently which stated that when doctors know that they're being monitored for using proper hand washing protocols, compliance was near the top end of the scale. But when they thought they were no longer being watched, compliance fell by something like two-thirds. So in other words, doctors (and no doubt others) probably still aren't anywhere close to doing it right.
  - Engineering-MD 8y ago
    That would be because “doing it right” is completely over the top. You are suppose to wash your hands before and after each patient interaction, which includes touching their notes, or coming into a patient environment without touching anything. This is impractical, as it would slow everything down so much, and would make carrying anything impossible, not to mention the negative impact of doctor-patient interactions. It also doesn’t match how patients then move around, touching objects around hospital and nullify the whole process. Doctors and nurses tend to common sense check when a hand wash/alcohol gel is needed, which I think is a good thing. I think some go too far the other way, but there will always be variation.
    - fabricexpert 8y ago
      How can you common sense check when handwashing is required? You can't see germs, it's not possible. The handwashing protocol is there to stop people getting sick, circumventing it because you're a doctor and you know better is not going to help.
timr 8y ago
Please don't ignore this very important quote from the article: "In fact, many of the initial 50 papers have been confirmed by other groups, as some of the RP:CB’s critics have pointed out." This article is staying assiduously neutral, but one perfectly valid interpretation is that the original initiative was flawed in a way that was predicted by the initiative's critics. Science is routinely reproduced -- just not by labs working in isolation, using publications as clean-room instruction books. This is the sort of thing that programmers believe about science, not something that scientists believe themselves. There are a great many serious, legitimate scientists who believe that this "reproducibility crisis" is verging on irrational, and it's important to consider their arguments. It's particularly scary to me that so many comments here are dovetailing with the sort of nonsense you encounter on anti-vax and global warming denial forums. We're literally gaslighting the process that has done more to advance society than any other in human history: http://www.pnas.org/content/115/20/5042 http://www.pnas.org/content/115/20/5042 "The discovery that an experiment does not replicate is not a lack of success but an opportunity. Many of the current concerns about reproducibility overlook the dynamic, iterative nature of the process of discovery where discordant results are essential to producing more integrated accounts and (eventually) translation. A failure to reproduce is only the first step in scientific inquiry. In many ways, how science responds to these failures is what determines whether it succeeds."
- southern_cross 8y ago
  And the attitude being described above seems to me to border on religious "just trust us" dogma. I mean, one of the big things that's supposed to distinguish real science from pseudoscience and such is the ability to fairly easily and routinely reproduce its results. And if that's not happening, and in fact it's being actively discouraged, then that's a real problem! "In fact, many of the initial 50 papers have been confirmed by other groups, as some of the RP:CB’s critics have pointed out." I don't know that I would be willing to take a statement like that at face value. I once read of a scientist who claimed that there was absolutely no need to try and independently reproduce his work, since it was being routinely reproduced in college labs everywhere as part of other experiments - or something like that. To me that meant that it should then be trivial to reproduce these results when attempted by trained professionals who were specifically trying to do that very thing. So why the reluctance to allow that to happen?
textmode 8y ago
There is an unwritten rule in biology that if you publish a paper that refers to uses of certain reagents that are not commercially available, then you are obligated to provide those reagents to other investigators who read the paper and request them. There can also be an expected obligation that the other researchers will share any data they generate using the reagents with the original authors. Outside of biology, I have seen many "academic" papers published on computer-related topics that refer to software programs developed by the papers' authors that are crucial to the research but not publicly available. Is there any similar unwritten rule to that in biology where another researcher reading these papers can request a copy of these programs from the authors? Obviously, in many cases other researchers cannot replicate and verify findings without access to the same research tools used in the published papers.
- your-nanny 8y ago
  I've frequently asked for researchers code. most are excited someone is interested in their work, if a little nervous, or embarrassed by their coding umm style (neuroscience)
doener 8y ago
https://news.ycombinator.com/item?id=17662185 https://news.ycombinator.com/item?id=17662185
- jwilk 8y ago
  (same article, 0 comments)
sytelus 8y ago
Every academic cries about this but no one wants to accept an obvious solution: Add reprodicibility criteria for peer review. Right now even Tier 1 conferences do not have this criteria in their peer review system. The reviewer simply can't reject paper because authors didn't provided reproducibility information! This is truely a plague in research right now. I've came across quite a few instances where authors told me that their experiments weren't reproducible even for them! They do not note this in paper because ultimately everyone needs to show some result for the funding they received (i.e. "published paper"). They obviously never want to share any code, data files, hardware etc. On one instance, author wrote me back that they can't share code with me because they lost all the code because their hard drive crashed! Reproducibility is a fundamental tenant of doing scientific work and this is actively and completely ignored in current peer review system. I think conference chairs needs to take stand on this. We get now 4X to 8X papers in tier 1 conferences. Reproducibility could be a great filter when area chairs are scrambling to find reasons to reject papers. Sure, there will be papers where very specialized hardware or internal infrastructure of 10,000 computers were used. But those papers would be great for Tier 2 conferences.
- mpfundstein 8y ago
  Listen to this man folks!!! Seriously, why do you think the academic world doesn’t change? Is there too muc vested interest in the current system?
- nonbel 8y ago
  > "Add reprodicibility criteria for peer review." Passing peer review is something used as a substitute for reproducibility. Just like testing a null hypothesis is a substitute for testing the research hypothesis, and citation count is used as a substitute for making precise and accurate predictions. The procedure (scientific method) that has lead to all the great stuff we have around us today, I'd even call it a pillar of civilization, has been undergoing a piecemeal replacement since approximately WWII.
  - rantanplan 8y ago
    > Passing peer review is something used as a substitute for reproducibility Wow, pretty low bar for science. You can have peer review for the actual results and what they mean, as well as them being reproducible. > The procedure (scientific method) that has lead to all the great stuff I don't think you know what the scientific method is.
fabian2k 8y ago
One big issue with making papers reproducible is that the incentives to do that simply don't exist in many cases. It is often not rewarded if you put additional effort there, and usually also not punished even if you don't do the minimum the journals require in their official rules. The official rules are slowly changing, and the funding agencies tend to require that scientists make raw data available, and put effort into making their experiments reproducible. But the reality is changing much more slowly than the rules.
- HelloNurse 8y ago
  Peer reviewers should summarily throw out papers: "unclear procedure", "missing data", "the software is not available at the time of review"...
- danieltillett 8y ago
  Actually the incentives encourage non-reproducibility. Any effort put into reproducibility increases the chance that the results won't be publishable. The worst outcome is not publishing crap, it is not publishing at all. This needs to change.
DoctorOetker 8y ago
For the price of 10% of subsidy, you could sponsor reproducing ~10% of discoveries (proportional to the subsidy they took). Just some simple public provably random commitment protocol. While you are doing the experiment, everyone knows theres a 10% probability of eventually being selected, so healthy pressure to make sure everything is properly documented, and everyone looks out to detect fraud by collaborators. I don't see the problem, unless its the 10% price hike... if 10% is too much, just do 90% of the usual number of projects. I'd prefer 10% less projects if it ensures much higher reproducibility rates...
- Q6T46nT668w6i3m 8y ago
  I don’t understand your math. Typically, for one paper, if you’re trying to reproduce the experiment (rather than just the analysis) the cost is closer to 110% because, for many researchers, their papers build off of their previous papers so they already have expertise with a particular assay, protocol, technology, etc.
  - DoctorOetker 8y ago
    The math stays the same, even if the numbers change: with your number of 110% of the original project price for reproduction by others we get the following price hike 110% * 1/10 = 11% big deal, 11% more expensive science, but 1 out of 10 results get reproduced, stoichiocratically, so you don't know if it will be reproduced until after publication... Also: your comment about reproducibility details being scattered over the previous work of the original authors... as I said, in a world where we use my system, you are incentivized to put all details for reproduction within the paper, since you wouldn't want to risk possible reproduction by others to fail simply because they didn't read your previous papers...

253 points by nonbel 8y ago

I was mentioning this in another thread and came across the new info I hadn't heard about yet. Here are some previous articles about this: March 2012: https://www.nature.com/articles/483531a https://www.nature.com/articles/483531a June 2015: http://www.sciencemag.org/content/348/6242/1411 http://www.sciencemag.org/content/348/6242/1411 Dec 2015: https://www.nature.com/news/cancer-reproducibility-project-scales-back-ambitions-1.18938 https://www.nature.com/news/cancer-reproducibility-project-s... Jan 2017: https://www.nature.com/news/cancer-reproducibility-project-releases-first-results-1.21304 https://www.nature.com/news/cancer-reproducibility-project-r...

Palomides 8y ago

>says total costs for the 18 completed studies averaged about $60,000, including two high-priced that seems really cheap to me based on my brief experience with lab research. replicating 18 studies is a great achievement.

jonlucc 8y ago
It's not terrible for in vitro studies, but that would be a stretch for in vivo studies at a CRO, especially for oncology, where lots of studies require tumor grafts. It's not my area, but I think those mice can run $1000 and up per animal. The cheap, more common strains are ~$25 each. I haven't looked at the papers they're trying to replicate, but in my lab, it's not unusual to have 60 mice in a study, so you can see how just getting animals into your facility can be expensive.
- dingo_bat 8y ago
  $1000 per mouse? Seems really expensive.
  - amelius 8y ago
    Annual cost of small mammal ownership: $300 Source: https://xkcd.com/980/huge/#x=-1166&y=-2798&z=5 https://xkcd.com/980/huge/#x=-1166&y=-2798&z=5 Add to that expenses for the experiment, and $1000 seems not far fetched.
    - mattkrause 8y ago
      That’s for a pet. While labs have huge economies of scale—-I worked somewhere with a quarter million mice—-the mice aren’t just random field mice. Some of the genetically engineered ones are hundreds of dollars a pair (and possibly more if custom, raised under unusual conditions, etc). Xenografts need skilled labor and very clean conditions for the immunocompromised mice. $1000 seems a little high, but not much, especially if the work isn’t being done by an underpaid grad student.
  - jonlucc 8y ago
    It is; that's my point. As mattkrause notes below, that number is for mice with human tumor xenografts. You have to harvest a human patient's tumor, blend it up, and inject it into immunocompromised mice. If you buy them from the vendor who has to buy their tumors, pay their labor, and make a profit, they're not cheap. If you have ready access to clinical tumors and cheap labor, the price drops precipitously to ~$70. By the way, you can poke around some of the animal vendor's sites. They mostly have their pricing on their websites along with a bunch of other information. Four really big ones are Taconic, Jackson Labs, Envigo, and Charles River.
deleted 8y ago
[deleted]

Alex3917 8y ago

> organizers realized they needed more information and materials from the original authors; Then the study already isn't replicable by definition, so why waste time asking the original authors? Just mark it down as 'not replicable' and move on.

jonlucc 8y ago
I'm not so sure this is the right way to go about it. They give the example of cell densities, but there are also tons of things that just are never reported in in vivo papers, including the light cycle in animal rooms (mice and rats are nocturnal, so some labs make their rooms dark during the day), the preparation of treatment compounds, information about animal ages, handling procedures, etc. In addition, many papers have length limits, so you can only include so much.
- Alex3917 8y ago
  Sure, all of which make the research non replicable. That stuff is all important, you can't just discount it because it's the norm not to include it.
  - TallGuyShort 8y ago
    It's important that it become the norm to include it, though. So rather than just moving on, they're trying to gather that information so that if they CAN reproduce it, they now know how to make sure the next experiment can be more reproducible.
  - flukus 8y ago
    They should be replicable despite a lot of that, if they aren't then then the original study is garbage.
    - Alex3917 8y ago
      Getting the same result doesn’t mean a study is replicable or a result is reproduceable. If you don’t use the same methodology it’s meaningless.

kayhi 8y ago

I wrote my concerns about replicating these studies a couple of years ago [1]. In short, most papers don't have enough details to allow for a third party to replicate them. Problem areas include detailed protocols and reagents used. If you don't what someone did exactly and what they used then replicating it is going to be very difficult. [1] https://www.linkedin.com/pulse/scientific-reproducibility-reagent-variability-sean-seaver/ https://www.linkedin.com/pulse/scientific-reproducibility-re...

nonbel 8y ago
That is exactly the problem. All this stuff is supposed to be in the paper, the paper is not a press release. And everyone understands that its hard to know exactly what info someone else will be missing, so it wont be perfect at first. That is why there is supposed to be a back and forth the first time a new method is used. Figuring out what needs to go into the methods is going to be an iterative process. As it is right now its a disaster though. Like they say in this article no one even knows the most basic info like cell density. The entire system is not set up to deal with replications at all because they haven't been doing them.
- rjkennedy98 8y ago
  > All this stuff is supposed to be in the paper, the paper is not a press release. Welcome to the real world. Science doesn't work like it was described in your 7th grade science class.
  - pas 8y ago
    But it can be approximated. Repeatable experiments. Done by machines, properly accounted ingredients and so on. Physics does it. It should be perfectly doable with cell cultures too. Mice are of course a lot harder. But if we wouldn't wait decades to replicate, but could simply get in touch with the originators - and so do a collaborative effort in establishing the parameters.
  - nonbel 8y ago
    These sayings like "welcome to the real world", "there's a difference between theory and practice", thought-killers are just excuses for being lazy. Science is hard, really hard. People who think doing a good job is too hard should get out of science instead of polluting the literature with their statistically significant results or whatever. Unfortunately its been the opposite happening and the people who cant handle it have been training the new people how to get away with half-assing it. This has been going on for a long time now, at least since the 1940s in some areas...

Avshalom 8y ago

turns out we can't raise the money to replicate researchers who barely had the money to do what they did.

nonbel 8y ago
I'd honestly say if the study isn't worth replicating then don't fund it to begin with. Its like funding it enough to get some animals or cells into your lab but then not enough for gloves and pipets and computers.
- Avshalom 8y ago
  That sounds great in abstract but every study is subject to that logic, including studies studying studies. Which means that every study should be matched by an infinite regression of studies, which is untenable to put it mildly and mathematically fucking ridiculous to put it conservatively.
  - nonbel 8y ago
    There is no infinite regression, what are you talking about? Its that at least two groups try to measure the same thing under the same conditions (as similar as you can get them).
    - Avshalom 8y ago
      how do you know that those two groups aren't collaborating for the same result? Or biased in the same direction of the result? you don't. You either add a 3rd party (4th... 5th...6th... Xth)or accept the conclusion. If you add a third party how do you know they are trust worthy: infinite regression.
      - nonbel 8y ago
        Its not about perfection, its about taking the simple, common sense, and time-tested precautions that gave us the technological marvels we enjoy today. But of course the more independent, even adversarial, the two groups are the better. Like in this story it mentions the "critics" of this project claim other labs (their friends?) have already replicated these studies, apparently using the super secret protocol they couldn't explain to this group. I'd compare it to using sms 2FA. Is it perfect, no. But it is far, far better than no 2FA at all. And there's nothing stopping you from putting your banking app on the same phone you use for 2FA.

Gatsky 8y ago

Replicating cancer papers: This is one of those ideas which although pointless and impractical, is seemingly impossible to criticize. Now the leaders of this project are basically saying - we can't do difficult experiments, and can't replicate studies which have already been replicated by others. What have we learned exactly, except that experimental biology is difficult?

drb91 8y ago
What's the point of cancer papers at all if they're un-reproducable? Are we just sinking government money into artificially inflated bio portfolios?
pietroglyph 8y ago
> pointless and impractical These papers underlie a lot of modern research and treatments. If they aren't reproducible, the papers' results, and other papers that use their results come into question. This is a serious, documented problem [0]. > What have we learned exactly, except that experimental biology is difficult? The problem wasn't that the project couldn't do difficult experiments, it was that not enough information was provided by the original papers. That's what we learned here. [0]: https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- Gatsky 8y ago
  I disagree with your characterisation of the problem and the proposed solution. Cancer research is not an edifice constructed on foundational results. It is not like physics in this regard, where for example, accurate calculation of the gravitational constant is vitally important, and over time repeat measurements are generally more precise but not wildly different, or do not dispute the importance of the gravitational constant. Take one of the papers they reproduced - Transcriptional amplification in tumor cells with elevated c-Myc. This is an interesting result conducted in highly contrived experimental conditions. Nobody is going to base their career, a drug development program, treat a patient or even start a PhD based on this result alone. I say this is a translational cancer scientist and medical oncologist. The contribution of this paper is to our knowledge of the biology of Myc, which has a multitude of actions which are context dependent. Myc is studied in a variety of different ways using many different methods. If this paper were being repeated today, the technologies and techniques would be quite different. The result of this paper is not plugged into the central dogma of cancer biology, setting us down an erroneous path for the next thousand years. So to return to my original question, what have we learned in trying to replicate this study, that we didn't already know? The money would have been better spent on orthogonal validation/extension of the result using modern techniques - another name for this is 'science'. The replication crisis suggests that this routine extension/validation process is somehow less important than going back and repeating the original experiment, which is I think a complete misunderstanding. You also seem to be saying that we should ask researchers to document in excruciating detail all experimental conditions such that a pastry chef or meteorologist could walk into a lab and successfully reproduce the experiment - this is an impossibly high bar to set for scientists who are already working under very difficult conditions, and is not the solution.

StanislavPetrov 8y ago

They should have just become sociologists so they could have dispensed with any reality-based interference at all.

titanix2 8y ago
I don’t get the downvotes, there is some truth here. I even skimmed a doctoral thesis supposed to be in the field of linguistics where the author declared her work was also made as an activist. Too many papers made by too many "academics" are straight made up ideological essays with no grounding in reality nor in science.
- deleted 8y ago
  [deleted]
- StanislavPetrov 8y ago
  Those ideological "academics" are the ones downvoting.
  - mattkrause 8y ago
    Nope—-one came from me, and I’m a systems neuro/CS staff scientist. Social science is incredibly important. Psychology departments have contributed so much to our understanding of perception and attention, which in turn has paved the way for safe and effective user interfaces, better compression algorithms, and prosthetics. Sociology can tell you how to optimize anything involving groups of people, whether it’s effectively provisioning government services or running an aircraft crew. Linguistics has obvious applications to speech and text processing, education, and even medicine. All this and more, plus you get to learn about what it means to be human! And it’s cheap to boot! Saving one airplane funds a hell of a lot of cockpit management studies. Sure, there is some crappy social science, but there are also crappy programmmers. The blatant dismissal of whole fields is the worst kind of pseudointellectual snobbery. By all means, call out work that’s subpar, but if you can’t see how social science is both interesting and useful, it’s only because you’re not trying.
8y ago

rrock 8y ago

Wait, they thought they could replicate 50 papers at the cost of $25k each, and all in the span of one year? Not a chance. I don’t think these guys have any idea what they’re doing. I feel sorry for the funders.

lsh 8y ago
It's good to have reasonable goals and a budget, even if some of them aren't met. The funders absolutely know what they are doing and handicapping a replication effort because it went a few thousands over budget would be outrageous. There are still unknown quantities at play here, a large scale replication effort hasn't been attempted before and those efforts themselves - the extent of the effort required to replicate and the impact of failure - are also goals. Failure results, of which there have been several already now, can severely damage the reputation of prestigious journals and teams of scientists. They're not doing this because replication is popular, or cheap or because they thought it would be a fun afternoon lark.
LyndsySimon 8y ago
They thought that because they’d already done 100 in another field: https://osf.io/ezcuj/ https://osf.io/ezcuj/ I worked there for three years :)
- lsh 8y ago
  thank you for the link
- rrock 8y ago
  I'm going to guess that psychology research is far less expensive. To give you a sense of perspective here, $25k will get you something like 4 months of a full time employee at the "fresh out of college" experience level, including benefits and research consumables. That also assumes that you have all of the equipment in place. If all there doing is running a few western blots, because that's all they can do on this budget, well then ok. But I'll venture to guess that that's a far cry from reproducing the most important results in these papers.

buchanae 8y ago

I have been working on reproducing computational biology papers from the cancer field lately. I am very frustrated. When the inputs and outputs are machine-readable data, there's no excuse for not making your work reproducible, in my opinion. Often the problem is plain laziness and disorganization. One major problem is that there's not much real incentive to make your work reproducible. Money granting organizations favor researchers breaking new and exciting ground, not those rehashing an already published method. Publishers don't require reproducible methods, and reviewers don't have the time, desire, nor expertise to do an in-depth methods review. Wet lab experiments are 1-2 orders of magnitude more expensive and difficult to reproduce, that's true, but we're not even getting the basics right!

buchanae 8y ago
And another thing! Why should a project be done when it gets published?! What kind of software project would make one release and call it good forever? Nonsense.
- fmap 8y ago
  In an ideal world you would be right, but that's not how the funding structures for academic research are set up. Think of a research lab as a company that gets paid per prototype and then has to market the concept for the next prototype in an infinite loop. If you can't package up what you're doing into a sequence of small prototypes then you're not getting paid.
  - ajuc 8y ago
    The requirement for publishing in a scientific journalshould be opensourcing all the code used. You can't expect the results to be reproductible 20 years later otherways.
    - skummetmaelk 8y ago
      20 year old code that has not been maintained most likely would not run on a modern system anyway, unless it is extremely simple.

gnusci 8y ago

This is a well known problem of scientific publications now-a-days: https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 https://www.nature.com/news/1-500-scientists-lift-the-lid-on... https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis Papers have become a target for success, so scientist need publications for better status and remuneration: http://www.sciencemag.org/news/2017/08/cash-bonuses-peer-reviewed-papers-go-global http://www.sciencemag.org/news/2017/08/cash-bonuses-peer-rev... A new scientific "mafia" is in place around the world: https://www.technologyreview.com/s/608266/the-truth-about-chinas-cash-for-publication-policy/ https://www.technologyreview.com/s/608266/the-truth-about-ch... https://retractionwatch.com/2017/08/10/paid-publish-not-just-china/ https://retractionwatch.com/2017/08/10/paid-publish-not-just... So papers have become a good business, no the way to disseminate outstanding research results.

lsh 8y ago
> So papers have become a good business, no the way to disseminate outstanding research results. That's awfully cynical and over-broad, but I agree to a point. Greedy and unscrupulous publishers are part of the problem, but so are lax or unprincipled scientists eager for prestige and a career-making publication in a top tier journal. It's an unfortunate chicken-and-egg cycle now with no easy way to cut it. Perhaps more emphasis on replication post-publication? Perhaps a reputation system for unethical publishers or scientists?
- timr 8y ago
  "Greedy and unscrupulous publishers are part of the problem, but so are lax or unprincipled scientists eager for prestige and a career-making publication in a top tier journal." That's just incredibly unfair. There are some fields and methodologies where p-hacking and cherry-picking have been a problem, but the primary reason that papers aren't reproducible is just noise and basic statistics. As a scientist, you control for what you can think of, but there are often way too many variables to control completely, and it's probable that you miss some. Those variables come to light when someone else tries to work with your method and can't reproduce it locally. However, real scientists don't stop and accuse the original authors of being "unprincipled" -- nine-point-nine times out of ten, they work with the original authors to discover the discrepancies. It isn't surprising at all to actual, working scientists that most papers are impossible to reproduce from a clean room, using only the original paper. It's the expected state of affairs when you're working with imperfect, noisy techniques, and trying tease out subtle phenomena.

atdt 8y ago

Why, apart from expedience, do designers of experiments share an institutional affiliation with the experimenters themselves? I wonder what would happen if the two groups were far apart, and if communication between them restricted to the transmission of an experimental protocol and the reception of raw observations.

nonbel 8y ago
This is a really good idea. I bet it is not as hard as some of the "real scientists" claim.

lsh 8y ago

A direct link to the results: https://elifesciences.org/collections/9b1e83d1/reproducibility-project-cancer-biology https://elifesciences.org/collections/9b1e83d1/reproducibili... There have been several news items on Hacker News recently about academic publishing, reproducibility in general and pre-print servers and I would love it if eLife got some more attention, for better or worse, for the excellent work they are doing and helping to fund. To those biology researchers in the comments, please submit your work! even failed results! Even if it's software heavy. Editors are listening closely to your feedback. edit: disclosure, I work for eLife

chmaynard 8y ago
Does eLife have an RSS feed? I can't find one listed on your website.
- rossmounce 8y ago
  Yes, various RSS feeds. Alert options explained here: https://elifesciences.org/alerts https://elifesciences.org/alerts early versions: https://elifesciences.org/rss/ahead.xml https://elifesciences.org/rss/ahead.xml final published versions: https://elifesciences.org/rss/recent.xml https://elifesciences.org/rss/recent.xml
Q6T46nT668w6i3m 8y ago
eLife is excellent.

anticensor 8y ago

It turned out remaining 32 are unworthy to replicate.

adiusmus 8y ago

Initiate Rant. Most science isn’t. Most of it is a set of nearly fanatical beliefs about reality only vaguely related to facts agreed through consensus by authority. Look at how strongly doctors fought against germ theory. They saw no reason to wash their hands for delivering a new born after having spent their morning cutting open cadavers. Their response was anything but scientific even in the face of easily tested claims and results. If research can’t be reproduced it’s just a story with interesting data. I doubt most scientists using statistics would be able to provide the alpha / p value / confidence interval etc for their hypothesis. When the bar is set so low should we be surprised by low quality? Rant complete. Initiate beer.

southern_cross 8y ago
Hand washing may still be a problem. I saw a blurb recently which stated that when doctors know that they're being monitored for using proper hand washing protocols, compliance was near the top end of the scale. But when they thought they were no longer being watched, compliance fell by something like two-thirds. So in other words, doctors (and no doubt others) probably still aren't anywhere close to doing it right.
- Engineering-MD 8y ago
  That would be because “doing it right” is completely over the top. You are suppose to wash your hands before and after each patient interaction, which includes touching their notes, or coming into a patient environment without touching anything. This is impractical, as it would slow everything down so much, and would make carrying anything impossible, not to mention the negative impact of doctor-patient interactions. It also doesn’t match how patients then move around, touching objects around hospital and nullify the whole process. Doctors and nurses tend to common sense check when a hand wash/alcohol gel is needed, which I think is a good thing. I think some go too far the other way, but there will always be variation.
  - fabricexpert 8y ago
    How can you common sense check when handwashing is required? You can't see germs, it's not possible. The handwashing protocol is there to stop people getting sick, circumventing it because you're a doctor and you know better is not going to help.

timr 8y ago

Please don't ignore this very important quote from the article: "In fact, many of the initial 50 papers have been confirmed by other groups, as some of the RP:CB’s critics have pointed out." This article is staying assiduously neutral, but one perfectly valid interpretation is that the original initiative was flawed in a way that was predicted by the initiative's critics. Science is routinely reproduced -- just not by labs working in isolation, using publications as clean-room instruction books. This is the sort of thing that programmers believe about science, not something that scientists believe themselves. There are a great many serious, legitimate scientists who believe that this "reproducibility crisis" is verging on irrational, and it's important to consider their arguments. It's particularly scary to me that so many comments here are dovetailing with the sort of nonsense you encounter on anti-vax and global warming denial forums. We're literally gaslighting the process that has done more to advance society than any other in human history: http://www.pnas.org/content/115/20/5042 http://www.pnas.org/content/115/20/5042 "The discovery that an experiment does not replicate is not a lack of success but an opportunity. Many of the current concerns about reproducibility overlook the dynamic, iterative nature of the process of discovery where discordant results are essential to producing more integrated accounts and (eventually) translation. A failure to reproduce is only the first step in scientific inquiry. In many ways, how science responds to these failures is what determines whether it succeeds."

southern_cross 8y ago
And the attitude being described above seems to me to border on religious "just trust us" dogma. I mean, one of the big things that's supposed to distinguish real science from pseudoscience and such is the ability to fairly easily and routinely reproduce its results. And if that's not happening, and in fact it's being actively discouraged, then that's a real problem! "In fact, many of the initial 50 papers have been confirmed by other groups, as some of the RP:CB’s critics have pointed out." I don't know that I would be willing to take a statement like that at face value. I once read of a scientist who claimed that there was absolutely no need to try and independently reproduce his work, since it was being routinely reproduced in college labs everywhere as part of other experiments - or something like that. To me that meant that it should then be trivial to reproduce these results when attempted by trained professionals who were specifically trying to do that very thing. So why the reluctance to allow that to happen?

textmode 8y ago

There is an unwritten rule in biology that if you publish a paper that refers to uses of certain reagents that are not commercially available, then you are obligated to provide those reagents to other investigators who read the paper and request them. There can also be an expected obligation that the other researchers will share any data they generate using the reagents with the original authors. Outside of biology, I have seen many "academic" papers published on computer-related topics that refer to software programs developed by the papers' authors that are crucial to the research but not publicly available. Is there any similar unwritten rule to that in biology where another researcher reading these papers can request a copy of these programs from the authors? Obviously, in many cases other researchers cannot replicate and verify findings without access to the same research tools used in the published papers.

your-nanny 8y ago
I've frequently asked for researchers code. most are excited someone is interested in their work, if a little nervous, or embarrassed by their coding umm style (neuroscience)

doener 8y ago

https://news.ycombinator.com/item?id=17662185 https://news.ycombinator.com/item?id=17662185

jwilk 8y ago
(same article, 0 comments)

sytelus 8y ago

Every academic cries about this but no one wants to accept an obvious solution: Add reprodicibility criteria for peer review. Right now even Tier 1 conferences do not have this criteria in their peer review system. The reviewer simply can't reject paper because authors didn't provided reproducibility information! This is truely a plague in research right now. I've came across quite a few instances where authors told me that their experiments weren't reproducible even for them! They do not note this in paper because ultimately everyone needs to show some result for the funding they received (i.e. "published paper"). They obviously never want to share any code, data files, hardware etc. On one instance, author wrote me back that they can't share code with me because they lost all the code because their hard drive crashed! Reproducibility is a fundamental tenant of doing scientific work and this is actively and completely ignored in current peer review system. I think conference chairs needs to take stand on this. We get now 4X to 8X papers in tier 1 conferences. Reproducibility could be a great filter when area chairs are scrambling to find reasons to reject papers. Sure, there will be papers where very specialized hardware or internal infrastructure of 10,000 computers were used. But those papers would be great for Tier 2 conferences.

mpfundstein 8y ago
Listen to this man folks!!! Seriously, why do you think the academic world doesn’t change? Is there too muc vested interest in the current system?
nonbel 8y ago
> "Add reprodicibility criteria for peer review." Passing peer review is something used as a substitute for reproducibility. Just like testing a null hypothesis is a substitute for testing the research hypothesis, and citation count is used as a substitute for making precise and accurate predictions. The procedure (scientific method) that has lead to all the great stuff we have around us today, I'd even call it a pillar of civilization, has been undergoing a piecemeal replacement since approximately WWII.
- rantanplan 8y ago
  > Passing peer review is something used as a substitute for reproducibility Wow, pretty low bar for science. You can have peer review for the actual results and what they mean, as well as them being reproducible. > The procedure (scientific method) that has lead to all the great stuff I don't think you know what the scientific method is.

fabian2k 8y ago

One big issue with making papers reproducible is that the incentives to do that simply don't exist in many cases. It is often not rewarded if you put additional effort there, and usually also not punished even if you don't do the minimum the journals require in their official rules. The official rules are slowly changing, and the funding agencies tend to require that scientists make raw data available, and put effort into making their experiments reproducible. But the reality is changing much more slowly than the rules.

HelloNurse 8y ago
Peer reviewers should summarily throw out papers: "unclear procedure", "missing data", "the software is not available at the time of review"...
danieltillett 8y ago
Actually the incentives encourage non-reproducibility. Any effort put into reproducibility increases the chance that the results won't be publishable. The worst outcome is not publishing crap, it is not publishing at all. This needs to change.

DoctorOetker 8y ago

For the price of 10% of subsidy, you could sponsor reproducing ~10% of discoveries (proportional to the subsidy they took). Just some simple public provably random commitment protocol. While you are doing the experiment, everyone knows theres a 10% probability of eventually being selected, so healthy pressure to make sure everything is properly documented, and everyone looks out to detect fraud by collaborators. I don't see the problem, unless its the 10% price hike... if 10% is too much, just do 90% of the usual number of projects. I'd prefer 10% less projects if it ensures much higher reproducibility rates...

Q6T46nT668w6i3m 8y ago
I don’t understand your math. Typically, for one paper, if you’re trying to reproduce the experiment (rather than just the analysis) the cost is closer to 110% because, for many researchers, their papers build off of their previous papers so they already have expertise with a particular assay, protocol, technology, etc.
- DoctorOetker 8y ago
  The math stays the same, even if the numbers change: with your number of 110% of the original project price for reproduction by others we get the following price hike 110% * 1/10 = 11% big deal, 11% more expensive science, but 1 out of 10 results get reproduced, stoichiocratically, so you don't know if it will be reproduced until after publication... Also: your comment about reproducibility details being scattered over the previous work of the original authors... as I said, in a world where we use my system, you are incentivized to put all details for reproduction within the paper, since you wouldn't want to risk possible reproduction by others to fail simply because they didn't read your previous papers...