6 ms·

Science Exchange Reproducibility Initiative

113 points by abbottry 14y ago

abbottry 14y ago
Some supporting information: - Nature: http://www.nature.com/news/independent-labs-to-verify-high-profile-papers-1.11176 http://www.nature.com/news/independent-labs-to-verify-high-p... - Slate: http://www.slate.com/articles/health_and_science/science/2012/08/reproducing_scientific_studies_a_good_housekeeping_seal_of_approval_.html?fb_action_ids=10102435251603084&fb_action_types=og.likes&fb_source=aggregation&fb_aggregation_id=288381481237582 http://www.slate.com/articles/health_and_science/science/201... - Reuters: http://www.reuters.com/article/2012/08/14/idUS117855+14-Aug-2012+BW20120814 http://www.reuters.com/article/2012/08/14/idUS117855+14-Aug-...
soperj 14y ago
Proofs are for math, not science.
- lobo_tuerto 14y ago
  Maybe the title should say "probe it". Just saying.
- diego 14y ago
  Agree, the title is poorly editorialized. Proofs and validations are very different things. Validation is about being confident that a model can make useful predictions.
aakil 14y ago
Plasmyd does this via crowdsourcing, it lets users comment on papers so that researchers can point out anomalies or post their inability to reproduce the same results. It also gives the author a chance to explain their work.
- bmahmood 14y ago
  Bilal from Science Exchange here. Definitely agree that Plasmyd is a valuable service. We feel this Initiative operates in complement, providing a mechanism for independent validation.
- pcrh 14y ago
  It is frustratingly difficult to get biomedical scientists to openly discuss the work in their field. PLoS tried to have discussion sections to papers, and Nature has tried various approaches as well. Neither have worked. I looked-up my favorite topic (a popular one) on Plasmyd, and found 20 papers, none of which had a single comment associated with it. It's honestly a vast waste of intellectual potential, I wish I knew the solution!
fbomb 14y ago
"Validations are conducted blind, on a fee-for-service basis." Seems like a potential conflict of interest there.
- archgoon 14y ago
  How? If they don't know what the answer is supposed to be, it's not clear how they can fake data. This can further be improved by universities contributing to a 'noise' fund, where requested experiments are expected to _not_ turn anything up. Also, the main purpose of this is not to catch corruption, just prevent mostly honest researchers from fooling themselves (and then others), and increasing the base line for research.
- bmahmood 14y ago
  Bilal from Science Exchange here. The validations are blind in the sense that we match an investigator who has submitted a study to a provider who can validate it. The investigator doesn't have a choice of who validates their study though. The fee-for-service actually helps to guard against biased results in two ways. Firstly, providers who validate studies are paid on completion, regardless of a reproducible or irreproducible outcome. Secondly, operating on a fee-for-service basis allows providers to operate outside academic incentives for publications, ensuring no incentive for biased-positive results.
- pcrh 14y ago
  I see it as something like the National Institute of Standards and Technology (http://www.nist.gov/index.html http://www.nist.gov/index.html). I don't think most academic labs will pay for this. However, I can see it as being something that VC investors might like to see the next time someone presents with a wonder drug.
Xcelerate 14y ago
This is great! Too often scientific papers are published that are never actually checked by anyone else. A key part of the scientific method is reproducibility though.
graeme 14y ago
I was very surprised when I first learned that results could be published without first having been reproduced. I thought at the least they should have been published with a separate label. This is a worthy initiative.
- tokenadult 14y ago
  I was very surprised when I first learned that results could be published without first having been reproduced. A lot of psychology journals have been burned by that over the years, http://chronicle.com/blogs/percolator/is-psychology-about-to-come-undone/29045 http://chronicle.com/blogs/percolator/is-psychology-about-to... http://simplystatistics.org/post/21326470429/replication-psychology-and-big-science http://simplystatistics.org/post/21326470429/replication-psy... http://scholarlykitchen.sspnet.org/2012/04/26/reproducibility-an-attempt-to-test-the-psychology-literature-underscores-a-growing-fault-line/ http://scholarlykitchen.sspnet.org/2012/04/26/reproducibilit... and as a consequence, just within the last year or so, the most prestigious psychology journals are beginning to ask authors of papers on new experimental research to show replication across at least two distinct data sets. This, in turn, is driving more collaboration among researchers at different research centers. It's a long overdue development, one in accord with best practice in science, http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html and I hope this practice spreads to most disciplines that publish new research reports in the major journals of the discipline. (Although the requirement for prepublication replication is not explicitly mentioned in the publication guidelines of Psychological Bulletin, http://www.apa.org/pubs/journals/bul/index.aspx http://www.apa.org/pubs/journals/bul/index.aspx which is the most prestigious psychology journal published in the United States, I have been told at the University of Minnesota behavior genetics "journal club" that replication across more than one data set is becoming a tacit requirement for publication in most of the better journals. Perhaps that guideline will eventually be made explicit for all of the better journals.)
- _delirium 14y ago
  Traditionally, the public literature is how scientists communicated, which is what made republication possible in the first place. Lab A, perhaps in Germany, works on a research project, and gets some results. They write up their methodology and results, and communicate them to other scientists by publishing the write-up in a journal. The journal will have the write-up peer-reviewed before publishing it, but that doesn't mean verifying it's correct, only that it's a contribution to the scientific literature worth communicating to its readership. For example, it would make sure the paper has done appropriate data analysis, has cited relevant existing work, is not outright kookery, etc. Now other scientists, anywhere in the world, will read this and decide what they think of it, which may include attempting to replicate it. If they find different results, or have a qualm with an interpretation or methodology, they can write a response, which will also (after peer-review) be communicated by the journal to its readership, either as another paper, or if shorter, as a letter. Thus the back-and-forth of claim/replication/critique happens, all in the open air of the journal's pages, with any lab anywhere in the world able to follow the conversation and jump in if they wish. To me that's more transparent than doing all this in some kind of closed pre-publication process. Perhaps in the modern era, the open communication doesn't necessarily have to happen in journals anymore: perhaps it should happen on arXiv, and journals should become more conservative in what they publish. But I would like it to still be public.
JohnnyBBad 14y ago
Lemme know when somebody reproduces monkeys evolving into humans.
podperson 14y ago
The title irks me. Instead of "prove it", how about "let us reproduce your results" or "show us the data". Proof in this context is a mistaken concept.
semenko 14y ago
There are very few journals that do this -- I can't imagine this ever taking off. Reproducing experiments seems like a costly (and mostly thankless) effort that few PIs would ever take up. The only journal I know of that completely reproduces results is Organic Syntheses (http://www.orgsyn.org/ http://www.orgsyn.org/), which reproduces every reaction before publication and has a Procedure Checklist for authors: http://www.orgsyn.org/AuthorChecklist.pdf http://www.orgsyn.org/AuthorChecklist.pdf
- bmahmood 14y ago
  Hi, this is Bilal from Science Exchange. Appreciate your concern, but that is why we feel Science Exchange is uniquely positioned to make a difference. We have developed a network of 1000 core facilities and CROs on our platform, who operate on a fee-for-service basis to conduct experimental services. The Reproducibility Initiative will leverage this network to outsource specific experiments within a study to these facilities.
- pcrh 14y ago
  It is quite common for experiments to be replicated, usually as preliminary to a further advance. If the findings claimed are significant enough, other labs will often attempt to replicate a few of the key experiments, and if they don't do so quickly, abandon that direction of investigation, and not report their negative findings.
abbottry 14y ago
Title changed -- geeeeeze #cowersinthecorner
kevinalexbrown 14y ago
This is excellent. I have a few questions: You need to provide the background of your study, the types of experiments undertaken, the materials and methods, and initial results of your study. Do the technicians reproducing the results get to see the initial results? It seems like it might be more accurate if they didn't. Lots of parameters can be fudged and adjusted, a la Millikan's Oil Drop, when the results don't quite match. I imagine this might be exacerbated with the necessity of researcher-validator communication. How are conflicts resolved? If my results are not validated, someone made a mistake - me or the validator. If both parties stand by their mutually incompatible results, where does it go from there? I can imagine a lot of researchers I know feeling annoyed that someone whose expertise they cannot verify (due to anonymity) won't "do my experiment correctly". I imagine that in time there might be specific requirements or explicit funding allocations for such reproduction on grant applications, which would really allow it to take off. As it stands, I imagine a lot of PI's would just ask "hmm, I can spend money that might risk my already high-impact paper, or I can keep the money and not be considered wrong." Still, this is a great first step toward facilitating a central tenet of the scientific method. Congratulations.
- bmahmood 14y ago
  This is Bilal from Science Exchange, and we greatly appreciate your support for our initiative! We will provide the methodology of the original study to those reproducing the results, while the results we believe will be helpful to check against. For conflicts, it's true we can't force an investigator to publish or note the lack of reproduced outcomes. We do hope they will through the PLOS Collection, for transparency. We do feel though it can provide a valuable check for 'failing fast', for those investigators who want robust results. In this initial stage, we also agree that funding will be difficult. That's why we hope to focus on small biotechs and research labs that are interested in commercializing their research, and need to show robustness of results for licensing opportunities. Hopefully then, it can serve as a proof-of-principle for funding agencies to provide a requirement or increase support for reproducibility.
  - pcrh 14y ago
    I think this will be a very valuable service for small biotechs. It would improve the value of their patents and increase chances of getting further funding. As an experienced scientist myself, I can say there is plenty of scope for misunderstanding and misinterpretation, no matter how careful the original authors were. So I only hope that there will be some (maybe blinded) mechanism for communication between the original researchers and those replicating the published findings, in the event of the "usual" complications. Also, who would do this replication? Would some of it be outsourced to academic labs with the requisite experience? Advanced findings often depend on advanced techniques. Outsourcing could be problematic, politically, since Big Shot No.1 might not be too interested in shooting down his pal, Alpha-MD-PhD.
frisco 14y ago
It's very expensive (prohibitively so) to reproduce any substantial study. It also feels like there's a lot of potential downside and very little reward, since there's a presumption of correctness in published papers today. Further, wet lab protocols are subjective enough that I'd imagine most labs would ignore a negative result as having been performed incorrectly. Where does the money for this come from? Are you expecting that labs will write this cost into their grants? Have you seen interest from grant agencies to then actually pay for this?
- deleted 14y ago
  [deleted]
- bmahmood 14y ago
  Hi, this is Bilal from Science Exchange. We are in discussions with funding agencies and foundations to possibly gather increased financial support. We feel this initial launch will provide a proof-of-principle to gather support. Currently though, we hope investigators will pay for the validation themselves. We believe it can provide a valuable service to those in small biotechs or those who want to commercialize their research, to improve the robustness of outcomes for potential licensing opportunities.
- gwern 14y ago
  > Further, wet lab protocols are subjective enough that I'd imagine most labs would ignore a negative result as having been performed incorrectly. That kind of illustrates the usefulness of replication! If the effect is so finicky that it cannot even be replicated in another lab, in what sense is that result interesting, useful, - or even real?
  - frisco 14y ago
    I think you overestimate most biology labs. It's really easy to screw up a protocol in subtle ways. Often it took the original lab weeks of debugging to get it to work (yes, to get it to work, not to produce a false positive). Things get more reliable as time goes on and methods become better understood, but the critical steps in most experiments are usually nontrivial to replicate in the beginning. Especially by CROs, which are notorious black boxes, incredibly expensive, and often just plain old unreliable.
joe_the_user 14y ago
I don't see this approach really able to solve the underlying problems that it references from http://www.nytimes.com/2012/04/17/science/rise-in-scientific-journal-retractions-prompts-calls-for-reform.html?_r=1 http://www.nytimes.com/2012/04/17/science/rise-in-scientific... and other articles. "Each year, every laboratory produces a new crop of Ph.D.’s, who must compete for a small number of jobs, and the competition is getting fiercer. In 1973, more than half of biologists had a tenure-track job within six years of getting a Ph.D. By 2006 the figure was down to 15 percent." I would claim that science requires some basic integrity in its practitioners and if an institution is treating it's members as so many throw-away resources, it is hard to expect those members to move ahead with great idealism. The model of every desperate competitor watching every other competitor seems to be the replacement for model of science as a high ideal. I don't see it working out well.
- bmahmood 14y ago
  It's true the issue of reproducibility in research is a complex one, with many underlying factors. Culture, misaligned incentives, and lack of resources play strong roles. We believe however our Initiative can help in laying an initial framework for how one can possibly address aspects of the problem. We've tried to build incentives (ease of outsourcing, rewarding publications) that factor into this issue, and can assist in improving outcomes. But we definitely agree that a holistic solution will require further changes to the academic research infrastructure.
jayzee 14y ago
The part I like about the execution is where you have CROs and Core Facilities, and not scientists themselves validate results. Apparently Milikan when he measured the electric charge was off (smaller than actual) in the measurements... but researchers who checked, worried perhaps about their academic reputation (speculating here) slowly adjusted this number upwards over several years till it asymptotically approached the truth. Having a 3rd-party, that is less impacted by academic politics might be a good thing. From Feynman's Caltech commencement speech on this: We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off, because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of the electron, after Millikan. If you plot them as a function of time, you find that one is a little bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher. Why didn't they discover that the new number was higher right away? It's a thing that scientists are ashamed of--this history--because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong--and they would look for and find a reason why something might be wrong. When they got a number closer to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that. We've learned those tricks nowadays, and now we don't have that kind of a disease. [1] http://www.lhup.edu/~dsimanek/cargocul.htm http://www.lhup.edu/~dsimanek/cargocul.htm
larsberg 14y ago
The NSF CISE guidelines (which apply to much of the funded CS research that folks on HN care about) currently require you to retain all data required to reproduce your experiment and make them available to any reasonable request: http://www.nsf.gov/cise/cise_dmp.jsp http://www.nsf.gov/cise/cise_dmp.jsp My advisor is on appointment there right now, and they're working on guidelines that require you to provide full reproducibility of your program results (modulo system differences). At least for software, things are looking up. Of course, reproducibility is both infinitely easier (I can do it on any system!) and harder (what do you mean the kernel patchlevel or CPU/GPU sub-model-number matters?).

113 points by abbottry 14y ago

Some supporting information: - Nature: http://www.nature.com/news/independent-labs-to-verify-high-profile-papers-1.11176 http://www.nature.com/news/independent-labs-to-verify-high-p... - Slate: http://www.slate.com/articles/health_and_science/science/2012/08/reproducing_scientific_studies_a_good_housekeeping_seal_of_approval_.html?fb_action_ids=10102435251603084&fb_action_types=og.likes&fb_source=aggregation&fb_aggregation_id=288381481237582 http://www.slate.com/articles/health_and_science/science/201... - Reuters: http://www.reuters.com/article/2012/08/14/idUS117855+14-Aug-2012+BW20120814 http://www.reuters.com/article/2012/08/14/idUS117855+14-Aug-...

soperj 14y ago

Proofs are for math, not science.

lobo_tuerto 14y ago
Maybe the title should say "probe it". Just saying.
diego 14y ago
Agree, the title is poorly editorialized. Proofs and validations are very different things. Validation is about being confident that a model can make useful predictions.

aakil 14y ago

Plasmyd does this via crowdsourcing, it lets users comment on papers so that researchers can point out anomalies or post their inability to reproduce the same results. It also gives the author a chance to explain their work.

bmahmood 14y ago
Bilal from Science Exchange here. Definitely agree that Plasmyd is a valuable service. We feel this Initiative operates in complement, providing a mechanism for independent validation.
pcrh 14y ago
It is frustratingly difficult to get biomedical scientists to openly discuss the work in their field. PLoS tried to have discussion sections to papers, and Nature has tried various approaches as well. Neither have worked. I looked-up my favorite topic (a popular one) on Plasmyd, and found 20 papers, none of which had a single comment associated with it. It's honestly a vast waste of intellectual potential, I wish I knew the solution!

fbomb 14y ago

"Validations are conducted blind, on a fee-for-service basis." Seems like a potential conflict of interest there.

archgoon 14y ago
How? If they don't know what the answer is supposed to be, it's not clear how they can fake data. This can further be improved by universities contributing to a 'noise' fund, where requested experiments are expected to _not_ turn anything up. Also, the main purpose of this is not to catch corruption, just prevent mostly honest researchers from fooling themselves (and then others), and increasing the base line for research.
bmahmood 14y ago
Bilal from Science Exchange here. The validations are blind in the sense that we match an investigator who has submitted a study to a provider who can validate it. The investigator doesn't have a choice of who validates their study though. The fee-for-service actually helps to guard against biased results in two ways. Firstly, providers who validate studies are paid on completion, regardless of a reproducible or irreproducible outcome. Secondly, operating on a fee-for-service basis allows providers to operate outside academic incentives for publications, ensuring no incentive for biased-positive results.
pcrh 14y ago
I see it as something like the National Institute of Standards and Technology (http://www.nist.gov/index.html http://www.nist.gov/index.html). I don't think most academic labs will pay for this. However, I can see it as being something that VC investors might like to see the next time someone presents with a wonder drug.

Xcelerate 14y ago

This is great! Too often scientific papers are published that are never actually checked by anyone else. A key part of the scientific method is reproducibility though.

graeme 14y ago

I was very surprised when I first learned that results could be published without first having been reproduced. I thought at the least they should have been published with a separate label. This is a worthy initiative.

tokenadult 14y ago
I was very surprised when I first learned that results could be published without first having been reproduced. A lot of psychology journals have been burned by that over the years, http://chronicle.com/blogs/percolator/is-psychology-about-to-come-undone/29045 http://chronicle.com/blogs/percolator/is-psychology-about-to... http://simplystatistics.org/post/21326470429/replication-psychology-and-big-science http://simplystatistics.org/post/21326470429/replication-psy... http://scholarlykitchen.sspnet.org/2012/04/26/reproducibility-an-attempt-to-test-the-psychology-literature-underscores-a-growing-fault-line/ http://scholarlykitchen.sspnet.org/2012/04/26/reproducibilit... and as a consequence, just within the last year or so, the most prestigious psychology journals are beginning to ask authors of papers on new experimental research to show replication across at least two distinct data sets. This, in turn, is driving more collaboration among researchers at different research centers. It's a long overdue development, one in accord with best practice in science, http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html and I hope this practice spreads to most disciplines that publish new research reports in the major journals of the discipline. (Although the requirement for prepublication replication is not explicitly mentioned in the publication guidelines of Psychological Bulletin, http://www.apa.org/pubs/journals/bul/index.aspx http://www.apa.org/pubs/journals/bul/index.aspx which is the most prestigious psychology journal published in the United States, I have been told at the University of Minnesota behavior genetics "journal club" that replication across more than one data set is becoming a tacit requirement for publication in most of the better journals. Perhaps that guideline will eventually be made explicit for all of the better journals.)
_delirium 14y ago
Traditionally, the public literature is how scientists communicated, which is what made republication possible in the first place. Lab A, perhaps in Germany, works on a research project, and gets some results. They write up their methodology and results, and communicate them to other scientists by publishing the write-up in a journal. The journal will have the write-up peer-reviewed before publishing it, but that doesn't mean verifying it's correct, only that it's a contribution to the scientific literature worth communicating to its readership. For example, it would make sure the paper has done appropriate data analysis, has cited relevant existing work, is not outright kookery, etc. Now other scientists, anywhere in the world, will read this and decide what they think of it, which may include attempting to replicate it. If they find different results, or have a qualm with an interpretation or methodology, they can write a response, which will also (after peer-review) be communicated by the journal to its readership, either as another paper, or if shorter, as a letter. Thus the back-and-forth of claim/replication/critique happens, all in the open air of the journal's pages, with any lab anywhere in the world able to follow the conversation and jump in if they wish. To me that's more transparent than doing all this in some kind of closed pre-publication process. Perhaps in the modern era, the open communication doesn't necessarily have to happen in journals anymore: perhaps it should happen on arXiv, and journals should become more conservative in what they publish. But I would like it to still be public.

JohnnyBBad 14y ago

Lemme know when somebody reproduces monkeys evolving into humans.

podperson 14y ago

The title irks me. Instead of "prove it", how about "let us reproduce your results" or "show us the data". Proof in this context is a mistaken concept.

semenko 14y ago

There are very few journals that do this -- I can't imagine this ever taking off. Reproducing experiments seems like a costly (and mostly thankless) effort that few PIs would ever take up. The only journal I know of that completely reproduces results is Organic Syntheses (http://www.orgsyn.org/ http://www.orgsyn.org/), which reproduces every reaction before publication and has a Procedure Checklist for authors: http://www.orgsyn.org/AuthorChecklist.pdf http://www.orgsyn.org/AuthorChecklist.pdf

bmahmood 14y ago
Hi, this is Bilal from Science Exchange. Appreciate your concern, but that is why we feel Science Exchange is uniquely positioned to make a difference. We have developed a network of 1000 core facilities and CROs on our platform, who operate on a fee-for-service basis to conduct experimental services. The Reproducibility Initiative will leverage this network to outsource specific experiments within a study to these facilities.
pcrh 14y ago
It is quite common for experiments to be replicated, usually as preliminary to a further advance. If the findings claimed are significant enough, other labs will often attempt to replicate a few of the key experiments, and if they don't do so quickly, abandon that direction of investigation, and not report their negative findings.

abbottry 14y ago

Title changed -- geeeeeze #cowersinthecorner

kevinalexbrown 14y ago

This is excellent. I have a few questions: You need to provide the background of your study, the types of experiments undertaken, the materials and methods, and initial results of your study. Do the technicians reproducing the results get to see the initial results? It seems like it might be more accurate if they didn't. Lots of parameters can be fudged and adjusted, a la Millikan's Oil Drop, when the results don't quite match. I imagine this might be exacerbated with the necessity of researcher-validator communication. How are conflicts resolved? If my results are not validated, someone made a mistake - me or the validator. If both parties stand by their mutually incompatible results, where does it go from there? I can imagine a lot of researchers I know feeling annoyed that someone whose expertise they cannot verify (due to anonymity) won't "do my experiment correctly". I imagine that in time there might be specific requirements or explicit funding allocations for such reproduction on grant applications, which would really allow it to take off. As it stands, I imagine a lot of PI's would just ask "hmm, I can spend money that might risk my already high-impact paper, or I can keep the money and not be considered wrong." Still, this is a great first step toward facilitating a central tenet of the scientific method. Congratulations.

bmahmood 14y ago
This is Bilal from Science Exchange, and we greatly appreciate your support for our initiative! We will provide the methodology of the original study to those reproducing the results, while the results we believe will be helpful to check against. For conflicts, it's true we can't force an investigator to publish or note the lack of reproduced outcomes. We do hope they will through the PLOS Collection, for transparency. We do feel though it can provide a valuable check for 'failing fast', for those investigators who want robust results. In this initial stage, we also agree that funding will be difficult. That's why we hope to focus on small biotechs and research labs that are interested in commercializing their research, and need to show robustness of results for licensing opportunities. Hopefully then, it can serve as a proof-of-principle for funding agencies to provide a requirement or increase support for reproducibility.
- pcrh 14y ago
  I think this will be a very valuable service for small biotechs. It would improve the value of their patents and increase chances of getting further funding. As an experienced scientist myself, I can say there is plenty of scope for misunderstanding and misinterpretation, no matter how careful the original authors were. So I only hope that there will be some (maybe blinded) mechanism for communication between the original researchers and those replicating the published findings, in the event of the "usual" complications. Also, who would do this replication? Would some of it be outsourced to academic labs with the requisite experience? Advanced findings often depend on advanced techniques. Outsourcing could be problematic, politically, since Big Shot No.1 might not be too interested in shooting down his pal, Alpha-MD-PhD.

frisco 14y ago

It's very expensive (prohibitively so) to reproduce any substantial study. It also feels like there's a lot of potential downside and very little reward, since there's a presumption of correctness in published papers today. Further, wet lab protocols are subjective enough that I'd imagine most labs would ignore a negative result as having been performed incorrectly. Where does the money for this come from? Are you expecting that labs will write this cost into their grants? Have you seen interest from grant agencies to then actually pay for this?

deleted 14y ago
[deleted]
bmahmood 14y ago
Hi, this is Bilal from Science Exchange. We are in discussions with funding agencies and foundations to possibly gather increased financial support. We feel this initial launch will provide a proof-of-principle to gather support. Currently though, we hope investigators will pay for the validation themselves. We believe it can provide a valuable service to those in small biotechs or those who want to commercialize their research, to improve the robustness of outcomes for potential licensing opportunities.
gwern 14y ago
> Further, wet lab protocols are subjective enough that I'd imagine most labs would ignore a negative result as having been performed incorrectly. That kind of illustrates the usefulness of replication! If the effect is so finicky that it cannot even be replicated in another lab, in what sense is that result interesting, useful, - or even real?
- frisco 14y ago
  I think you overestimate most biology labs. It's really easy to screw up a protocol in subtle ways. Often it took the original lab weeks of debugging to get it to work (yes, to get it to work, not to produce a false positive). Things get more reliable as time goes on and methods become better understood, but the critical steps in most experiments are usually nontrivial to replicate in the beginning. Especially by CROs, which are notorious black boxes, incredibly expensive, and often just plain old unreliable.

joe_the_user 14y ago

I don't see this approach really able to solve the underlying problems that it references from http://www.nytimes.com/2012/04/17/science/rise-in-scientific-journal-retractions-prompts-calls-for-reform.html?_r=1 http://www.nytimes.com/2012/04/17/science/rise-in-scientific... and other articles. "Each year, every laboratory produces a new crop of Ph.D.’s, who must compete for a small number of jobs, and the competition is getting fiercer. In 1973, more than half of biologists had a tenure-track job within six years of getting a Ph.D. By 2006 the figure was down to 15 percent." I would claim that science requires some basic integrity in its practitioners and if an institution is treating it's members as so many throw-away resources, it is hard to expect those members to move ahead with great idealism. The model of every desperate competitor watching every other competitor seems to be the replacement for model of science as a high ideal. I don't see it working out well.

bmahmood 14y ago
It's true the issue of reproducibility in research is a complex one, with many underlying factors. Culture, misaligned incentives, and lack of resources play strong roles. We believe however our Initiative can help in laying an initial framework for how one can possibly address aspects of the problem. We've tried to build incentives (ease of outsourcing, rewarding publications) that factor into this issue, and can assist in improving outcomes. But we definitely agree that a holistic solution will require further changes to the academic research infrastructure.

jayzee 14y ago

The part I like about the execution is where you have CROs and Core Facilities, and not scientists themselves validate results. Apparently Milikan when he measured the electric charge was off (smaller than actual) in the measurements... but researchers who checked, worried perhaps about their academic reputation (speculating here) slowly adjusted this number upwards over several years till it asymptotically approached the truth. Having a 3rd-party, that is less impacted by academic politics might be a good thing. From Feynman's Caltech commencement speech on this: We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off, because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of the electron, after Millikan. If you plot them as a function of time, you find that one is a little bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher. Why didn't they discover that the new number was higher right away? It's a thing that scientists are ashamed of--this history--because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong--and they would look for and find a reason why something might be wrong. When they got a number closer to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that. We've learned those tricks nowadays, and now we don't have that kind of a disease. [1] http://www.lhup.edu/~dsimanek/cargocul.htm http://www.lhup.edu/~dsimanek/cargocul.htm

larsberg 14y ago

The NSF CISE guidelines (which apply to much of the funded CS research that folks on HN care about) currently require you to retain all data required to reproduce your experiment and make them available to any reasonable request: http://www.nsf.gov/cise/cise_dmp.jsp http://www.nsf.gov/cise/cise_dmp.jsp My advisor is on appointment there right now, and they're working on guidelines that require you to provide full reproducibility of your program results (modulo system differences). At least for software, things are looking up. Of course, reproducibility is both infinitely easier (I can do it on any system!) and harder (what do you mean the kernel patchlevel or CPU/GPU sub-model-number matters?).