5 ms·

Every academic cries about this but no one wants to accept an obvious solution: Add reprodicibility criteria for peer review. Right now even Tier 1 conferences

Every academic cries about this but no one wants to accept an obvious solution: Add reprodicibility criteria for peer review. Right now even Tier 1 conferences do not have this criteria in their peer review system. The reviewer simply can't reject paper because authors didn't provided reproducibility information! This is truely a plague in research right now. I've came across quite a few instances where authors told me that their experiments weren't reproducible even for them! They do not note this in paper because ultimately everyone needs to show some result for the funding they received (i.e. "published paper"). They obviously never want to share any code, data files, hardware etc. On one instance, author wrote me back that they can't share code with me because they lost all the code because their hard drive crashed! Reproducibility is a fundamental tenant of doing scientific work and this is actively and completely ignored in current peer review system. I think conference chairs needs to take stand on this. We get now 4X to 8X papers in tier 1 conferences. Reproducibility could be a great filter when area chairs are scrambling to find reasons to reject papers. Sure, there will be papers where very specialized hardware or internal infrastructure of 10,000 computers were used. But those papers would be great for Tier 2 conferences.

mpfundstein 8y ago
Listen to this man folks!!! Seriously, why do you think the academic world doesn’t change? Is there too muc vested interest in the current system?
nonbel 8y ago
> "Add reprodicibility criteria for peer review." Passing peer review is something used as a substitute for reproducibility. Just like testing a null hypothesis is a substitute for testing the research hypothesis, and citation count is used as a substitute for making precise and accurate predictions. The procedure (scientific method) that has lead to all the great stuff we have around us today, I'd even call it a pillar of civilization, has been undergoing a piecemeal replacement since approximately WWII.
- rantanplan 8y ago
  > Passing peer review is something used as a substitute for reproducibility Wow, pretty low bar for science. You can have peer review for the actual results and what they mean, as well as them being reproducible. > The procedure (scientific method) that has lead to all the great stuff I don't think you know what the scientific method is.
  - nonbel 8y ago
    Scientific method (basically me just paraphrasing Imre Lakatos): 1) Explore and describe aspects of the world in detail, figure out what situations naturally arise or that can be devised that produce consistent, stable phenomenon. 2) Abduce (guess) an explanation for these consistent, stable phenomenon. 3) Explore the logical consequences of assuming your guess is correct. Figure out a few otherwise surprising (ie, inconsistent with other explanations people may have) predictions that can be deduced from it. 4) Collect data and compare it to the predictions generated in #3 5) Discard the guess or modify it to make it more consistent with the data produced in #4 6a) If the guess is modified, Go to 3. 6b) If the guess is discarded, Go to 2. A lot of cancer research seems to be failing at step 1, ie they can't even get into the "testing otherwise surprising predictions" loop since there is no consistent, stable phenomenon to trust.
    - rantanplan 8y ago
      From wikipedia: 1. Define a question 2. Gather information and resources (observe) 3. Form an explanatory hypothesis 4. Test the hypothesis by performing an experiment and collecting data in a reproducible manner 5. Analyze the data 6. Interpret the data and draw conclusions that serve as a starting point for new hypothesis 7. Publish results 8. Retest (frequently done by other scientists) Emphasis on 4) and the "reproducible manner". Otherwise what's the point really? We would resort to trust or belief in one's sayings.
      - nonbel 8y ago
        The wikipedia one is ok, but really most of that is stuffed into my #1 and it treats the "forming and testing a hypothesis" part too superficially. What did I write that makes you think the two are in conflict? I wrote: "figure out what situations naturally arise or that can be devised that produce consistent, stable phenomenon." This is the same as figuring out the "reproducible manner".
ThePhysicist 8y ago
I think this is a great idea at least for fields where reproducibility is comparatively easy to achieve, e.g. computer science: Include your data, code and run environment (using a suitable technology) with the paper to give reviewers a chance to "play" with your experiments (if they actually have the time for this). In other fields such as experimental physics or biology it might be more difficult to achieve the same effect though, as experiments are quite hard to repeat in general: For example, in my former field (experimental quantum computing), building the setup and fabricating the sample required for a given experiment could take years of effort, making it almost impossible to "just" reproduce someone's work for the sake of verification. That said, in experimental quantum physics the exciting results tended to get replicated within a couple of years by different teams anyway, not because these teams wanted to verify the results but rather because they wanted to build their own experiments on top of them (I imagine this is similar in other fields). Another natural way of exchanging knowledge and improving reproducibility was via the exchange of PostDocs and PhD students: If you do good work in one group, another group will usually be very eager to give you a position so that you can help them to set up the experiments there as well. I'd even argue that this is one of the main mode of knowledge dissemination in experimental science today, as most research papers are just extremely hard to reproduce without the specific -and often not encoded- knowledge of the individuals that ran the experiments. I'm not sure though if it is practical to document everything in a research paper in such a way so that any person can reproduce a given experiment, as many of the techniques are very specialized and a lot of the equipment in the labs (at least in physics) like sample holders, electronics, chip design templates and fabrication recipes is custom-built, so documenting down to the last detail would take years of effort. That's why (IMHO) written PhD theses are so important, because that's kind of the only place where you can write 200-400 pages about your work, and where you can include minute details such as your chip fabrication recipe, a description of your custom-built measurement software and a detailed summary of the experimental techniques used in your work. In that sense, PhD theses are probably more important to reproducibility than short papers.
- dekhn 8y ago
  Your description of experimental quantum physics exactly matches the work done in state of the art biology. It basically takes 10-20 years of training and excellent brains to reproduce your competitor's paper in a way that lets you build on top of it. Many people who complain that papers aren't reproducible just don't have the skill to move the state of the art, because it's become so esoteric and challenging to run the critical experiments. I used to be a "everything must be replicable by even the simplest of people" person, but I changed to "for progress to be made there must be <X> competent people who can reproduce challenging experiments and run new ones".
probably_wrong 8y ago
While I agree with you in general, I'd like to point out a maybe-not-so-small detail: > Sure, there will be papers where very specialized hardware or internal infrastructure of 10,000 computers were used. But those papers would be great for Tier 2 conferences. This is an accurate description of a _lot_ of Google papers when it comes to Computational Linguistics. And I don't know whether a Tier 1 conference that rejects the biggest players could remain a Tier 1 conference for long. There are many papers in Comp. Ling. right now that claim state of the art by running neural networks for a couple weeks on very expensive hardware. If those papers are not allowed in Tier 1 conferences then we should call it "state-of-the-cheap-art", but accepting them undermines the whole point since only a select few can afford it. I don't say it can't be done. But I do think that it's not trivial.
- rightbyte 8y ago
  "A pile of money for computer time" is a valid ingredience for a reproducable paper. Having opaque methodology is not on the other end.
  - deleted 8y ago
    [deleted]
dekhn 8y ago
If a researcher told me they couldn't provide primary data because their hard drive crashed, I'd contact their dean and the editor of the journal that published the article, and let them know. Researchers have a duty (unfortunately not well codified) to retain primary source data, and "hard drive crash" isn't an excuse (it means they didn't take even basic backup and archival measures).
Q6T46nT668w6i3m 8y ago
Sure, but I think non-preprint cultures would instantly become only-preprint cultures because I believe (and this might be folly) that most researchers rely on work they are confident about based on experience with the authors and details from the paper. I don’t know if this is that bad of a change since I admittedly do this already.
setgree 8y ago
Hi sytelus, Nature is piloting (analytical/computational) reproducibility as part of peer review with, disclosure, the company I work for, Code Ocean: http://blogs.nature.com/ofschemesandmemes/2018/08/01/nature-research-journals-trial-new-tools-to-enhance-code-peer-review-and-publication http://blogs.nature.com/ofschemesandmemes/2018/08/01/nature-... A few political science journals also hire staff to make sure that the code reproduce the results in the article: https://ajps.org/ajps-replication-policy/ https://ajps.org/ajps-replication-policy/ A nice writeup of the AJPS's experiences: https://www.insidehighered.com/blogs/rethinking-research/should-journals-be-responsible-reproducibility https://www.insidehighered.com/blogs/rethinking-research/sho...

mpfundstein 8y ago

Listen to this man folks!!! Seriously, why do you think the academic world doesn’t change? Is there too muc vested interest in the current system?

nonbel 8y ago

> "Add reprodicibility criteria for peer review." Passing peer review is something used as a substitute for reproducibility. Just like testing a null hypothesis is a substitute for testing the research hypothesis, and citation count is used as a substitute for making precise and accurate predictions. The procedure (scientific method) that has lead to all the great stuff we have around us today, I'd even call it a pillar of civilization, has been undergoing a piecemeal replacement since approximately WWII.

rantanplan 8y ago
> Passing peer review is something used as a substitute for reproducibility Wow, pretty low bar for science. You can have peer review for the actual results and what they mean, as well as them being reproducible. > The procedure (scientific method) that has lead to all the great stuff I don't think you know what the scientific method is.
- nonbel 8y ago
  Scientific method (basically me just paraphrasing Imre Lakatos): 1) Explore and describe aspects of the world in detail, figure out what situations naturally arise or that can be devised that produce consistent, stable phenomenon. 2) Abduce (guess) an explanation for these consistent, stable phenomenon. 3) Explore the logical consequences of assuming your guess is correct. Figure out a few otherwise surprising (ie, inconsistent with other explanations people may have) predictions that can be deduced from it. 4) Collect data and compare it to the predictions generated in #3 5) Discard the guess or modify it to make it more consistent with the data produced in #4 6a) If the guess is modified, Go to 3. 6b) If the guess is discarded, Go to 2. A lot of cancer research seems to be failing at step 1, ie they can't even get into the "testing otherwise surprising predictions" loop since there is no consistent, stable phenomenon to trust.
  - rantanplan 8y ago
    From wikipedia: 1. Define a question 2. Gather information and resources (observe) 3. Form an explanatory hypothesis 4. Test the hypothesis by performing an experiment and collecting data in a reproducible manner 5. Analyze the data 6. Interpret the data and draw conclusions that serve as a starting point for new hypothesis 7. Publish results 8. Retest (frequently done by other scientists) Emphasis on 4) and the "reproducible manner". Otherwise what's the point really? We would resort to trust or belief in one's sayings.
    - nonbel 8y ago
      The wikipedia one is ok, but really most of that is stuffed into my #1 and it treats the "forming and testing a hypothesis" part too superficially. What did I write that makes you think the two are in conflict? I wrote: "figure out what situations naturally arise or that can be devised that produce consistent, stable phenomenon." This is the same as figuring out the "reproducible manner".

ThePhysicist 8y ago

I think this is a great idea at least for fields where reproducibility is comparatively easy to achieve, e.g. computer science: Include your data, code and run environment (using a suitable technology) with the paper to give reviewers a chance to "play" with your experiments (if they actually have the time for this). In other fields such as experimental physics or biology it might be more difficult to achieve the same effect though, as experiments are quite hard to repeat in general: For example, in my former field (experimental quantum computing), building the setup and fabricating the sample required for a given experiment could take years of effort, making it almost impossible to "just" reproduce someone's work for the sake of verification. That said, in experimental quantum physics the exciting results tended to get replicated within a couple of years by different teams anyway, not because these teams wanted to verify the results but rather because they wanted to build their own experiments on top of them (I imagine this is similar in other fields). Another natural way of exchanging knowledge and improving reproducibility was via the exchange of PostDocs and PhD students: If you do good work in one group, another group will usually be very eager to give you a position so that you can help them to set up the experiments there as well. I'd even argue that this is one of the main mode of knowledge dissemination in experimental science today, as most research papers are just extremely hard to reproduce without the specific -and often not encoded- knowledge of the individuals that ran the experiments. I'm not sure though if it is practical to document everything in a research paper in such a way so that any person can reproduce a given experiment, as many of the techniques are very specialized and a lot of the equipment in the labs (at least in physics) like sample holders, electronics, chip design templates and fabrication recipes is custom-built, so documenting down to the last detail would take years of effort. That's why (IMHO) written PhD theses are so important, because that's kind of the only place where you can write 200-400 pages about your work, and where you can include minute details such as your chip fabrication recipe, a description of your custom-built measurement software and a detailed summary of the experimental techniques used in your work. In that sense, PhD theses are probably more important to reproducibility than short papers.

dekhn 8y ago
Your description of experimental quantum physics exactly matches the work done in state of the art biology. It basically takes 10-20 years of training and excellent brains to reproduce your competitor's paper in a way that lets you build on top of it. Many people who complain that papers aren't reproducible just don't have the skill to move the state of the art, because it's become so esoteric and challenging to run the critical experiments. I used to be a "everything must be replicable by even the simplest of people" person, but I changed to "for progress to be made there must be <X> competent people who can reproduce challenging experiments and run new ones".

probably_wrong 8y ago

While I agree with you in general, I'd like to point out a maybe-not-so-small detail: > Sure, there will be papers where very specialized hardware or internal infrastructure of 10,000 computers were used. But those papers would be great for Tier 2 conferences. This is an accurate description of a _lot_ of Google papers when it comes to Computational Linguistics. And I don't know whether a Tier 1 conference that rejects the biggest players could remain a Tier 1 conference for long. There are many papers in Comp. Ling. right now that claim state of the art by running neural networks for a couple weeks on very expensive hardware. If those papers are not allowed in Tier 1 conferences then we should call it "state-of-the-cheap-art", but accepting them undermines the whole point since only a select few can afford it. I don't say it can't be done. But I do think that it's not trivial.

rightbyte 8y ago
"A pile of money for computer time" is a valid ingredience for a reproducable paper. Having opaque methodology is not on the other end.
- deleted 8y ago
  [deleted]

dekhn 8y ago

If a researcher told me they couldn't provide primary data because their hard drive crashed, I'd contact their dean and the editor of the journal that published the article, and let them know. Researchers have a duty (unfortunately not well codified) to retain primary source data, and "hard drive crash" isn't an excuse (it means they didn't take even basic backup and archival measures).

Q6T46nT668w6i3m 8y ago

Sure, but I think non-preprint cultures would instantly become only-preprint cultures because I believe (and this might be folly) that most researchers rely on work they are confident about based on experience with the authors and details from the paper. I don’t know if this is that bad of a change since I admittedly do this already.

setgree 8y ago

Hi sytelus, Nature is piloting (analytical/computational) reproducibility as part of peer review with, disclosure, the company I work for, Code Ocean: http://blogs.nature.com/ofschemesandmemes/2018/08/01/nature-research-journals-trial-new-tools-to-enhance-code-peer-review-and-publication http://blogs.nature.com/ofschemesandmemes/2018/08/01/nature-... A few political science journals also hire staff to make sure that the code reproduce the results in the article: https://ajps.org/ajps-replication-policy/ https://ajps.org/ajps-replication-policy/ A nice writeup of the AJPS's experiences: https://www.insidehighered.com/blogs/rethinking-research/should-journals-be-responsible-reproducibility https://www.insidehighered.com/blogs/rethinking-research/sho...