5 ms·

One idea to improve the state of things is to require graduate students to verify some number of external studies. In addition to helping with the problem of no

by tensor 13y ago

One idea to improve the state of things is to require graduate students to verify some number of external studies. In addition to helping with the problem of not enough review, it would make an excellent practical test for doctoral candidates. It wouldn't work for every field and area, but it could work for a significant subset of research.

burntsushi 13y ago
As a PhD student in CS, this is an incredibly good idea---even if it stops short of verification and stays somewhere at "reproduce". It can be a lot of work to reproduce a single publication, and often requires very careful reading and attention to detail. I've done it a few times myself (partially because my work requires it and partially because I wanted to convince myself of the work I was using), and it was an incredibly valuable learning experience.
- abecedarius 13y ago
  In the case of CS, replication should become a matter of 'git clone' and 'make'. (Or running a prepared virtual machine, etc.) Yes, it can be worthwhile learning by reimplementing from a high-level description -- I've done it a lot -- but that doesn't excuse making replication difficult.
  - dllthomas 13y ago
    I'm not convinced that's "replication" in the sense that's important in science. Bugs in the original test will be reproduced in the attempted reproduction. It's useful for "this person didn't flat out lie about their results", which is probably also a good thing, and for finding issues when replication fails, so I don't disagree that it should happen. I just disagree with calling it "replication".
    - abecedarius 13y ago
      Fair enough -- maybe we should find a different word. It's a reasonable bare minimum standard in the computing world, where it's practical: it makes it possible to trace back from questions and problems with the claim.
      - dllthomas 13y ago
        Sure - it's clearly a good idea, where feasible (and git repo + whatever-special-hardware should allow the same thing, for the one reasonable case I can think of where it wouldn't be feasible).
  - hershel 13y ago
    Not just in the case of cs. It could work on electronics, where building some stuff is relatively easy, given design data. In general, a replication requirement might drive us to build replication tools for each science, which could be quite useful.
  - burntsushi 13y ago
    > Yes, it can be worthwhile learning by reimplementing from a high-level description -- I've done it a lot -- but that doesn't excuse making replication difficult. I don't understand what you're trying to achieve by saying this. Did I somehow imply that people should be excused for not making their work reasonably reproducible? It my sub-field, it is utterly ridiculous to expect to reproduce work by running `git clone && make` or by getting a prepared VM. Computational biologists are not known for their systems or software skills.
    - abecedarius 13y ago
      Well, the context here is discussion of the problem of unreplicable research. I do get the impression (including from your post and reply) that computational biologists need to get their act together on that front; see http://ivory.idyll.org/blog/ http://ivory.idyll.org/blog/ for one researcher in that field who sometimes blogs on this theme.
      - burntsushi 13y ago
        Indeed, we do need to get our act together. I'm still just starting, but I have a couple of ideas on proactive things I can do, but it's an uphill battle. Thanks for that link---it's always nice to see other people in the field concerned with this. There aren't enough of them.
        abecedarius 13y ago
        Good luck! I know it's a big task to change the culture (and I didn't mean to imply I thought it'd be easy). You might like http://michaelnielsen.org/blog/reinventing-discovery/ http://michaelnielsen.org/blog/reinventing-discovery/ too.
moocowduckquack 13y ago
require graduate students to verify some number of external studies. That's how Reinhart and Rogoff's “Growth in a Time of Debt” spreadsheet error was finally unearthed, unfortunately that was after it had already been instituted internationally as public policy, but hey, you can't have everything. http://www.dailyfinance.com/on/reinhart-rogoff-debt-GDP-spreadsheet-error/ http://www.dailyfinance.com/on/reinhart-rogoff-debt-GDP-spre...
PeterisP 13y ago
Motivation is an issue - each verification would require a bunch man-months of effort, so it won't happen unless there is separate funding for that or somehow magically started to be as prestigious as putting the same effort in a new experiment/publication. "Requiring" has the same motivation problem - those who could require it, currently would rather require those students to do something that brings funding or prestige, so they won't.
- tensor 13y ago
  Graduate students are already required to take courses and various tests that don't bring funding or prestige. This could simply be an additional requirement, or replace an existing requirement. In time, such a program would bring quite a lot of prestige as flaws are discovered and fixed in existing work. It's really the easiest and most immediate way to address this problem since it only needs to involve single institutions (whose faculty presumably care deeply about this issue). It doesn't seem likely that journals will suddenly start valuing verification work. Similarly, politicians and funding agencies appear uninterested in actual science; they care only about their careers or immediate application to the politically popular cause of the day.
  - XorNot 13y ago
    Graduate students already verify people's work. Because every new graduate project involves building on the work of previous science. Which by definition involves re-verifying the work to show you get the same results. If the results are hard to replicate, or dependent on other causes, then usually that's when a project shifts or when the knowledge pool expands (for example chemistry is fraught with environmental effect dangers - fluorescent lights provide UV to reactions, your glassware has imperfections, the temperature and humidity of labs varies with climate).
- eli_gottlieb 13y ago
  I would say that falsifying an existing publication should easily be at least as prestigious and funding-worthy as having made that publication in the first place.
thearn4 13y ago
I like the idea. Though in my field (mathematics), I think this is actually not at all uncommon already (reviewing and sanity checking unpublished work of your adviser and their colleagues). Then again, we have the benefit of being able to "make" our own data so to speak, don't have IRBs to worry about, etc. This is trickier in other disciplines.
cossatot 13y ago
This is an excellent idea for a graduate course; it could be more collaborative for bigger projects, or one-offs for smaller ones, dependent on the field and type of study, as you've mentioned. I think students should get course credit for it, instead of adding another requirement for PhDs, just to keep them from taking longer than they already do...
daughart 13y ago
I think this is a horrible idea. The average age of R01 has risen to over 40! Why add another 2 years by requiring students to reproduce work that is already published? It adds virtually no value to a graduate degree. The most difficult part of science is creating new hypotheses and designing and executing the experiments to test them. It takes years to learn how to do this as is, without adding on time to use a method you have no interest in, to re-demonstrate an experiment that has already been designed, to test a hypothesis that is already in the literature.

by tensor 13y ago

burntsushi 13y ago

As a PhD student in CS, this is an incredibly good idea---even if it stops short of verification and stays somewhere at "reproduce". It can be a lot of work to reproduce a single publication, and often requires very careful reading and attention to detail. I've done it a few times myself (partially because my work requires it and partially because I wanted to convince myself of the work I was using), and it was an incredibly valuable learning experience.

abecedarius 13y ago
In the case of CS, replication should become a matter of 'git clone' and 'make'. (Or running a prepared virtual machine, etc.) Yes, it can be worthwhile learning by reimplementing from a high-level description -- I've done it a lot -- but that doesn't excuse making replication difficult.
- dllthomas 13y ago
  I'm not convinced that's "replication" in the sense that's important in science. Bugs in the original test will be reproduced in the attempted reproduction. It's useful for "this person didn't flat out lie about their results", which is probably also a good thing, and for finding issues when replication fails, so I don't disagree that it should happen. I just disagree with calling it "replication".
  - abecedarius 13y ago
    Fair enough -- maybe we should find a different word. It's a reasonable bare minimum standard in the computing world, where it's practical: it makes it possible to trace back from questions and problems with the claim.
    - dllthomas 13y ago
      Sure - it's clearly a good idea, where feasible (and git repo + whatever-special-hardware should allow the same thing, for the one reasonable case I can think of where it wouldn't be feasible).
- hershel 13y ago
  Not just in the case of cs. It could work on electronics, where building some stuff is relatively easy, given design data. In general, a replication requirement might drive us to build replication tools for each science, which could be quite useful.
- burntsushi 13y ago
  > Yes, it can be worthwhile learning by reimplementing from a high-level description -- I've done it a lot -- but that doesn't excuse making replication difficult. I don't understand what you're trying to achieve by saying this. Did I somehow imply that people should be excused for not making their work reasonably reproducible? It my sub-field, it is utterly ridiculous to expect to reproduce work by running `git clone && make` or by getting a prepared VM. Computational biologists are not known for their systems or software skills.
  - abecedarius 13y ago
    Well, the context here is discussion of the problem of unreplicable research. I do get the impression (including from your post and reply) that computational biologists need to get their act together on that front; see http://ivory.idyll.org/blog/ http://ivory.idyll.org/blog/ for one researcher in that field who sometimes blogs on this theme.
    - burntsushi 13y ago
      Indeed, we do need to get our act together. I'm still just starting, but I have a couple of ideas on proactive things I can do, but it's an uphill battle. Thanks for that link---it's always nice to see other people in the field concerned with this. There aren't enough of them.
      - abecedarius 13y ago
        Good luck! I know it's a big task to change the culture (and I didn't mean to imply I thought it'd be easy). You might like http://michaelnielsen.org/blog/reinventing-discovery/ http://michaelnielsen.org/blog/reinventing-discovery/ too.

moocowduckquack 13y ago

require graduate students to verify some number of external studies. That's how Reinhart and Rogoff's “Growth in a Time of Debt” spreadsheet error was finally unearthed, unfortunately that was after it had already been instituted internationally as public policy, but hey, you can't have everything. http://www.dailyfinance.com/on/reinhart-rogoff-debt-GDP-spreadsheet-error/ http://www.dailyfinance.com/on/reinhart-rogoff-debt-GDP-spre...

PeterisP 13y ago

Motivation is an issue - each verification would require a bunch man-months of effort, so it won't happen unless there is separate funding for that or somehow magically started to be as prestigious as putting the same effort in a new experiment/publication. "Requiring" has the same motivation problem - those who could require it, currently would rather require those students to do something that brings funding or prestige, so they won't.

tensor 13y ago
Graduate students are already required to take courses and various tests that don't bring funding or prestige. This could simply be an additional requirement, or replace an existing requirement. In time, such a program would bring quite a lot of prestige as flaws are discovered and fixed in existing work. It's really the easiest and most immediate way to address this problem since it only needs to involve single institutions (whose faculty presumably care deeply about this issue). It doesn't seem likely that journals will suddenly start valuing verification work. Similarly, politicians and funding agencies appear uninterested in actual science; they care only about their careers or immediate application to the politically popular cause of the day.
- XorNot 13y ago
  Graduate students already verify people's work. Because every new graduate project involves building on the work of previous science. Which by definition involves re-verifying the work to show you get the same results. If the results are hard to replicate, or dependent on other causes, then usually that's when a project shifts or when the knowledge pool expands (for example chemistry is fraught with environmental effect dangers - fluorescent lights provide UV to reactions, your glassware has imperfections, the temperature and humidity of labs varies with climate).
eli_gottlieb 13y ago
I would say that falsifying an existing publication should easily be at least as prestigious and funding-worthy as having made that publication in the first place.

thearn4 13y ago

I like the idea. Though in my field (mathematics), I think this is actually not at all uncommon already (reviewing and sanity checking unpublished work of your adviser and their colleagues). Then again, we have the benefit of being able to "make" our own data so to speak, don't have IRBs to worry about, etc. This is trickier in other disciplines.

cossatot 13y ago

This is an excellent idea for a graduate course; it could be more collaborative for bigger projects, or one-offs for smaller ones, dependent on the field and type of study, as you've mentioned. I think students should get course credit for it, instead of adding another requirement for PhDs, just to keep them from taking longer than they already do...

daughart 13y ago

I think this is a horrible idea. The average age of R01 has risen to over 40! Why add another 2 years by requiring students to reproduce work that is already published? It adds virtually no value to a graduate degree. The most difficult part of science is creating new hypotheses and designing and executing the experiments to test them. It takes years to learn how to do this as is, without adding on time to use a method you have no interest in, to re-demonstrate an experiment that has already been designed, to test a hypothesis that is already in the literature.