4 ms·

You ask that like it is something new. Reproducibility[0] is something at the core of science. The problem these days is that spending time doing replication i

by zouhair 10y ago

You ask that like it is something new. Reproducibility[0] is something at the core of science. The problem these days is that spending time doing replication is not glamourous and will not help you get funds. [0]: https://en.wikipedia.org/wiki/Reproducibility https://en.wikipedia.org/wiki/Reproducibility

analog31 10y ago
I suppose the funding agencies could play a role in this. When you submit a proposal that cites prior work as a basis or motivation for you project, you should be required to show that either 1) The cited works have been reproduced, or 2) you're going to reproduce them.
- zouhair 10y ago
  And also we need to start publishing negative studies way more.
  - analog31 10y ago
    I dimly recall some physics papers from years ago, with titles such as: "New experimental limit on the existence of..." Whenever my project was going particularly badly, we'd lighten things up by describing it as a negative study. One such highlight was: "New experimental limit on the existence of oil in the vacuum pump."
Obi_Juan_Kenobi 10y ago
The problem is worst in fields that don't have reproducibility 'built in' to the field. I do genetics and development, and the main sanity check we have is the distribution of mutant lines. If you say that mutant X does Y, other people are likely going to see that (or not) when they get their hands on it and start poking around. This strength of working with mutants is at the core of the success of molecular biology. Even if don't set out to confirm someone else's results, you're quite likely to come across some inconsistencies in the course of investigation. If a field lacks that sort of mechanism, they need to take special care to address reproducibility.
- zouhair 10y ago
  Add to that the impossibility to publish negative studies. Science really needs to get its shit straight.
- toufka 10y ago
  Exactly. Most 'good science' start with premises A, B, and C from their respective papers, and go on to show that X is likely true because of this new data. In general, it would be impossible to propose X if A, B, and C were not themselves true - they were the foundations of this new research. In that way, even though the publication of X may not have directly (though often precisely directly) repeat the experiments of A, B, and C, they still have replicated the core concepts. As noted, this does not get rid of the confounding negative effects, where paper D was also used as a (in retrospect, faulty) premise as well. Though no one ever actually comes out and says 'D' is bad. Over time, A, B, and C theories accrue significant weight while D falls off. It's not as explicit as some may like, but in the end the spirit of replication is well and alive.

by zouhair 10y ago

analog31 10y ago

I suppose the funding agencies could play a role in this. When you submit a proposal that cites prior work as a basis or motivation for you project, you should be required to show that either 1) The cited works have been reproduced, or 2) you're going to reproduce them.

zouhair 10y ago
And also we need to start publishing negative studies way more.
- analog31 10y ago
  I dimly recall some physics papers from years ago, with titles such as: "New experimental limit on the existence of..." Whenever my project was going particularly badly, we'd lighten things up by describing it as a negative study. One such highlight was: "New experimental limit on the existence of oil in the vacuum pump."

Obi_Juan_Kenobi 10y ago

The problem is worst in fields that don't have reproducibility 'built in' to the field. I do genetics and development, and the main sanity check we have is the distribution of mutant lines. If you say that mutant X does Y, other people are likely going to see that (or not) when they get their hands on it and start poking around. This strength of working with mutants is at the core of the success of molecular biology. Even if don't set out to confirm someone else's results, you're quite likely to come across some inconsistencies in the course of investigation. If a field lacks that sort of mechanism, they need to take special care to address reproducibility.

zouhair 10y ago
Add to that the impossibility to publish negative studies. Science really needs to get its shit straight.
toufka 10y ago
Exactly. Most 'good science' start with premises A, B, and C from their respective papers, and go on to show that X is likely true because of this new data. In general, it would be impossible to propose X if A, B, and C were not themselves true - they were the foundations of this new research. In that way, even though the publication of X may not have directly (though often precisely directly) repeat the experiments of A, B, and C, they still have replicated the core concepts. As noted, this does not get rid of the confounding negative effects, where paper D was also used as a (in retrospect, faulty) premise as well. Though no one ever actually comes out and says 'D' is bad. Over time, A, B, and C theories accrue significant weight while D falls off. It's not as explicit as some may like, but in the end the spirit of replication is well and alive.