3 ms·

In the natural sciences, reproducibility means that a separate team run an experiment on independent apparatus, configured from written information (e.g., the p

In the natural sciences, reproducibility means that a separate team run an experiment on independent apparatus, configured from written information (e.g., the paper). Not that you re-run the experiment on the original apparatus set up by and made available by the original authors. Clicking a button is not "reproducibility" in that sense, though it is better than not being able to do even that. Shipping code/VMs/etc. and having a 2nd team just re-run it has too much of the original lab in it to be real replication; it's more like just inspection of the 1st setup. Which is better than no inspection, but worse than independent reproduction of the results. Applied to CS, there's really no way around it: reproduction requires that researchers claiming to replicate a result implement it independently. If there are not two independently produced implementations that both confirm the result, the research hasn't been reproduced. The process of doing so helps to discover cases where the original results were due to idiosyncracies of the original implementation, test setup, etc. It'd even be ideal if replication was done in as dissimilar a setup as possible, to find cases where the results unexpectedly depend on details of the original setup not thought to be important. If anything I think there are quite some dangers that reproduction will be decreased by the current trend towards what's questionably called "reproducible research" as a euphemism for "code reuse". If people reuse code rather than doing their own independent reimplementation of methods as stated in papers, erroneous results can lurk for years and infect other research as well. (I think code reuse is good for engineering practicality, and making permissively licensed code available is also a good way of getting researchy methods out of academia into the real world. But I think it is quite wrong to call reusing the original researcher's setup, rather than independently producing your own, "reproduction" in the scientific sense.)

sjwright 12y ago
The necessity of truly independent reproduction diminishes when the researchers/engineers can supply an implementation that does real work in real situations. A very high bar of self-evidence, if you will. True reproduction remains necessary to confirm that the stated underlying science is the cause, and not some other variable that might be legitimate and remarkable but undocumented.
- drewcrawford 12y ago
  Providing an implementation of a network protocol is hardly doing "real work in real situations". I mean, it ships the packets from here to there but so does every other implementation of every other protocol. The thing that is interesting or novel about this protocol as opposed to the thousands of other proposals is that it's supposed to be faster than the other proposals for a broad range of applications with a broad range of link topologies and across a broad range of software/hardware platforms. Of course, to establish that claim replication is completely critical.
  - sjwright 12y ago
    > Providing an implementation of a network protocol is hardly doing "real work in real situations". In what sense is a functional implementation not able to do real work?

sjwright 12y ago

The necessity of truly independent reproduction diminishes when the researchers/engineers can supply an implementation that does real work in real situations. A very high bar of self-evidence, if you will. True reproduction remains necessary to confirm that the stated underlying science is the cause, and not some other variable that might be legitimate and remarkable but undocumented.

drewcrawford 12y ago
Providing an implementation of a network protocol is hardly doing "real work in real situations". I mean, it ships the packets from here to there but so does every other implementation of every other protocol. The thing that is interesting or novel about this protocol as opposed to the thousands of other proposals is that it's supposed to be faster than the other proposals for a broad range of applications with a broad range of link topologies and across a broad range of software/hardware platforms. Of course, to establish that claim replication is completely critical.
- sjwright 12y ago
  > Providing an implementation of a network protocol is hardly doing "real work in real situations". In what sense is a functional implementation not able to do real work?