3 ms·

"In academia, the end product is a publication and your code needs to work only once." In my experience as ex-academic, the above is all too true in too many c

"In academia, the end product is a publication and your code needs to work only once." In my experience as ex-academic, the above is all too true in too many computational fields (and many sub-fields of CS). So I ask what scientific value (or any kind of value, for that matter) is being generated by grinding out publications with results that won't be repeatable?

a_bonobo 15y ago
Depends on the journal you're publishing in. Lots of journals (at least in my biological sciences field) require you to submit your code along with your publication so that a) the editor can repeat your results to accept your publication and b) for other scientists to repeat your stuff. Biostatistics AFAIK has an editor responsible solely for reproduction of results. Personally saying lots of my code is repeatedly re-used by colleagues who can't program, for example scripts that parse BLAST-output and get the best result for each query etc.
- imaginaryunit 15y ago
  Your point is well-taken and (thankfully) there are safeguards in place at top journals. However, Rob Tibshirani examined this a while back with a case study and came up with some unsettling conclusions about reproducibility: http://www-stat.stanford.edu/~tibs/FL/report/ http://www-stat.stanford.edu/~tibs/FL/report/. The case study highlights the importance of debugging not only code, but also the entire experimental process.
  - a_bonobo 15y ago
    True! If you want a newer example, here's a very recent paper which looks at the results of a couple of RNA-Seq-papers and after being able to reproduce only 12% of the results, concludes that they all used the wrong assumptions in their statistics: http://www.plosgenetics.org/article/info%3Adoi%2F10.1371%2Fjournal.pgen.1002600 http://www.plosgenetics.org/article/info%3Adoi%2F10.1371%2Fj... Which shows that the editor really needs to know what's going on and why in each paper - I don't think every editor can know everything about his or her field.

a_bonobo 15y ago

Depends on the journal you're publishing in. Lots of journals (at least in my biological sciences field) require you to submit your code along with your publication so that a) the editor can repeat your results to accept your publication and b) for other scientists to repeat your stuff. Biostatistics AFAIK has an editor responsible solely for reproduction of results. Personally saying lots of my code is repeatedly re-used by colleagues who can't program, for example scripts that parse BLAST-output and get the best result for each query etc.

imaginaryunit 15y ago
Your point is well-taken and (thankfully) there are safeguards in place at top journals. However, Rob Tibshirani examined this a while back with a case study and came up with some unsettling conclusions about reproducibility: http://www-stat.stanford.edu/~tibs/FL/report/ http://www-stat.stanford.edu/~tibs/FL/report/. The case study highlights the importance of debugging not only code, but also the entire experimental process.
- a_bonobo 15y ago
  True! If you want a newer example, here's a very recent paper which looks at the results of a couple of RNA-Seq-papers and after being able to reproduce only 12% of the results, concludes that they all used the wrong assumptions in their statistics: http://www.plosgenetics.org/article/info%3Adoi%2F10.1371%2Fjournal.pgen.1002600 http://www.plosgenetics.org/article/info%3Adoi%2F10.1371%2Fj... Which shows that the editor really needs to know what's going on and why in each paper - I don't think every editor can know everything about his or her field.