4 ms·

This is not clear at all. It depends on the "result" in question. If I wrote a paper describing a super numerical algorithm for inverting matrices, and no one

by kkylin 6y ago

This is not clear at all. It depends on the "result" in question. If I wrote a paper describing a super numerical algorithm for inverting matrices, and no one is able to replicate the superior performance of my algorithm despite following the recipe in my paper, then whether they can run my code or not doesn't seem to be of the highest priority. Edit: more careful phrasing.

a1369209993 6y ago
> whether they can run my code or not doesn't seem to be of the highest priority. On the contrary; in that case there are four possibilities: a: your algorithm doesn't work at all, and your observations are a artifact of convenient inputs or inept measurements. b: your algorithm works, but the description in the paper is wrong or incomplete c: your algorithm works as described, but the replicater implemented it incorrectly d: other Having the original implementation code is necessary to distinguish between cases a and b versus case c, and if the former, the code for the test harness is likely to help distinguish a versus b. (Case d is of course still a problem, but that doesn't mean it's reasonable to just give up.)
- kkylin 6y ago
  I agree with the case analysis, but disagree with the implication that the code needs to be runnable (which is seems to be the point of the discussion at hand). In many cases having source code, even if it no longer runs, should be sufficient.
  - kkylin 6y ago
    I complete agree with your case analysis, but disagree with the conclusion that the code needs to be runnable for it to be useful -- I thought this was the point of the discussion at hand? In most situations, having source code, even if it no longer runs, would be sufficient to conduct the analysis you describe. I'm all for more transparency, and this includes making codes and data public as much as is reasonable. But the real test is if someone can independently replicate the result, and how to incentivize replication studies (in both computational and experimental science) is also important, and in my view should not be divorced from discussions of reproducibility. Edit: rewrote to clarify my position.

by kkylin 6y ago

a1369209993 6y ago

> whether they can run my code or not doesn't seem to be of the highest priority. On the contrary; in that case there are four possibilities: a: your algorithm doesn't work at all, and your observations are a artifact of convenient inputs or inept measurements. b: your algorithm works, but the description in the paper is wrong or incomplete c: your algorithm works as described, but the replicater implemented it incorrectly d: other Having the original implementation code is necessary to distinguish between cases a and b versus case c, and if the former, the code for the test harness is likely to help distinguish a versus b. (Case d is of course still a problem, but that doesn't mean it's reasonable to just give up.)

kkylin 6y ago
I agree with the case analysis, but disagree with the implication that the code needs to be runnable (which is seems to be the point of the discussion at hand). In many cases having source code, even if it no longer runs, should be sufficient.
- kkylin 6y ago
  I complete agree with your case analysis, but disagree with the conclusion that the code needs to be runnable for it to be useful -- I thought this was the point of the discussion at hand? In most situations, having source code, even if it no longer runs, would be sufficient to conduct the analysis you describe. I'm all for more transparency, and this includes making codes and data public as much as is reasonable. But the real test is if someone can independently replicate the result, and how to incentivize replication studies (in both computational and experimental science) is also important, and in my view should not be divorced from discussions of reproducibility. Edit: rewrote to clarify my position.