4 ms·

> I think there's an argument, currently, for telling people "I could give you the code - but it'll only take you a couple of days to write your own code to rep

by rflrob 15y ago

> I think there's an argument, currently, for telling people "I could give you the code - but it'll only take you a couple of days to write your own code to replicate the results, and it'd be much better if you could do that" - but I'm sure this varies drastically across domains. While I can definitely see a PI setting this kind of task to a student, I don't think that by releasing the code the task suddenly becomes impossible. If (and when) the two independently written programs disagree, it will then be possible to immediately step through each and figure out where the point of divergence happens, and which (if either) implementation is more likely correct, rather than then starting an email chain with the original authors saying, "we get different results, but we don't know why".

feral 15y ago
If people continued to write independent programs, purely to replicate and verify claimed results, then certainly it'd be beneficial for them to have the source, in order to track down where the discrepancy arises. This is often how it works currently, in practice, in my limited experience; you mail the authors and ask them to help you track down the problem, and get either code, or support; but I acknowledge this wont always work. My concerns are that 1) I don't trust people to do the hard thing, and re-implement to replicate, rather than take the easier way, and just use the code that's provided. There's very little credit currently given for replicating existing (even recent) results. 2) More importantly, when replicating a paper, thats a little vague, it'll be more and more tempting to just peek at the source; and suddenly the paper isn't the document of record anymore. I feel that if we go down the route where the code becomes the detailed documentation of the scientific process, that's a very fundamental shift from the current model, where the paper is supposed to be repeatable, in and of itself. If we go down that road, we probably need a whole different review infrastructure; are reviewers really going to spend the time to review large and hastily written scientific codebases? I doubt it; so how does review work when: "The code is the only definitive expression of the data-processing methods used: without the code, readers cannot fully consider, criticize, or improve upon the methods." Will it no longer be possible to criticise a paper for lacking sufficient detail to reproduce the results? Will the reply be 'read the source' ? Maybe that's just the way things are going to go. There's a lot to like in that manifesto. But there's going to positive and negatives to letting the source become the documentation. The discussion around the manifesto on their website does not acknowledge such tradeoffs; its taking a pretty one-sided view. Maybe that's just how you are supposed to write manifestos :-) But I'd like to see some discussion of these tradeoffs.

by rflrob 15y ago

feral 15y ago

If people continued to write independent programs, purely to replicate and verify claimed results, then certainly it'd be beneficial for them to have the source, in order to track down where the discrepancy arises. This is often how it works currently, in practice, in my limited experience; you mail the authors and ask them to help you track down the problem, and get either code, or support; but I acknowledge this wont always work. My concerns are that 1) I don't trust people to do the hard thing, and re-implement to replicate, rather than take the easier way, and just use the code that's provided. There's very little credit currently given for replicating existing (even recent) results. 2) More importantly, when replicating a paper, thats a little vague, it'll be more and more tempting to just peek at the source; and suddenly the paper isn't the document of record anymore. I feel that if we go down the route where the code becomes the detailed documentation of the scientific process, that's a very fundamental shift from the current model, where the paper is supposed to be repeatable, in and of itself. If we go down that road, we probably need a whole different review infrastructure; are reviewers really going to spend the time to review large and hastily written scientific codebases? I doubt it; so how does review work when: "The code is the only definitive expression of the data-processing methods used: without the code, readers cannot fully consider, criticize, or improve upon the methods." Will it no longer be possible to criticise a paper for lacking sufficient detail to reproduce the results? Will the reply be 'read the source' ? Maybe that's just the way things are going to go. There's a lot to like in that manifesto. But there's going to positive and negatives to letting the source become the documentation. The discussion around the manifesto on their website does not acknowledge such tradeoffs; its taking a pretty one-sided view. Maybe that's just how you are supposed to write manifestos :-) But I'd like to see some discussion of these tradeoffs.