5 ms·
Another conclusion is that scientists gain very little by publishing their code. In research terms, they're evaluated on published papers, receiving grants and
by scott_s 16y ago
Another conclusion is that scientists gain very little by publishing their code. In research terms, they're evaluated on published papers, receiving grants and graduating students. The incentives just aren't there.
- cdavid 16y agoThe article is just really poor, and mostly shows the author's complete lack of knowledge on how science is done, and even what it is about. One the biggest reason for not opening your code is to avoid given an unfair advantage to other researchers: academic research is extremely competitive, and the risk for free ride is significant. I am all for opening tools more, but I don't think reproducibility is the reason for it. In any case, reproducibility does not mean everyone using the same software (with the same bugs) - it means independent experiments.
- ntoshev 16y agoAre there incentives to have a lower/higher barrier for someone else to reproduce your findings and build on them?
- dkarl 16y agoIf they are willing to be honest, they can publish the code in the condition it's in. What's the extra work in uploading a tarball or exporting a single version to a site like github or Sourceforge? The paper (and the data, which should be published as well) provides all the context needed. I suspect the extra work involved in publishing code is because the code is in such bad shape that it would decrease confidence in the work that depends on it. If that's the case, then it's all the more important that researchers establish it as a standard, non-optional part of publishing research.
- scott_s 16y agoAgain, there's no incentive. You're arguing that it's not a burden, but you're not presenting any benefit it has to the researchers. If you want researchers to start doing this, then you need to figure out how to incentivize it.
- dkarl 16y agoSure I am. There's general disrespect in academia for work that is incompletely reported in a way that hinders review and reproduction. It's a historical accident that refusing to publish source code has come to be treated as a special exception to the principles of good scientific practice. The more the basis for that exception is undermined, the more justified researchers will feel in demanding source code when they're skeptical of a result. And here's the key incentive: lack of source code can be used as an excuse for ignoring and devaluing competitors' contributions. Scientists may be reluctant to raise a standard that would result in more work for everybody, but they can't resist a means of denigrating their rivals' work. Once the cat is out of the bag, scientists with weak reputations and disruptive results will be forced to publish source code to prevent their work being round-filed. Established scientists will find it untenable to hold themselves to an explicitly lower standard than everybody else, so they will eventually be forced to start publishing their code, too.
- scott_s 16y agoIt's not a refusal, it's a neglect. What you're saying is, "If the culture was different, the incentives would be there." But the culture is not different. You can't just wave a magic wand. Changing culture is hard.
- dkarl 16y agoI explained how the culture will change. It will change because academics will use failure to release source code to throw doubt on the results of their rivals, and people (especially those who aren't already established in their fields) will answer that criticism by releasing code, which will raise the standards for everybody else. There's disincentive to be first, because everybody will pretend to be shocked at the buggy, undocumented code, but not everyone writes bad code and not everyone will feel able to refuse. You want me to actually change the culture right now in a comment thread on Hacker News? I think predicting how it will happen is the best I can do, sorry.