3 ms·
> I think there's not enough researchers that publish code. I agree, although lately there's been some effort by academia to make authors publish their code, o
by lou1306 6y ago
> I think there's not enough researchers that publish code.
I agree, although lately there's been some effort by academia to make authors publish their code, or at least disclose it to the reviewers.
Several conferences have an artifact evaluation committee, which tries to reproduce the experimental part of submitted papers. Some conferences actually require a successful artifact evaluation to be accepted (see, for instance, the tool tracks at CAV [1] and TACAS [2]).
Others, while not requiring an artifact evaluation, may encourage it by other means.
The ACM, for instance, marks accepted papers with special badges [3] reflecting how well the alleged findings can be reproduced and whether the code is publicly available.
[1] http://i-cav.org/2021/artifact-evaluation/ http://i-cav.org/2021/artifact-evaluation/
[2] https://etaps.org/2021/call-for-papers https://etaps.org/2021/call-for-papers
[3] https://www.acm.org/publications/policies/artifact-review-and-badging-current https://www.acm.org/publications/policies/artifact-review-an...
- cratermoon 6y agoThis feels like the right approach. If peer review were to include artifact evaluation, including some kind of code review, and require certain standards be met for acceptance, things would change. As others have noted here, the mechanisms of grant-funded work strongly discourage attention to code quality, and that would have to change as well. I'm not in academia now, but I started out my career doing sysops and programming in a lab at a medical school and have worked with academics a bit since. I don't do it much because it's basically volunteer work, and it's almost impossible to contribute meaningfully unless you are also well-versed in the field.