3 ms·

I think there's not enough researchers that publish code. For example, discrete optimization research (nurse rostering, travelling salesman, vehicle routing pr

I think there's not enough researchers that publish code. For example, discrete optimization research (nurse rostering, travelling salesman, vehicle routing problem, etc.) is filled with papers where people are evaluating their methods on public benchmarks but code never sees the day. There's a lot of state-of-the-art methods that never have their code released. I'm pretty sure it's like that elsewhere. Machine learning and deep learning for some reason has a lot of code in the open but that's not the norm. I'd prefer the code to be open first. Once that's abundant then I might prefer the code to also be well designed.

lou1306 6y ago
> I think there's not enough researchers that publish code. I agree, although lately there's been some effort by academia to make authors publish their code, or at least disclose it to the reviewers. Several conferences have an artifact evaluation committee, which tries to reproduce the experimental part of submitted papers. Some conferences actually require a successful artifact evaluation to be accepted (see, for instance, the tool tracks at CAV [1] and TACAS [2]). Others, while not requiring an artifact evaluation, may encourage it by other means. The ACM, for instance, marks accepted papers with special badges [3] reflecting how well the alleged findings can be reproduced and whether the code is publicly available. [1] http://i-cav.org/2021/artifact-evaluation/ http://i-cav.org/2021/artifact-evaluation/ [2] https://etaps.org/2021/call-for-papers https://etaps.org/2021/call-for-papers [3] https://www.acm.org/publications/policies/artifact-review-and-badging-current https://www.acm.org/publications/policies/artifact-review-an...
- cratermoon 6y ago
  This feels like the right approach. If peer review were to include artifact evaluation, including some kind of code review, and require certain standards be met for acceptance, things would change. As others have noted here, the mechanisms of grant-funded work strongly discourage attention to code quality, and that would have to change as well. I'm not in academia now, but I started out my career doing sysops and programming in a lab at a medical school and have worked with academics a bit since. I don't do it much because it's basically volunteer work, and it's almost impossible to contribute meaningfully unless you are also well-versed in the field.

lou1306 6y ago

> I think there's not enough researchers that publish code. I agree, although lately there's been some effort by academia to make authors publish their code, or at least disclose it to the reviewers. Several conferences have an artifact evaluation committee, which tries to reproduce the experimental part of submitted papers. Some conferences actually require a successful artifact evaluation to be accepted (see, for instance, the tool tracks at CAV [1] and TACAS [2]). Others, while not requiring an artifact evaluation, may encourage it by other means. The ACM, for instance, marks accepted papers with special badges [3] reflecting how well the alleged findings can be reproduced and whether the code is publicly available. [1] http://i-cav.org/2021/artifact-evaluation/ http://i-cav.org/2021/artifact-evaluation/ [2] https://etaps.org/2021/call-for-papers https://etaps.org/2021/call-for-papers [3] https://www.acm.org/publications/policies/artifact-review-and-badging-current https://www.acm.org/publications/policies/artifact-review-an...

cratermoon 6y ago
This feels like the right approach. If peer review were to include artifact evaluation, including some kind of code review, and require certain standards be met for acceptance, things would change. As others have noted here, the mechanisms of grant-funded work strongly discourage attention to code quality, and that would have to change as well. I'm not in academia now, but I started out my career doing sysops and programming in a lab at a medical school and have worked with academics a bit since. I don't do it much because it's basically volunteer work, and it's almost impossible to contribute meaningfully unless you are also well-versed in the field.