2 ms·

You're completely wrong. So are most people in the comment section. Research papers should NOT release code. Ever. Researchers don't care, and should not care

You're completely wrong. So are most people in the comment section. Research papers should NOT release code. Ever. Researchers don't care, and should not care about your setup, your IDE, your OS, your dependencies, your processor architecture etc. None of that matters, unless it's the central point of the paper. You literally ask researchers to write coding tutorials or something. It's simply comedic. Most machine learning and CS research revolves around Methods, Architectures, Processes and other abstract approaches to solving a problem. Their algorithms are always reproducible with any implementation you like. Sure, the training data should be supplied, because in AI/learning it's part of the algorithm, but even if sometimes they aren't, you can try and reproduce the relative changes with another similar database. If their point is that one training method works better than another, the result still should be reproducible with similar datasets. I get where you're coming from, after all most people on this site are programmers, but research is not about publishing code that works, it's about publishing ideas that work.

Donald 9y ago
The big issue is the low-quality of machine learning / applied CS literature in general. Pull a paper and try implementing their algorithm(s). You'll be almost certainly guaranteed to encounter barriers ranging from mistakes in the paper, omissions / ambiguities in the paper, to even downright fabrications. The entire field has a reproducibility problem, which implies that as a academic community we're just churning out pubs and not actually doing science. I'm helping form Nature's foray into ML (https://www.nature.com/natmachintell/ https://www.nature.com/natmachintell/), and we're strongly considering enforcing a requirement that FOSS source code be published in a public venue (like github), and requiring that at least some benchmarks are established on publicly (but perhaps not freely) available data sets. This won't be appropriate for every paper of course, but our editorial staff is certainly going to be focused on maintaining scientific quality.
lacker 9y ago
Their algorithms are always reproducible with any implementation you like. Well, that's the dream. In practice their algorithms are often not reproducible at all. Requiring code would get rid of the papers that cannot be made to work, where the authors stretched the truth because without code they could not be caught.

Donald 9y ago

The big issue is the low-quality of machine learning / applied CS literature in general. Pull a paper and try implementing their algorithm(s). You'll be almost certainly guaranteed to encounter barriers ranging from mistakes in the paper, omissions / ambiguities in the paper, to even downright fabrications. The entire field has a reproducibility problem, which implies that as a academic community we're just churning out pubs and not actually doing science. I'm helping form Nature's foray into ML (https://www.nature.com/natmachintell/ https://www.nature.com/natmachintell/), and we're strongly considering enforcing a requirement that FOSS source code be published in a public venue (like github), and requiring that at least some benchmarks are established on publicly (but perhaps not freely) available data sets. This won't be appropriate for every paper of course, but our editorial staff is certainly going to be focused on maintaining scientific quality.

lacker 9y ago

Their algorithms are always reproducible with any implementation you like. Well, that's the dream. In practice their algorithms are often not reproducible at all. Requiring code would get rid of the papers that cannot be made to work, where the authors stretched the truth because without code they could not be caught.