3 ms·
Yeah, but we'd expect papers that predominantly run code to be much, much better than this. Like, there is definitely a set of code that produced the results, a
by disgruntledphd2 7y ago
Yeah, but we'd expect papers that predominantly run code to be much, much better than this. Like, there is definitely a set of code that produced the results, and presumably a repository somewhere with some of the scripts that did this. Supplying this to the world is not too much to ask (and should 100% be a requirement of funding agencies (which would work as long as users of data and code were also required to cite the original article (isn't nesting parentheses fun?))).
- TallGuyShort 7y agoI agree, but all of that is also true of a lot of papers in a lot of other fields. I've seen papers about better complex simulations of natural processes - I don't think I've ever seen one with complete code. I've seen papers that described the experiment and the results of their analysis with hand-wavy descriptions of which algorithms and statistical methods they used - but no code. My point is that the findings of this article are not necessarily indicative of machine learning being junk science, but it's more likely indicative of a systemic problem with all research, where there's incentive to publish the paper, but not the data, code, models, or even generally enough information of any kind to successfully replicate the experiment more often than 10-15% of the time. I remember when LIGO first detected a black-hole collision, they released the raw data and the Jupyter notebooks that were actually used to go from the raw data to the published results. Everyone was floored. It shouldn't be that amazing - it should be the standard. For every field.