3 ms·

There are plenty of valid reasons to be critical / skeptical of the data science / machine learning bubble, but the last few articles I've read about the resear

There are plenty of valid reasons to be critical / skeptical of the data science / machine learning bubble, but the last few articles I've read about the research actually makes it sound not-too-bad. 7/18 releasing enough code to reproduce sure beats the rate at which other fields release code, even when the code is still critical to confirming the results. In fact it's almost more understandable since the code is a larger portion of the intellectual property that might be valuable to whoever is funding the research, if it's a private entity, or university staff who might still want to monetize the research. I saw another one recently about reproducibility, something like 85%. Which sounds terrible - until you remember that something like 90% [1] of science papers weren't successfully reproduced in an attempt to do so. I'm gonna go dig up citations if I can find them quickly... [1] 1 quick citation before I get back to work: https://www.bewellbuzz.com/technology/many-published-studies-complete-bs/ https://www.bewellbuzz.com/technology/many-published-studies... It's terrible, but no more terrible than many other comparable fields.

disgruntledphd2 7y ago
Yeah, but we'd expect papers that predominantly run code to be much, much better than this. Like, there is definitely a set of code that produced the results, and presumably a repository somewhere with some of the scripts that did this. Supplying this to the world is not too much to ask (and should 100% be a requirement of funding agencies (which would work as long as users of data and code were also required to cite the original article (isn't nesting parentheses fun?))).
- TallGuyShort 7y ago
  I agree, but all of that is also true of a lot of papers in a lot of other fields. I've seen papers about better complex simulations of natural processes - I don't think I've ever seen one with complete code. I've seen papers that described the experiment and the results of their analysis with hand-wavy descriptions of which algorithms and statistical methods they used - but no code. My point is that the findings of this article are not necessarily indicative of machine learning being junk science, but it's more likely indicative of a systemic problem with all research, where there's incentive to publish the paper, but not the data, code, models, or even generally enough information of any kind to successfully replicate the experiment more often than 10-15% of the time. I remember when LIGO first detected a black-hole collision, they released the raw data and the Jupyter notebooks that were actually used to go from the raw data to the published results. Everyone was floored. It shouldn't be that amazing - it should be the standard. For every field.

disgruntledphd2 7y ago

Yeah, but we'd expect papers that predominantly run code to be much, much better than this. Like, there is definitely a set of code that produced the results, and presumably a repository somewhere with some of the scripts that did this. Supplying this to the world is not too much to ask (and should 100% be a requirement of funding agencies (which would work as long as users of data and code were also required to cite the original article (isn't nesting parentheses fun?))).

TallGuyShort 7y ago
I agree, but all of that is also true of a lot of papers in a lot of other fields. I've seen papers about better complex simulations of natural processes - I don't think I've ever seen one with complete code. I've seen papers that described the experiment and the results of their analysis with hand-wavy descriptions of which algorithms and statistical methods they used - but no code. My point is that the findings of this article are not necessarily indicative of machine learning being junk science, but it's more likely indicative of a systemic problem with all research, where there's incentive to publish the paper, but not the data, code, models, or even generally enough information of any kind to successfully replicate the experiment more often than 10-15% of the time. I remember when LIGO first detected a black-hole collision, they released the raw data and the Jupyter notebooks that were actually used to go from the raw data to the published results. Everyone was floored. It shouldn't be that amazing - it should be the standard. For every field.