3 ms·
The vast majority of such papers don't really involve a claim that they wrote a program that does X. I mean, they probably did write such a program, but that's
by PeterisP 9y ago
The vast majority of such papers don't really involve a claim that they wrote a program that does X. I mean, they probably did write such a program, but that's tangential to the paper and not particularly relevant; the fact that they did isn't one of claims in the paper, it's not something they're trying to prove/show in the conclusions. There are some papers in the style "Software package for doing X", but those are a minority and generally limited to major releases of widely used open source tools for that problem; most CS papers are not about software, just like most astronomy papers are not about telescopes.
E.g. the paper is about some new method to approach X. In general, the paper could be valid even without any program implementation whatsoever; but you'll likely might supplement the method description with some experimental evaluation - but the paper is not about the experimental evaluation or the "experimental apparatus" i.e. the code they used. I mean, the paper is likely making some claims about the method as such, not about any single particular implementation of the method including their own. As a part of the main claim "this method seems good and interesting" they're providing some evidence "we tried, and in certain conditions described here applying method X was 10% better than method Y" - but the actual code used to do that is just supplementary material; the code is a tool they used in research, not the result of the research they published. The code is the "telescope" you used to make an observation, but the paper is about the observation, not about the telescope.
It's just as with medicine - we generally accept papers that say "well, we tried this procedure on 100 patients and 73 of them got better" without requiring video evidence of those 73 patients actually getting better; in the same manner, we accept CS papers that say "well, we tried this procedure on 100 datapoints and 73 of them got correct results" and don't require the reviewers to reproduce the experiments and verify if the experimental results aren't falsified; just as reviewers don't try to reproduce experiments in pretty much any other science.
- infogulch 9y agoThe difference is that when a medical paper claims > well, we tried this procedure on 100 patients "this procedure" is the most important part, and they describe it in detail in the paper, hopefully well enough that someone could attempt to replicate it, and some do attempt to replicate it. (Not that procedure descriptions in such papers are always sufficient for this.) The difference between that and > we tried this procedure on 100 datapoints Is that it's nigh impossible to describe a ML procedure in enough detail to reproduce with just the description in the paper. Tiny changes in the parameters and construction can completely change the result; the only way to be able to reproduce it is if you had the source code. And also the source data which is just if not more important as the source code (see sibling thread). The opportunity that academic CS has over every other science is that they could empower every reader with the capability to verify the results of every paper they read, and this is actually attainable. Reviewers of other sciences don't reproduce findings themselves for purely practical reasons that don't need to exist in CS.
- joepvd 9y agoTo try out a procedure on one dataset amounts to a data point. You need to process a bunch of datasets to establish the performance of the procedure under study. I though ML folks would have the statistical background to know you cannot infer a true statement from a single occurrence?
- lacker 9y agoHmm, it probably depends on your subfield of computer science. These AI papers almost always claim they wrote a program that does X, and so did the stuff I was working on in grad school, but more theoretical things probably are more similar to what you describe.