4 ms·
Are the 30% of papers including training data actually hosting the data set somewhere for download, or are they referencing a public data set elsewhere? Both ap
by jackcosgrove 8y ago
Are the 30% of papers including training data actually hosting the data set somewhere for download, or are they referencing a public data set elsewhere? Both approaches are acceptable in my book.
If a study uses a private data set, or one that the researcher controls and only gives out to approved partners, that study should be discounted.
I understand corporate labs cannot give away their data in many cases, but corporate research carries less authority than academic research anyways. Academic research should always make the data set publicly available.
- 8note 8y agoI think if you're going to do a replication study, you should collect a different data set about the same subject, and use the same method, then see if you get the same results.
- mamon 8y agoYes, simply compiling the provided code and running it against provided data set does not really count as replication. Writing your own code and creating your own dataset is the simplest way to rule out the situations where there is something fishy in either the original code or data. And the paper itself should contain enough details to make it possible to recreate the experiment this way.
- spc476 8y agoI think compiling the provided code and running it against the provided data set does do something---you know the code reports on the data. If you get a different result with the provided code and data, then there's something different with your environment vs. the original researcher, perhaps the rounding mode, or some assumption [1]. Once that's straightened out, then you can use new data and see if you can replicate the result with the provided code and new data. Think of the provided data as a sanity test. [1] I recently fixed a bug wherein I was inadvertently relying upon Linux-specific behavior that failed when tested under Solaris.