8 ms·
Editors gotta be more rigorous and only accept papers with completely reproducible portable examples, i.e., docker images, literate code and source code repos.
by bayareanative 7y ago
Editors gotta be more rigorous and only accept papers with completely reproducible portable examples, i.e., docker images, literate code and source code repos. Pseudocode is helpful to be platform neutral, but if it's not precise enough to implemented as code, then it's still a proprietary figment of someone else's imagination akin to the squishy social sciences where almost anything goes, not rigorously reproducible science. Keep the standards high or the quality will taper off.
PS: Sometimes I think some researchers think they're helping themselves keep their research proprietary so they will able to monetize their special knowledge or implementation, especially if no one else can make it work ("knowledge" (job) security/silo). Why do the hard work of figuring out how to make a novel AI/ML algorithm if it can be readily commercially monetized without recompense? (Modern Western civ doesn't have a good patronage system to uniformly support arts, trades and sciences.)
- opportune 7y agoIn some fields, like in ML/AI or in other data-sciencey fields, keeping your code / training data closed prevents other researchers from building or improving on your work. It's more than just monetization, in that case it's just tragedy-of-the-commons career growth
- duckmysick 7y ago> keeping your code / training data closed prevents other researchers from building or improving on your work. I might be naive in this regard, but isn't that the main point of doing research?
- dekhn 7y agoSure, but the challenge is, if you're in a competitive field and you make it easy for your competitors, you're going to have a harder time establishing job security. Currently many fields like biology, CS, ML, physics have fewer well-paying job positions than contenders, so every competitive advantage makes a difference. I know plenty of professors who got there by carefully writing papers that lacked specific details on how to reproduce a cutting-edge, "hot" paper.
- opportune 7y agoIn theory, yes. The actual main point of doing research, from the researcher's point of view, is to advance their career. Typically this means posting cutting-edge results in high impact journals using novel methods. Relinquishing control over crucial details allowing a researcher to continue publishing high-impact papers would increase the competition for the researcher in that academic space, making it harder for that individual to advance their career.
- duckmysick 7y agoHow do the sources of funding (R&D divisions, universities, governments) fit in this picture? Do they have access to the "secret sauce" or do they have to pay extra consultation fees on top of research grants.
- p1esk 7y agoAnd who exactly is supposed to verify that code is reproducible for all the papers that get accepted?
- Qworg 7y agoThe editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.
- p1esk 7y agoSome code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper. Are you sure you have thought this through?
- tsbinz 7y ago8 gpus and a week is cheap even, there are popular papers that need upwards of 10k GPU hours.
- pfortuny 7y agoIt is not like, ehem, Elsevier does not have the money to pay for this...
- solidasparagus 7y agoAI papers do not get published by Elsevier
- Qworg 7y agoI have thought it through. There are multiple major papers that are published each year that purport to show an advancement, but fail to demonstrate that advancement under different random parameters. This was a major topic at NIPS 2017, yet doesn't seem to be important enough to address.
- sampo 7y ago
- gdy 7y ago"Sometimes I think some researchers think they're helping themselves keep their research proprietary so they will able to monetize their special knowledge or implementation, especially if no one else can make it work ("knowledge" (job) security/silo)." My very limited experience says it is true. While our team worked for a commercial company in a certain highly competetive industry, I've both read many papers describing algos that on a closer look couldn't work as described and myself was forbidden to publish anything allowing our competitors or potential buyers to reproduce our algorithms. Our team ended up submitting a paper that was turned down by a reviewer and called an advertisement for the lack of details. None of us wanted to 'fix' that by adding fake details as in the papers we had read, so we ended up not publishing anything while working on the projects in that industry.