3 ms·
Some code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper. Are you sure you have thought th
by p1esk 7y ago
Some code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper.
Are you sure you have thought this through?
- tsbinz 7y ago8 gpus and a week is cheap even, there are popular papers that need upwards of 10k GPU hours.
- pfortuny 7y agoIt is not like, ehem, Elsevier does not have the money to pay for this...
- solidasparagus 7y agoAI papers do not get published by Elsevier
- Qworg 7y agoI have thought it through. There are multiple major papers that are published each year that purport to show an advancement, but fail to demonstrate that advancement under different random parameters. This was a major topic at NIPS 2017, yet doesn't seem to be important enough to address.
- sampo 7y agoWell, it usually takes months to get a paper reviewed anyway, so it's not like there isn't time for that...
- anewhnaccount2 7y agoThe experiments themselves should be conducted on fully logged public infrastructure so that it impossible to miss anything out. This final log would include the inputs: shell commands/git repositories/Dockerfiles, training and testing data; important intermediate data: Docker images and trained models; and outputs: tables of numbers or what have you. This way, the whole thing is vouched for automatically, and the editors don't need to do anything. It would be a big change, but this is actually possible, and it would make things easier for everyone down the line.
- solidasparagus 7y agoWho is going to pay for that?