3 ms·
regarding 1, even papers with high quality published code fall prey to bit rot very quickly -- go dig through some git repos from ML papers from 2015 and try to
by currymj 7y ago
regarding 1, even papers with high quality published code fall prey to bit rot very quickly -- go dig through some git repos from ML papers from 2015 and try to run them.
distributing Docker containers works better but even that won't necessarily help, because especially if you have to use GPUs the container software has been shifting underneath you. plus it's still cumbersome to make a fair comparison between containerized code and a new approach implemented somewhere else, especially if it's on different data.
Would it be worth making everyone expend a huge amount of unpaid additional effort maintaining the code for every ML paper ever published, for years to come? I'm not sure -- many are not very good and should probably be left to die. On the other hand it's common and immensely frustrating to be unable to reproduce results.
regarding 2, it seems like if the technique itself works in a substantially different way from the state of the art, even if it can't consistently beat it, that's still a positive result -- something new.
- amelius 7y agoWhat if, instead of publishing just the code, authors would publish intermediate results (e.g. tensors) of running the code? So basically sprinkle the code with print statements. Then if a future researcher has difficulty reproducing the result, they know exactly at which step the problem occurs.
- newen 7y agoHaha that's not going to work for typical models that have millions of parameters. Besides, it's not the code that's important (except for verifying that it's a good method) it's the actual method, it's description, why it works, how it works, how it can be improved, adpated, etc.