3 ms·
> if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve
by throwawayjava 7y ago
> if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results.
This is the actually important definition of "reproducible".
"has an install.sh" is really nice, but it's far more important that an informed reader can recreate the artifact for themselves from the written description.
The "must be able to apt install or it's not real science" is a particularly dangerous path to go down. It's how you end up with e.g. shit loads of well-engineered LISP or FORTRAN code with no actual scientific insights or knowledge transfer. Which has actually happened in the past. A lot.
- telchar 7y agoI don't think people in this thread are insisting on a downloadable package. Just that the algorithm published in the write-up accurately describe the work that was actually done to achieve the stated results. If even that threshold can't be met, IMO the authors are polluting the literature with bad work.