2 ms·
> Re-running their crappy code and getting the same result they got doesn't really prove or verify anything. Yes it does. Very often, the data selected for pu
by beagle3 14y ago
> Re-running their crappy code and getting the same result they got doesn't really prove or verify anything.
Yes it does.
Very often, the data selected for publication is cherry-picked. Running the same crappy code on a more complete data set, (or alternatively, on a partial data set) would give a very quick indication of the robustness of the results - and unlike re-implementing, might be doable in a day rather than months of effort.
Furthermore, when you actually re-implement (if you do), it is extremely helpful to compare intermediate results, which is impossible unless you have the original everything.
> Re-implementing the algorithm they describe in the paper and getting the same result (or not) is far more interesting.
Yes, but very rarely done in fields that are not CS or EE (and not very common in these either). Usually, results are just taken as gospel.
Also, there is a ridiculous amount of negligence (and even fraud) in publications. just running the crappy code, seeing the results, and having a cursory look at the code and data would reveal a lot of that.