4 ms·
What nonsense! Copying something is reproducing it. If it cannot be copied then it cannot be reproduced. If it can be copied then it can be investigated. I ran
by worik 6y ago
What nonsense!
Copying something is reproducing it.
If it cannot be copied then it cannot be reproduced.
If it can be copied then it can be investigated.
I ran into exactly this problem with:
White's Reality Check (WRC), described in White's (2000) Econometrica paper titled "A Reality Check for Data Snooping",
The algorithms that were being tested were so poorly described that less than half could be reproduced.
Once we got there for what we could we found two fundamental problems with "Whites Reality Check", one theoretical that is blindingly obvious once you see it and the other that they ignored trading costs. When we measured the levels of trading costs that would permit the algorithms to work it was clear they could not.
That we could not deduce without reproducing the results. If we could have copied the original code it would have shaved about eight weeks from my research.
BTW "Whites Reality Check" was at the time pone of the most cited paper in its, rather narrow, field.
- projektfu 6y agoBut running their code, presumably, would get their result on their data, unless it was a fraud or weird misconfiguration. We’re not talking about fraud, we’re talking about reproducibility supporting a theory. You suspected the result was wrong, so your interest was in debugging their code to contradict their result. That has value but also is not a reproduction of the study.