4 ms·

If you run their code on their data you haven't reproduced their result. You have merely copied it. If their code works on another very similar data set to pr

If you run their code on their data you haven't reproduced their result. You have merely copied it. If their code works on another very similar data set to produce a very similar result, then you have something. Or if you're trying to understand the technique, you have to write your code and see if it reproduces their result on their data.

bordercases 6y ago
> If you run their code on their data you haven't reproduced their result. You have merely copied it. That's like saying you can't learn from exercises in the book if the answers are in the back.
- canjobear 6y ago
  You certainly don't learn anything by running `sh run.sh`.
  - asdff 6y ago
    you learn run.sh actually works
  - bordercases 6y ago
    You learn if it's deterministic or not under the data given as input and the operating environment.
worik 6y ago
What nonsense! Copying something is reproducing it. If it cannot be copied then it cannot be reproduced. If it can be copied then it can be investigated. I ran into exactly this problem with: White's Reality Check (WRC), described in White's (2000) Econometrica paper titled "A Reality Check for Data Snooping", The algorithms that were being tested were so poorly described that less than half could be reproduced. Once we got there for what we could we found two fundamental problems with "Whites Reality Check", one theoretical that is blindingly obvious once you see it and the other that they ignored trading costs. When we measured the levels of trading costs that would permit the algorithms to work it was clear they could not. That we could not deduce without reproducing the results. If we could have copied the original code it would have shaved about eight weeks from my research. BTW "Whites Reality Check" was at the time pone of the most cited paper in its, rather narrow, field.
- projektfu 6y ago
  But running their code, presumably, would get their result on their data, unless it was a fraud or weird misconfiguration. We’re not talking about fraud, we’re talking about reproducibility supporting a theory. You suspected the result was wrong, so your interest was in debugging their code to contradict their result. That has value but also is not a reproduction of the study.

bordercases 6y ago

> If you run their code on their data you haven't reproduced their result. You have merely copied it. That's like saying you can't learn from exercises in the book if the answers are in the back.

canjobear 6y ago
You certainly don't learn anything by running `sh run.sh`.
- asdff 6y ago
  you learn run.sh actually works
- bordercases 6y ago
  You learn if it's deterministic or not under the data given as input and the operating environment.

worik 6y ago

What nonsense! Copying something is reproducing it. If it cannot be copied then it cannot be reproduced. If it can be copied then it can be investigated. I ran into exactly this problem with: White's Reality Check (WRC), described in White's (2000) Econometrica paper titled "A Reality Check for Data Snooping", The algorithms that were being tested were so poorly described that less than half could be reproduced. Once we got there for what we could we found two fundamental problems with "Whites Reality Check", one theoretical that is blindingly obvious once you see it and the other that they ignored trading costs. When we measured the levels of trading costs that would permit the algorithms to work it was clear they could not. That we could not deduce without reproducing the results. If we could have copied the original code it would have shaved about eight weeks from my research. BTW "Whites Reality Check" was at the time pone of the most cited paper in its, rather narrow, field.

projektfu 6y ago
But running their code, presumably, would get their result on their data, unless it was a fraud or weird misconfiguration. We’re not talking about fraud, we’re talking about reproducibility supporting a theory. You suspected the result was wrong, so your interest was in debugging their code to contradict their result. That has value but also is not a reproduction of the study.