3 ms·

I don't think that analogy is at all accurate and I think the conclusion that you reach from it is completely incorrect.

by tgb 10y ago

ZeroGravitas 10y ago
I think I must have expressed myself poorly, as I think my conclusion is the same as the article suggests i.e. that the science/code shouldn't be considered "done", until it's been peer-reviewed, since it's easy to fool yourself and others if you're not actually reviewing and testing your code. What did you take away from my comment?
- gcp 10y ago
  Peer review does not imply reproducibility, and it's the latter that is the problem. I can confirm, as a reviewer, that your methodology and analysis looks sensible, but the flaws may be deeper, and the fact that you didn't publish the 19 other studies that failed, but that this is the "lucky one", or that you simply cherry picked the data, is not something I can see as a reviewer. This is especially true if the experiment is nontrivial to re-do.
  - ZeroGravitas 10y ago
    " Peer review does not imply reproducibility, and it's the latter that is the problem." I think this is the key to it, I'm suggesting that reproducibility should be part of considering something peer-reviewed, but of course as currently practised, that isn't true. Of course in a software metaphor, that would probably cover both code review and QA, which is sometimes done by a different job role which further muddies the water.
    - pvaldes 10y ago
      This would be like expecting a car brand to open its code to its competitors before to release a new product. Will lead typically also to the peers rushing to publish the same discovering on disguise before the original.
- tgb 10y ago
  After reading your explanations elsewhere, I take back my statements. Your statements are literally correct, although easy to take incorrectly, and I support them: in particular you seem to be supporting that 100% of researchers should attempt to replicate studies and do so sufficiently that all of them will eventually fail to replicate at least some studies. I think most of us took this to mean that you thought every study should fail to replicate (by analogy that every software has at least some bugs), but I see now your intent and that your original wording backed that up.

I don't think that analogy is at all accurate and I think the conclusion that you reach from it is completely incorrect.

by tgb 10y ago

I don't think that analogy is at all accurate and I think the conclusion that you reach from it is completely incorrect.

ZeroGravitas 10y ago

I think I must have expressed myself poorly, as I think my conclusion is the same as the article suggests i.e. that the science/code shouldn't be considered "done", until it's been peer-reviewed, since it's easy to fool yourself and others if you're not actually reviewing and testing your code. What did you take away from my comment?

gcp 10y ago
Peer review does not imply reproducibility, and it's the latter that is the problem. I can confirm, as a reviewer, that your methodology and analysis looks sensible, but the flaws may be deeper, and the fact that you didn't publish the 19 other studies that failed, but that this is the "lucky one", or that you simply cherry picked the data, is not something I can see as a reviewer. This is especially true if the experiment is nontrivial to re-do.
- ZeroGravitas 10y ago
  " Peer review does not imply reproducibility, and it's the latter that is the problem." I think this is the key to it, I'm suggesting that reproducibility should be part of considering something peer-reviewed, but of course as currently practised, that isn't true. Of course in a software metaphor, that would probably cover both code review and QA, which is sometimes done by a different job role which further muddies the water.
  - pvaldes 10y ago
    This would be like expecting a car brand to open its code to its competitors before to release a new product. Will lead typically also to the peers rushing to publish the same discovering on disguise before the original.
tgb 10y ago
After reading your explanations elsewhere, I take back my statements. Your statements are literally correct, although easy to take incorrectly, and I support them: in particular you seem to be supporting that 100% of researchers should attempt to replicate studies and do so sufficiently that all of them will eventually fail to replicate at least some studies. I think most of us took this to mean that you thought every study should fail to replicate (by analogy that every software has at least some bugs), but I see now your intent and that your original wording backed that up.