4 ms·

I was more struck by this part: > How many Nature papers are not reproducible? To call out a new bunch of people who are trying something different because peo

I was more struck by this part: > How many Nature papers are not reproducible? To call out a new bunch of people who are trying something different because people are afraid they’ll fail is sort of hypocritical given that we don’t really check whether anybody else’s results are correct, either. Peer review is more about politics then reproducibility. Reproducibility is really important, but Nature doesn't seem to agree.

ihnorton 13y ago
This comes up a lot, and there is truth to it. But my flippant response is usually "the ones that aren't that important". In just the past two months we've seen examples of the system working well for both reproduction (CRISPR) and invalidation (acid bath-induced pluripotency) of spectacular results.
- tdaltonc 13y ago
  Those are victories, but I wouldn't credit them to peer review. I think of "peer review" as what happens between a manuscript being submitted and it being published.
collyw 13y ago
I have a theory with peer review, if it works, it is important and it is reproducible, then people will reproduce it and build on it. If any of the three parts are not there, then no one will want / need / be able to reproduce it, and it will fade into scientific obscurity, while the rest of the scientific community keep moving forwards.
- omnisci 13y ago
  Or people spend a lot of time and money trying to chase down something that a ground "found" and published. It's very common to read a paper, think it sounds good, then try build on it only to find that you can't. Peer review is completely flawed. Until it's made more transparent, like at F1000, it's going to continue that way. Not being a negative Nancy here, but I've been on both sides of the fence (reviewed and had grants/papers reviewed) and it isn't an efficient system. Much like many other things in science, it's an antiquated process that needs to be overhauled to get back to it's "roots".
  - collyw 13y ago
    Peer review is not perfect, but its still better than the way the mainstream media reports the news.
    - omnisci 13y ago
      That is 100% true, but that isn't a fair comparison. Peer review should be open, scientists shouldn't be allowed to choose who reviews/doesn't review their papers, and the process by which selection is done needs to be more rigorous. My suggestion, open it up to everyone in the field and make sure that the reviewers are identified. Anonymous peer review has the same effect that anonymous posting on internet forums has, people become e-thugs or just say things that they shouldn't ever say because they know the reader doesn't know who they are. This sounds like it doesn't happen, but you would be surprised that the stuff that happens behind closed doors. Scientists are people, people are flawed (which is fine), but that shouldn't hold back scientific discoveries.
  - collyw 13y ago
    "Or people spend a lot of time and money trying to chase down something that a ground "found" and published." Then again, you have the problem that negative results are rarely published. Many groups may repeat the same thing over and over, not realizing that many other groups have confirmed a negative result. (Actually I think I saw a post here about a journal that aims to adress this problem).

ihnorton 13y ago

This comes up a lot, and there is truth to it. But my flippant response is usually "the ones that aren't that important". In just the past two months we've seen examples of the system working well for both reproduction (CRISPR) and invalidation (acid bath-induced pluripotency) of spectacular results.

tdaltonc 13y ago
Those are victories, but I wouldn't credit them to peer review. I think of "peer review" as what happens between a manuscript being submitted and it being published.

collyw 13y ago

I have a theory with peer review, if it works, it is important and it is reproducible, then people will reproduce it and build on it. If any of the three parts are not there, then no one will want / need / be able to reproduce it, and it will fade into scientific obscurity, while the rest of the scientific community keep moving forwards.

omnisci 13y ago
Or people spend a lot of time and money trying to chase down something that a ground "found" and published. It's very common to read a paper, think it sounds good, then try build on it only to find that you can't. Peer review is completely flawed. Until it's made more transparent, like at F1000, it's going to continue that way. Not being a negative Nancy here, but I've been on both sides of the fence (reviewed and had grants/papers reviewed) and it isn't an efficient system. Much like many other things in science, it's an antiquated process that needs to be overhauled to get back to it's "roots".
- collyw 13y ago
  Peer review is not perfect, but its still better than the way the mainstream media reports the news.
  - omnisci 13y ago
    That is 100% true, but that isn't a fair comparison. Peer review should be open, scientists shouldn't be allowed to choose who reviews/doesn't review their papers, and the process by which selection is done needs to be more rigorous. My suggestion, open it up to everyone in the field and make sure that the reviewers are identified. Anonymous peer review has the same effect that anonymous posting on internet forums has, people become e-thugs or just say things that they shouldn't ever say because they know the reader doesn't know who they are. This sounds like it doesn't happen, but you would be surprised that the stuff that happens behind closed doors. Scientists are people, people are flawed (which is fine), but that shouldn't hold back scientific discoveries.
- collyw 13y ago
  "Or people spend a lot of time and money trying to chase down something that a ground "found" and published." Then again, you have the problem that negative results are rarely published. Many groups may repeat the same thing over and over, not realizing that many other groups have confirmed a negative result. (Actually I think I saw a post here about a journal that aims to adress this problem).