4 ms·

> That’s because we’ve basically reinterpreted what “peer review” is. Who is "we" in this scenario? Because that's certainly not how I've seen peer review work

> That’s because we’ve basically reinterpreted what “peer review” is. Who is "we" in this scenario? Because that's certainly not how I've seen peer review work. The editor would ask a small group of people in the field to act as reviewers and then send them the papers. They review it and send it back with any requests for changes prior to publication. So they're the peers that are reviewing, not the editor.

pessimizer 1y ago
I think they meant "reinterpreted" over the last century, not over the span of your personal experience and career.
- olddustytrail 1y ago
  They're saying it changed to not being reviewed properly and I'm saying from recent experience that it is.
godelski 1y ago
Look at the history of Peer Review. What you see post 1950 is pretty different than what you see prior to that. I think this quote is the best one-liner, though I think everyone should dig much more into the question > in the early 20th century, "the burden of proof was generally on the opponents rather than the proponents of new ideas. That is, the reviewers had a higher burden than the authors. The bias is towards acceptance rather than rejection. In a perfect world we could only accept good papers and could reject bad papers, but we don't live in that world. So the question is "when we fail, which way do we want to fail?" Obviously, I'm on the side of Blackstone here https://en.wikipedia.org/wiki/Scholarly_peer_review#History https://en.wikipedia.org/wiki/Scholarly_peer_review#History
- olddustytrail 1y ago
  But that's the opposite of what the person I'm replying to said. They're saying everything is acceptable and I'm saying it's actually reviewed. Ok, maybe that's not what you meant. Peer review doesn't reject papers because they don't agree with the orthodoxy; they reject them because they're not competent. Is that what you were getting at?
  - godelski 1y ago
    I'm a bit confused. What you described is what happens today. Yes. That has been my experience too, serving as a reviewer. I understood xondono to be referencing the history that I mentioned, which is where these reviewers didn't exist. So the requirement was different, which is what I'm saying about the burden of proof. > Peer review doesn't reject papers because they don't agree with the orthodoxy; they reject them because they're not competent. This is absolutely false and I don't know a single academic who hasn't seen competent papers get rejected. Reviewers can reject for any reason. The system is built on trust, but incentivized to reject. "Not competent" is too vague of a term, just like "not novel." In my other comment[0] I even reference one of the famous works that got rejected 3 times for being "not competent". This isn't a one-off case here, it is a common occurrence. On several occasions I've had to champion papers which were clearly competent yet my fellow reviewers simply were not familiar with the domain (they admitted this during discussion). I've also killed papers for similar reasons (a very rare event as I strongly bias towards accepting). So I'm sorry, saying papers are only rejected because they are "not competent" is incredibly naive. And I'm sorry, but the claim that "works aren't rejected because they don't agree with orthodoxy" is simply laughable. There's a long history of peers rejecting discoveries that upset the norms. This has happened to the majority of well known scientists. I'm not talking about like the Church going after Galileo, I'm talking about things like Galilelo arguing with Tycho Brahe or Christoph Scheiner. Einstein was critical of Bohr. Hertz was critical of Bell. The list goes on and on. The criticism was explicitly about running counter to orthodoxy. This is such a common thread in history that there's even Max Planck stated "Science advances one funeral at a time." [0] https://news.ycombinator.com/item?id=44587535 https://news.ycombinator.com/item?id=44587535
    - 542354234235 1y ago
      But wouldn’t a consequence of failing to err on the side of caution and “orthodoxy” mean a proliferation of junk and pseudoscience? We already had that problem now but the change you are proposing seems to put a foot on the gas for those issues. Sure, some valid papers get rejected, but how many bad papers are also rejected? How would it affect the wider scientific community if 1 additional good paper is “approved” and also 100 bad papers? Does good science still make it through eventually, or are these rejected papers losing valuable insights “forever”? What kind of damage can a bad paper cause and how often? It just seems like it isn’t as clear cut as “the current system is flawed…and the alternative is objectively better” to me.

pessimizer 1y ago

I think they meant "reinterpreted" over the last century, not over the span of your personal experience and career.

olddustytrail 1y ago
They're saying it changed to not being reviewed properly and I'm saying from recent experience that it is.

godelski 1y ago

Look at the history of Peer Review. What you see post 1950 is pretty different than what you see prior to that. I think this quote is the best one-liner, though I think everyone should dig much more into the question > in the early 20th century, "the burden of proof was generally on the opponents rather than the proponents of new ideas. That is, the reviewers had a higher burden than the authors. The bias is towards acceptance rather than rejection. In a perfect world we could only accept good papers and could reject bad papers, but we don't live in that world. So the question is "when we fail, which way do we want to fail?" Obviously, I'm on the side of Blackstone here https://en.wikipedia.org/wiki/Scholarly_peer_review#History https://en.wikipedia.org/wiki/Scholarly_peer_review#History

olddustytrail 1y ago
But that's the opposite of what the person I'm replying to said. They're saying everything is acceptable and I'm saying it's actually reviewed. Ok, maybe that's not what you meant. Peer review doesn't reject papers because they don't agree with the orthodoxy; they reject them because they're not competent. Is that what you were getting at?
- godelski 1y ago
  I'm a bit confused. What you described is what happens today. Yes. That has been my experience too, serving as a reviewer. I understood xondono to be referencing the history that I mentioned, which is where these reviewers didn't exist. So the requirement was different, which is what I'm saying about the burden of proof. > Peer review doesn't reject papers because they don't agree with the orthodoxy; they reject them because they're not competent. This is absolutely false and I don't know a single academic who hasn't seen competent papers get rejected. Reviewers can reject for any reason. The system is built on trust, but incentivized to reject. "Not competent" is too vague of a term, just like "not novel." In my other comment[0] I even reference one of the famous works that got rejected 3 times for being "not competent". This isn't a one-off case here, it is a common occurrence. On several occasions I've had to champion papers which were clearly competent yet my fellow reviewers simply were not familiar with the domain (they admitted this during discussion). I've also killed papers for similar reasons (a very rare event as I strongly bias towards accepting). So I'm sorry, saying papers are only rejected because they are "not competent" is incredibly naive. And I'm sorry, but the claim that "works aren't rejected because they don't agree with orthodoxy" is simply laughable. There's a long history of peers rejecting discoveries that upset the norms. This has happened to the majority of well known scientists. I'm not talking about like the Church going after Galileo, I'm talking about things like Galilelo arguing with Tycho Brahe or Christoph Scheiner. Einstein was critical of Bohr. Hertz was critical of Bell. The list goes on and on. The criticism was explicitly about running counter to orthodoxy. This is such a common thread in history that there's even Max Planck stated "Science advances one funeral at a time." [0] https://news.ycombinator.com/item?id=44587535 https://news.ycombinator.com/item?id=44587535
  - 542354234235 1y ago
    But wouldn’t a consequence of failing to err on the side of caution and “orthodoxy” mean a proliferation of junk and pseudoscience? We already had that problem now but the change you are proposing seems to put a foot on the gas for those issues. Sure, some valid papers get rejected, but how many bad papers are also rejected? How would it affect the wider scientific community if 1 additional good paper is “approved” and also 100 bad papers? Does good science still make it through eventually, or are these rejected papers losing valuable insights “forever”? What kind of damage can a bad paper cause and how often? It just seems like it isn’t as clear cut as “the current system is flawed…and the alternative is objectively better” to me.