3 ms·

where things seem to really go wrong with peer review is when entities outside a scientific community want to use published research to set public policy, decid

where things seem to really go wrong with peer review is when entities outside a scientific community want to use published research to set public policy, decide where to invest, or make high-stakes hiring decisions. doing this actually requires a costly investment in deep understanding of the research and the literature around it, to know if it is sound and high-impact. but nobody wants to make this investment, and they've mostly convinced themselves that just free-riding off peer review is good enough. of course it is not, flawed papers make it through peer review all the time. also this outside use of peer review introduces extremely strong distorting incentives -- an even hugely greater desire to be published in specific prestigious outlets -- in the face of which peer review is not really adequate to catch misconduct. I think the ideal form of peer-reviewed journal is unfortunately logically impossible: its contents would be completely open access, costing nothing for scientists who want to build on the results. But it would require an extremely expensive subscription to find out who has published in it, the money going to fund extremely thorough peer review, so that outside decision-makers can't try to free ride and distort the incentives.

bee_rider 3y ago
I think this is not even really a problem with the peer review system, but a problem with the (mostly, nonexistence of) an in depth science journalism field. Peer review just means it is ok to stick in a scientific journal and have peers read it… people with advanced bullshit detectors and nuance parsers in a particular domain. A paper is a brick. A policy is a building. It is not a problem in the brick manufacturing field, if people keep trying to build houses without any mortar. (Just to be explicit, I think you are right on the money and just wanted to elaborate/rant).
- thaumasiotes 3y ago
  > Peer review just means it is ok to stick in a scientific journal and have peers read it… people with advanced bullshit detectors and nuance parsers in a particular domain. That's not what peer review means. If that were what it meant, it would have no purpose - the people you describe don't benefit from the paper they're reading having been through peer review. They already understand the domain. Compare https://genomesunzipped.org/2012/08/the-first-steps-towards-a-modern-system-of-scientific-publication.php https://genomesunzipped.org/2012/08/the-first-steps-towards-... : > The solution to this problem relies on a simple observation – in my field, I am completely indifferent to whether a paper has been “peer-reviewed” for the basic reason that I consider myself a “peer”. I do not think it extremely hubristic to say that I am reasonably capable of evaluating whether a paper in my field is worth reading, and then if so, of judging its merits. The opinions of other people in the field are of course important, but in no way does the fact that two or three nameless people thought a paper worth publishing influence my opinion of it. The only goal that peer review serves is letting people who aren't peers and don't understand what the paper is saying believe that they can tell whether the paper is good or not. They still can't, but because of the peer review system, they feel that they can.
  - bee_rider 3y ago
    It isn’t pointless, researchers have finite time and attention, the point of peer review is to keep out the really bad papers so that they don’t waste their time on them.
- godelski 3y ago
  > I think this is not even really a problem with the peer review system, but a problem with the (mostly, nonexistence of) an in depth science journalism field. Peer review just means it is ok to stick in a scientific journal and have peers read it… people with advanced bullshit detectors and nuance parsers in a particular domain. I don't agree with this. There is deception in its name: peer review. Even "preprints" (things on arxiv and not accepted in journals or conferences) are still read and evaluated by peers. LK-99 is a great example of this, and that's not an uncommon process. The problem is that venues perpetuate the idea that being published within them is a mark of validity. But, like you suggest, it isn't. And "peer review" doesn't mean that it is just okay for peers to read it because we're reading each others works long before they are released in venues. Unfortunately most people (including researchers) don't have advanced bullshit detectors or nuance parsers. These are often fairly narrow and there's not that strong of alignment between the subject matter that a reviewer is reviewing and their niche domain expertise. I agree journalism and politicians need to do better, but so do we in the scientific community. There's more nuance. It is definitely a problem with the brick manufacturer if the marketing team implies you don't need mortar or even says "generally you don't need mortar."
pdonis 3y ago
> entities outside a scientific community want to use published research to set public policy I don't think it's limited to entities outside a scientific community, because many of the people who want to use published research to set public policy are themselves scientists (often the same scientists who published the research).
thaumasiotes 3y ago
> where things seem to really go wrong with peer review is when entities outside a scientific community want to use published research to set public policy, decide where to invest, or make high-stakes hiring decisions OK, but those are the only use cases for peer review.

bee_rider 3y ago

I think this is not even really a problem with the peer review system, but a problem with the (mostly, nonexistence of) an in depth science journalism field. Peer review just means it is ok to stick in a scientific journal and have peers read it… people with advanced bullshit detectors and nuance parsers in a particular domain. A paper is a brick. A policy is a building. It is not a problem in the brick manufacturing field, if people keep trying to build houses without any mortar. (Just to be explicit, I think you are right on the money and just wanted to elaborate/rant).

thaumasiotes 3y ago
> Peer review just means it is ok to stick in a scientific journal and have peers read it… people with advanced bullshit detectors and nuance parsers in a particular domain. That's not what peer review means. If that were what it meant, it would have no purpose - the people you describe don't benefit from the paper they're reading having been through peer review. They already understand the domain. Compare https://genomesunzipped.org/2012/08/the-first-steps-towards-a-modern-system-of-scientific-publication.php https://genomesunzipped.org/2012/08/the-first-steps-towards-... : > The solution to this problem relies on a simple observation – in my field, I am completely indifferent to whether a paper has been “peer-reviewed” for the basic reason that I consider myself a “peer”. I do not think it extremely hubristic to say that I am reasonably capable of evaluating whether a paper in my field is worth reading, and then if so, of judging its merits. The opinions of other people in the field are of course important, but in no way does the fact that two or three nameless people thought a paper worth publishing influence my opinion of it. The only goal that peer review serves is letting people who aren't peers and don't understand what the paper is saying believe that they can tell whether the paper is good or not. They still can't, but because of the peer review system, they feel that they can.
- bee_rider 3y ago
  It isn’t pointless, researchers have finite time and attention, the point of peer review is to keep out the really bad papers so that they don’t waste their time on them.
godelski 3y ago
> I think this is not even really a problem with the peer review system, but a problem with the (mostly, nonexistence of) an in depth science journalism field. Peer review just means it is ok to stick in a scientific journal and have peers read it… people with advanced bullshit detectors and nuance parsers in a particular domain. I don't agree with this. There is deception in its name: peer review. Even "preprints" (things on arxiv and not accepted in journals or conferences) are still read and evaluated by peers. LK-99 is a great example of this, and that's not an uncommon process. The problem is that venues perpetuate the idea that being published within them is a mark of validity. But, like you suggest, it isn't. And "peer review" doesn't mean that it is just okay for peers to read it because we're reading each others works long before they are released in venues. Unfortunately most people (including researchers) don't have advanced bullshit detectors or nuance parsers. These are often fairly narrow and there's not that strong of alignment between the subject matter that a reviewer is reviewing and their niche domain expertise. I agree journalism and politicians need to do better, but so do we in the scientific community. There's more nuance. It is definitely a problem with the brick manufacturer if the marketing team implies you don't need mortar or even says "generally you don't need mortar."

pdonis 3y ago

> entities outside a scientific community want to use published research to set public policy I don't think it's limited to entities outside a scientific community, because many of the people who want to use published research to set public policy are themselves scientists (often the same scientists who published the research).

thaumasiotes 3y ago

> where things seem to really go wrong with peer review is when entities outside a scientific community want to use published research to set public policy, decide where to invest, or make high-stakes hiring decisions OK, but those are the only use cases for peer review.