4 ms·

Wow. Pretty clear this paper was not peer reviewed. Many of the arguments would not get past a high school English teacher. ‘A whole lot of money for nothin’

Wow. Pretty clear this paper was not peer reviewed. Many of the arguments would not get past a high school English teacher. ‘A whole lot of money for nothin’ perhaps, but that money is not spent on peer review. Peer review doesn’t improve research productivity …. This reminds me of the argument that we should not spend more money on education, because education budgets have gone up, but test scores have not. The argument implies (but does not state, because of the absurdity), that spending less money would raise scores. And, remarkably (/s), peer review does not guarantee that all scientific results are reproducible and correct. As the author points out, no one has done the control. We don’t know how much worse the scientific literature would be without peer review, and we don’t no how many important insights are lost because of poor reviews. But as someone who has done a lot of peer review (and has had papers rejected with uninformed reviews), it is difficult to state how much worse things could be. (But perhaps that is an overstatement, as very few papers do not find their way into some journal). The scientific literature today is overwhelming. Peer review helps make it less so.

Vinnl 4y ago
> it is difficult to state how much worse things could be Because there's no proof. The only thing we know for certain, is that a lot of money is being spent on it, and that some of the mechanisms that are supposed to work can't work (i.e. in the case where people look at and trust preprints anyway). There will definitely be anecdotes of peer review having caught things (though it'll be harder to show that that prevented harm). But likewise, there are also many, many anecdotes that it has been actively harmful. No proven upside, limited anecdotal upside, and proven downside. If peer review as a publication gatekeeper hadn't already been a thing, we wouldn't introduce it — that would be considered unscientific. Sounds like something that's ripe for reconsideration.
- fastaguy88 4y ago
  I don’t understand how we know that a whole lot of money is spent on it. A lot of volunteer (unpaid) time is spent on it, but not much money. And I’m also unaware of the “proven” downside. But you might ask yourself, why do so many scientists waste so much of their time doing peer review. Perhaps they are ignorant, or easily duped, or perhaps they believe that peer review is a valuable use of their time.
  - ProjectArcturis 4y ago
    That time is coming from somewhere -- it's time that's not spent doing actual research. It's certainly not free. When I was in academia, I did peer review initially because it was fun to be on the other side for once. And later mostly as a favor to the editors who asked me.
    - pixl97 4y ago
      And how many things did you find that were incorrect in the peer review? That seems to be the point that the anti-peer review people are missing.
      - ProjectArcturis 4y ago
        I found some things wrong. Occasionally these were minor things that the authors corrected. Sometimes they were major problems, and I recommended that the paper be rejected because they were unfixable. In that case I'd usually see the same paper get published in a different journal later on.
        cauch 4y ago
        So your conclusion should be: peer-reviewing is working, but some people are not using it right. You then need to assess exactly what is this percentage of still being published despite having things wrong in it. If this percentage is small, peer-reviewing is still more profitable then no peer-reviewing at all (in which 100% of the bad papers are published).
    - cauch 4y ago
      > it's time that's not spent doing actual research Actual research includes keeping up to date with peers progress. If you were doing research but not regularly reading new publications anyway, either you were working on a less-fundamental-research side of things, or, without even realising it, you were exploring directions already explored or losing time by not profiting from a new useful approach. It is just incorrect to say that the time used to read carefully a peer paper is taken over the time of doing research: if you were not peer-reviewing this paper, you maybe would have been reading carefully the same paper in order to keep up to date with your peers. In fact, one can lose its time peer-reviewing, but its their own fault: in general, you just need to target the review for articles that are useful for you to read anyway.
- poulpy123 4y ago
  > Because there's no proof. Have you ever worked in research on a topic a bit trendy or fascinating for the general population? Because the world is full of people convinced that they are the new Einstein
- sseagull 4y ago
  A paper that is not peer reviewed is basically like a blog article. There are great blog articles, but also a lot with basically nonsense written by people who think they know something but don't. There's a lot of noise. How is someone going to know, before they read it, whether they should spend time on it? It's a bar to clear. It helps to increase the signal to noise ratio. It's not perfect - bad articles get published, good articles get rejected. But imperfection is part of life. One thing is that knowing your paper is going to be peer reviewed might discourage a lot of cranks from trying to publish. Also, I believe editors have the ability to outright reject papers before they even go out to peer review. This cuts down on completely irrelevant and terrible articles.
  - SpicyLemonZest 4y ago
    I guess they'd know the same way as they do for blog articles? I'm confused about the premise here. As the author says, all the academics I know already do spend lots of their research time reading blog posts and preprints other sources which haven't been vetted for quality.
    - Vinnl 4y ago
      And one can come up with plenty of alternative ways of discovering relevant research without blocking or delaying publication or having researchers spend many hours bending over backwards for all kinds of minutiae that don't meaningfully help with that goal. (Disclosure: I volunteer for one attempt at such an alternative, https://plaudit.pub https://plaudit.pub.)
  - timr 4y ago
    > A paper that is not peer reviewed is basically like a blog article. The one thing has nothing to do with the other. I have seen many "peer-reviewed" articles regarding Covid in 2020-2022 that were no better than blog posts. It isn't a panacea, and the last two years have drawn into sharp relief one of the major problems of the system: confirmation bias. Peer review is worth no more than the "peers" who review it, their objectivity, personal investment in the claim, and the level of effort they put into the review. For example, if the "peers" are a group of people who are all seeking to confirm their prior beliefs about a controversial topic, the peer review process is almost completely useless as a signal of quality. Papers that confirm groupthink are routinely shunted to the high-status members of the group for like-minded "review" and rubber-stamp approval. Papers that challenge groupthink are almost invariably nit-picked and/or dismissed entirely. Many never even make it to peer review, because the journal editor rejects the submission. Throw in issues like reviewer bias toward "established" authors and fields, as well as lack of blinding and selection bias amongst the reviewer pool, and you have a recipe for a formalized system of groupthink.
ren_engineer 4y ago
>We don’t know how much worse the scientific literature would be without peer review all of human history prior to the last 50 years? The term "peer review" didn't even exist until the 60s https://scalar.usc.edu/works/the-art-of-academic-peer-reviewing/media/PeerReviewonGoogleBooksNGramViewer.jpg https://scalar.usc.edu/works/the-art-of-academic-peer-review...
- fastaguy88 4y ago
  I do not understand this argument. Just as peer review is relatively young, so is the massive expansion of the scientific literature. Perhaps in the 50’s (when there was peer review, at least in biochemistry and microbiology), an investigator might reasonably believe they were familiar with most of the literature in their field. That has not been true for decades. So, by volume (if not impact), most scientific results are young and have been subject to peer review. Perhaps part of the argument has to do with the specific form of peer review. In the 18th and 19th centuries, much more science was communicated via correspondence and books. But I suspect there was still a review process, just one that was less democratic and more dependent on reputation.
- wolverine876 4y ago
  The progress of science has never been greater than under the peer review system. I'm sure we can improve it, but if science since the 1960s is a failure, than almost everything in human history is a failure.
  - shrubby 4y ago
    The planet is burning and a lot of people are dying from preventable diseases and malnutrition. So yes. Practically everything in human history is a failure.
    - wolverine876 4y ago
      Far fewer people die of preventable diseases and malnutrition than any point in history; we've accomplished incredible things, especially since 1960. What are we writing this on?
photochemsyn 4y ago
Arxiv doesn't use peer review and seems to have lots of readable quality papers.
xracy 4y ago
I've long wondered if a part of peer review needs to be replication. OR if there needs to be better incentives around finding nothing out. Like, given the goal is getting published, it feels like there should be better incentives for papers around not finding anything in an interesting way, or hypothesis testing and discovering your hypothesis was wrong.
svachalek 4y ago
> This reminds me of the argument that we should not spend more money on education, because education budgets have gone up, but test scores have not. The argument implies (but does not state, because of the absurdity), that spending less money would raise scores. The argument perhaps implies that spending less money would not lower scores, but as put here does not imply anything about raising scores.
trompetenaccoun 4y ago
>Wow. Pretty clear this paper was not peer reviewed. Many of the arguments would not get past a high school English teacher. How is that clear? There are plenty of papers out there that contain glaring errors and got accepted anyway, if you regularly read publications you should know this better than anyone. Also you confirm one of their criticisms and contradict your own argument that peer review helps improve scientific literature. What do you base that on, apart from belief?
kansface 4y ago
> The argument implies (but does not state, because of the absurdity), that spending less money would raise scores. I think the correct take away is spending on education is only very loosely coupled to outcomes once some very low threshold is met.
- fastaguy88 4y ago
  Or that there are many factors that affect test scores (and many more alternatives competing for students’ attention). And, looking at average scores and average spending probably makes more subtle relationships more difficult to tease out. But it is hard for me to believe that increased funding is a bad thing for students.
AlbertCory 4y ago
> The argument implies ... that spending less money would raise scores only in your imagination does it imply that.

Vinnl 4y ago

> it is difficult to state how much worse things could be Because there's no proof. The only thing we know for certain, is that a lot of money is being spent on it, and that some of the mechanisms that are supposed to work can't work (i.e. in the case where people look at and trust preprints anyway). There will definitely be anecdotes of peer review having caught things (though it'll be harder to show that that prevented harm). But likewise, there are also many, many anecdotes that it has been actively harmful. No proven upside, limited anecdotal upside, and proven downside. If peer review as a publication gatekeeper hadn't already been a thing, we wouldn't introduce it — that would be considered unscientific. Sounds like something that's ripe for reconsideration.

fastaguy88 4y ago
I don’t understand how we know that a whole lot of money is spent on it. A lot of volunteer (unpaid) time is spent on it, but not much money. And I’m also unaware of the “proven” downside. But you might ask yourself, why do so many scientists waste so much of their time doing peer review. Perhaps they are ignorant, or easily duped, or perhaps they believe that peer review is a valuable use of their time.
- ProjectArcturis 4y ago
  That time is coming from somewhere -- it's time that's not spent doing actual research. It's certainly not free. When I was in academia, I did peer review initially because it was fun to be on the other side for once. And later mostly as a favor to the editors who asked me.
  - pixl97 4y ago
    And how many things did you find that were incorrect in the peer review? That seems to be the point that the anti-peer review people are missing.
    - ProjectArcturis 4y ago
      I found some things wrong. Occasionally these were minor things that the authors corrected. Sometimes they were major problems, and I recommended that the paper be rejected because they were unfixable. In that case I'd usually see the same paper get published in a different journal later on.
      - cauch 4y ago
        So your conclusion should be: peer-reviewing is working, but some people are not using it right. You then need to assess exactly what is this percentage of still being published despite having things wrong in it. If this percentage is small, peer-reviewing is still more profitable then no peer-reviewing at all (in which 100% of the bad papers are published).
  - cauch 4y ago
    > it's time that's not spent doing actual research Actual research includes keeping up to date with peers progress. If you were doing research but not regularly reading new publications anyway, either you were working on a less-fundamental-research side of things, or, without even realising it, you were exploring directions already explored or losing time by not profiting from a new useful approach. It is just incorrect to say that the time used to read carefully a peer paper is taken over the time of doing research: if you were not peer-reviewing this paper, you maybe would have been reading carefully the same paper in order to keep up to date with your peers. In fact, one can lose its time peer-reviewing, but its their own fault: in general, you just need to target the review for articles that are useful for you to read anyway.
poulpy123 4y ago
> Because there's no proof. Have you ever worked in research on a topic a bit trendy or fascinating for the general population? Because the world is full of people convinced that they are the new Einstein
sseagull 4y ago
A paper that is not peer reviewed is basically like a blog article. There are great blog articles, but also a lot with basically nonsense written by people who think they know something but don't. There's a lot of noise. How is someone going to know, before they read it, whether they should spend time on it? It's a bar to clear. It helps to increase the signal to noise ratio. It's not perfect - bad articles get published, good articles get rejected. But imperfection is part of life. One thing is that knowing your paper is going to be peer reviewed might discourage a lot of cranks from trying to publish. Also, I believe editors have the ability to outright reject papers before they even go out to peer review. This cuts down on completely irrelevant and terrible articles.
- SpicyLemonZest 4y ago
  I guess they'd know the same way as they do for blog articles? I'm confused about the premise here. As the author says, all the academics I know already do spend lots of their research time reading blog posts and preprints other sources which haven't been vetted for quality.
  - Vinnl 4y ago
    And one can come up with plenty of alternative ways of discovering relevant research without blocking or delaying publication or having researchers spend many hours bending over backwards for all kinds of minutiae that don't meaningfully help with that goal. (Disclosure: I volunteer for one attempt at such an alternative, https://plaudit.pub https://plaudit.pub.)
- timr 4y ago
  > A paper that is not peer reviewed is basically like a blog article. The one thing has nothing to do with the other. I have seen many "peer-reviewed" articles regarding Covid in 2020-2022 that were no better than blog posts. It isn't a panacea, and the last two years have drawn into sharp relief one of the major problems of the system: confirmation bias. Peer review is worth no more than the "peers" who review it, their objectivity, personal investment in the claim, and the level of effort they put into the review. For example, if the "peers" are a group of people who are all seeking to confirm their prior beliefs about a controversial topic, the peer review process is almost completely useless as a signal of quality. Papers that confirm groupthink are routinely shunted to the high-status members of the group for like-minded "review" and rubber-stamp approval. Papers that challenge groupthink are almost invariably nit-picked and/or dismissed entirely. Many never even make it to peer review, because the journal editor rejects the submission. Throw in issues like reviewer bias toward "established" authors and fields, as well as lack of blinding and selection bias amongst the reviewer pool, and you have a recipe for a formalized system of groupthink.

ren_engineer 4y ago

>We don’t know how much worse the scientific literature would be without peer review all of human history prior to the last 50 years? The term "peer review" didn't even exist until the 60s https://scalar.usc.edu/works/the-art-of-academic-peer-reviewing/media/PeerReviewonGoogleBooksNGramViewer.jpg https://scalar.usc.edu/works/the-art-of-academic-peer-review...

fastaguy88 4y ago
I do not understand this argument. Just as peer review is relatively young, so is the massive expansion of the scientific literature. Perhaps in the 50’s (when there was peer review, at least in biochemistry and microbiology), an investigator might reasonably believe they were familiar with most of the literature in their field. That has not been true for decades. So, by volume (if not impact), most scientific results are young and have been subject to peer review. Perhaps part of the argument has to do with the specific form of peer review. In the 18th and 19th centuries, much more science was communicated via correspondence and books. But I suspect there was still a review process, just one that was less democratic and more dependent on reputation.
wolverine876 4y ago
The progress of science has never been greater than under the peer review system. I'm sure we can improve it, but if science since the 1960s is a failure, than almost everything in human history is a failure.
- shrubby 4y ago
  The planet is burning and a lot of people are dying from preventable diseases and malnutrition. So yes. Practically everything in human history is a failure.
  - wolverine876 4y ago
    Far fewer people die of preventable diseases and malnutrition than any point in history; we've accomplished incredible things, especially since 1960. What are we writing this on?

photochemsyn 4y ago

Arxiv doesn't use peer review and seems to have lots of readable quality papers.

xracy 4y ago

I've long wondered if a part of peer review needs to be replication. OR if there needs to be better incentives around finding nothing out. Like, given the goal is getting published, it feels like there should be better incentives for papers around not finding anything in an interesting way, or hypothesis testing and discovering your hypothesis was wrong.

svachalek 4y ago

> This reminds me of the argument that we should not spend more money on education, because education budgets have gone up, but test scores have not. The argument implies (but does not state, because of the absurdity), that spending less money would raise scores. The argument perhaps implies that spending less money would not lower scores, but as put here does not imply anything about raising scores.

trompetenaccoun 4y ago

>Wow. Pretty clear this paper was not peer reviewed. Many of the arguments would not get past a high school English teacher. How is that clear? There are plenty of papers out there that contain glaring errors and got accepted anyway, if you regularly read publications you should know this better than anyone. Also you confirm one of their criticisms and contradict your own argument that peer review helps improve scientific literature. What do you base that on, apart from belief?

kansface 4y ago

> The argument implies (but does not state, because of the absurdity), that spending less money would raise scores. I think the correct take away is spending on education is only very loosely coupled to outcomes once some very low threshold is met.

fastaguy88 4y ago
Or that there are many factors that affect test scores (and many more alternatives competing for students’ attention). And, looking at average scores and average spending probably makes more subtle relationships more difficult to tease out. But it is hard for me to believe that increased funding is a bad thing for students.

AlbertCory 4y ago

> The argument implies ... that spending less money would raise scores only in your imagination does it imply that.