4 ms·

> it is difficult to state how much worse things could be Because there's no proof. The only thing we know for certain, is that a lot of money is being spent

by Vinnl 4y ago

> it is difficult to state how much worse things could be Because there's no proof. The only thing we know for certain, is that a lot of money is being spent on it, and that some of the mechanisms that are supposed to work can't work (i.e. in the case where people look at and trust preprints anyway). There will definitely be anecdotes of peer review having caught things (though it'll be harder to show that that prevented harm). But likewise, there are also many, many anecdotes that it has been actively harmful. No proven upside, limited anecdotal upside, and proven downside. If peer review as a publication gatekeeper hadn't already been a thing, we wouldn't introduce it — that would be considered unscientific. Sounds like something that's ripe for reconsideration.

fastaguy88 4y ago
I don’t understand how we know that a whole lot of money is spent on it. A lot of volunteer (unpaid) time is spent on it, but not much money. And I’m also unaware of the “proven” downside. But you might ask yourself, why do so many scientists waste so much of their time doing peer review. Perhaps they are ignorant, or easily duped, or perhaps they believe that peer review is a valuable use of their time.
- ProjectArcturis 4y ago
  That time is coming from somewhere -- it's time that's not spent doing actual research. It's certainly not free. When I was in academia, I did peer review initially because it was fun to be on the other side for once. And later mostly as a favor to the editors who asked me.
  - pixl97 4y ago
    And how many things did you find that were incorrect in the peer review? That seems to be the point that the anti-peer review people are missing.
    - ProjectArcturis 4y ago
      I found some things wrong. Occasionally these were minor things that the authors corrected. Sometimes they were major problems, and I recommended that the paper be rejected because they were unfixable. In that case I'd usually see the same paper get published in a different journal later on.
      - cauch 4y ago
        So your conclusion should be: peer-reviewing is working, but some people are not using it right. You then need to assess exactly what is this percentage of still being published despite having things wrong in it. If this percentage is small, peer-reviewing is still more profitable then no peer-reviewing at all (in which 100% of the bad papers are published).
  - cauch 4y ago
    > it's time that's not spent doing actual research Actual research includes keeping up to date with peers progress. If you were doing research but not regularly reading new publications anyway, either you were working on a less-fundamental-research side of things, or, without even realising it, you were exploring directions already explored or losing time by not profiting from a new useful approach. It is just incorrect to say that the time used to read carefully a peer paper is taken over the time of doing research: if you were not peer-reviewing this paper, you maybe would have been reading carefully the same paper in order to keep up to date with your peers. In fact, one can lose its time peer-reviewing, but its their own fault: in general, you just need to target the review for articles that are useful for you to read anyway.
- _aavaa_ 4y ago
  For starters we know that a lot of money is spent on it in terms of getting the end results. As you say the peer review is done by volunteers, but the final product is then sold back to those same reviewers (or their institutions) for an insane markup. > or perhaps they believe that peer review is a valuable use of their time. Or perhaps they believe that if they don’t do that work then bad papers will be published under the authoritative rubber stamp of the journals and that they will not need to spend the same amount of time down the road finding out if/how the fascinating study is wrong after having paid 40$ for the pdf.
- Vinnl 4y ago
  The proven downside is the money/time spent on it. And plenty of that "volunteer" time is time researchers would otherwise have spent on research. (Note: I'm not counting the time spent doing the review and feeding that back to the researcher — that can be valuable. I am counting time spent doing countless revisions just to please a reviewer, time spent resubmitting work elsewhere, etc.) And don't get me started on money wasted on price-gouging publishers thanks to their role as gatekeepers of career credentials. The question is not why scientists spend time doing peer review. The question is: why do academics waste time bending over backwards to adapt to the whims of a particular reviewer even beyond when they feel the input is useful? (And the answer is above: the career credentials.) Again, I'm not agains peer review by itself. But it should not be a gatekeeper to publication, nor have this much leverage on people's careers (a binary decision for every article).
poulpy123 4y ago
> Because there's no proof. Have you ever worked in research on a topic a bit trendy or fascinating for the general population? Because the world is full of people convinced that they are the new Einstein
sseagull 4y ago
A paper that is not peer reviewed is basically like a blog article. There are great blog articles, but also a lot with basically nonsense written by people who think they know something but don't. There's a lot of noise. How is someone going to know, before they read it, whether they should spend time on it? It's a bar to clear. It helps to increase the signal to noise ratio. It's not perfect - bad articles get published, good articles get rejected. But imperfection is part of life. One thing is that knowing your paper is going to be peer reviewed might discourage a lot of cranks from trying to publish. Also, I believe editors have the ability to outright reject papers before they even go out to peer review. This cuts down on completely irrelevant and terrible articles.
- SpicyLemonZest 4y ago
  I guess they'd know the same way as they do for blog articles? I'm confused about the premise here. As the author says, all the academics I know already do spend lots of their research time reading blog posts and preprints other sources which haven't been vetted for quality.
  - Vinnl 4y ago
    And one can come up with plenty of alternative ways of discovering relevant research without blocking or delaying publication or having researchers spend many hours bending over backwards for all kinds of minutiae that don't meaningfully help with that goal. (Disclosure: I volunteer for one attempt at such an alternative, https://plaudit.pub https://plaudit.pub.)
- timr 4y ago
  > A paper that is not peer reviewed is basically like a blog article. The one thing has nothing to do with the other. I have seen many "peer-reviewed" articles regarding Covid in 2020-2022 that were no better than blog posts. It isn't a panacea, and the last two years have drawn into sharp relief one of the major problems of the system: confirmation bias. Peer review is worth no more than the "peers" who review it, their objectivity, personal investment in the claim, and the level of effort they put into the review. For example, if the "peers" are a group of people who are all seeking to confirm their prior beliefs about a controversial topic, the peer review process is almost completely useless as a signal of quality. Papers that confirm groupthink are routinely shunted to the high-status members of the group for like-minded "review" and rubber-stamp approval. Papers that challenge groupthink are almost invariably nit-picked and/or dismissed entirely. Many never even make it to peer review, because the journal editor rejects the submission. Throw in issues like reviewer bias toward "established" authors and fields, as well as lack of blinding and selection bias amongst the reviewer pool, and you have a recipe for a formalized system of groupthink.

by Vinnl 4y ago

fastaguy88 4y ago

I don’t understand how we know that a whole lot of money is spent on it. A lot of volunteer (unpaid) time is spent on it, but not much money. And I’m also unaware of the “proven” downside. But you might ask yourself, why do so many scientists waste so much of their time doing peer review. Perhaps they are ignorant, or easily duped, or perhaps they believe that peer review is a valuable use of their time.

ProjectArcturis 4y ago
That time is coming from somewhere -- it's time that's not spent doing actual research. It's certainly not free. When I was in academia, I did peer review initially because it was fun to be on the other side for once. And later mostly as a favor to the editors who asked me.
- pixl97 4y ago
  And how many things did you find that were incorrect in the peer review? That seems to be the point that the anti-peer review people are missing.
  - ProjectArcturis 4y ago
    I found some things wrong. Occasionally these were minor things that the authors corrected. Sometimes they were major problems, and I recommended that the paper be rejected because they were unfixable. In that case I'd usually see the same paper get published in a different journal later on.
    - cauch 4y ago
      So your conclusion should be: peer-reviewing is working, but some people are not using it right. You then need to assess exactly what is this percentage of still being published despite having things wrong in it. If this percentage is small, peer-reviewing is still more profitable then no peer-reviewing at all (in which 100% of the bad papers are published).
- cauch 4y ago
  > it's time that's not spent doing actual research Actual research includes keeping up to date with peers progress. If you were doing research but not regularly reading new publications anyway, either you were working on a less-fundamental-research side of things, or, without even realising it, you were exploring directions already explored or losing time by not profiting from a new useful approach. It is just incorrect to say that the time used to read carefully a peer paper is taken over the time of doing research: if you were not peer-reviewing this paper, you maybe would have been reading carefully the same paper in order to keep up to date with your peers. In fact, one can lose its time peer-reviewing, but its their own fault: in general, you just need to target the review for articles that are useful for you to read anyway.
_aavaa_ 4y ago
For starters we know that a lot of money is spent on it in terms of getting the end results. As you say the peer review is done by volunteers, but the final product is then sold back to those same reviewers (or their institutions) for an insane markup. > or perhaps they believe that peer review is a valuable use of their time. Or perhaps they believe that if they don’t do that work then bad papers will be published under the authoritative rubber stamp of the journals and that they will not need to spend the same amount of time down the road finding out if/how the fascinating study is wrong after having paid 40$ for the pdf.
Vinnl 4y ago
The proven downside is the money/time spent on it. And plenty of that "volunteer" time is time researchers would otherwise have spent on research. (Note: I'm not counting the time spent doing the review and feeding that back to the researcher — that can be valuable. I am counting time spent doing countless revisions just to please a reviewer, time spent resubmitting work elsewhere, etc.) And don't get me started on money wasted on price-gouging publishers thanks to their role as gatekeepers of career credentials. The question is not why scientists spend time doing peer review. The question is: why do academics waste time bending over backwards to adapt to the whims of a particular reviewer even beyond when they feel the input is useful? (And the answer is above: the career credentials.) Again, I'm not agains peer review by itself. But it should not be a gatekeeper to publication, nor have this much leverage on people's careers (a binary decision for every article).

poulpy123 4y ago

> Because there's no proof. Have you ever worked in research on a topic a bit trendy or fascinating for the general population? Because the world is full of people convinced that they are the new Einstein

sseagull 4y ago

A paper that is not peer reviewed is basically like a blog article. There are great blog articles, but also a lot with basically nonsense written by people who think they know something but don't. There's a lot of noise. How is someone going to know, before they read it, whether they should spend time on it? It's a bar to clear. It helps to increase the signal to noise ratio. It's not perfect - bad articles get published, good articles get rejected. But imperfection is part of life. One thing is that knowing your paper is going to be peer reviewed might discourage a lot of cranks from trying to publish. Also, I believe editors have the ability to outright reject papers before they even go out to peer review. This cuts down on completely irrelevant and terrible articles.

SpicyLemonZest 4y ago
I guess they'd know the same way as they do for blog articles? I'm confused about the premise here. As the author says, all the academics I know already do spend lots of their research time reading blog posts and preprints other sources which haven't been vetted for quality.
- Vinnl 4y ago
  And one can come up with plenty of alternative ways of discovering relevant research without blocking or delaying publication or having researchers spend many hours bending over backwards for all kinds of minutiae that don't meaningfully help with that goal. (Disclosure: I volunteer for one attempt at such an alternative, https://plaudit.pub https://plaudit.pub.)
timr 4y ago
> A paper that is not peer reviewed is basically like a blog article. The one thing has nothing to do with the other. I have seen many "peer-reviewed" articles regarding Covid in 2020-2022 that were no better than blog posts. It isn't a panacea, and the last two years have drawn into sharp relief one of the major problems of the system: confirmation bias. Peer review is worth no more than the "peers" who review it, their objectivity, personal investment in the claim, and the level of effort they put into the review. For example, if the "peers" are a group of people who are all seeking to confirm their prior beliefs about a controversial topic, the peer review process is almost completely useless as a signal of quality. Papers that confirm groupthink are routinely shunted to the high-status members of the group for like-minded "review" and rubber-stamp approval. Papers that challenge groupthink are almost invariably nit-picked and/or dismissed entirely. Many never even make it to peer review, because the journal editor rejects the submission. Throw in issues like reviewer bias toward "established" authors and fields, as well as lack of blinding and selection bias amongst the reviewer pool, and you have a recipe for a formalized system of groupthink.