3 ms·

The other factor preventing a fix is that people with no actual serious experience of academic publishing and peer review will defend these journals, because th

The other factor preventing a fix is that people with no actual serious experience of academic publishing and peer review will defend these journals, because they still think that (journal-based) peer review acts like some kind of meaningful quality filter. But, it really doesn't. Because someone is surely going to try to defend journals via peer review in this thread, I want to provide a counter to the arguments that journal peer review does much good. Also, since everyone knows that if you just go to a poor enough journal, you can be published, I am going to focus on the (IMO mostly false) claim that higher-profile journals are still doing a good thing here. There are numerous studies showing that higher-profile journals in general have more retractions and research misconduct [1-2], lower research quality [3], in fact weaker statistical power and reliability [4], and that statistical reliability even in high prestige journals is still extremely poor overall [5]. Also, making it through peer review is highly random and dependent on who you get as a reviewer [6], or is just basically a coin toss even when looking at reviewer groups: In 2014, 49.5% of the papers accepted by the first committee were rejected by the second (with a fairly wide confidence interval as the experiment included only 116 papers). This year, this number was 50.6%. We can also look at the probability that a randomly chosen rejected paper would have been accepted if it were re-reviewed. This number was 14.9% this year, compared to 17.5% in 2014. [7] We should just move to arXiv-like approaches and allow the scientific community to broadly judge relevance and quality. Journals just slow things down and burn funding for very little gain or benefit to anyone other than the journal owners. [1] https://pmc.ncbi.nlm.nih.gov/articles/PMC3187237/ https://pmc.ncbi.nlm.nih.gov/articles/PMC3187237/ [2] https://www.pnas.org/doi/10.1073/pnas.1212247109 https://www.pnas.org/doi/10.1073/pnas.1212247109 [3] https://pmc.ncbi.nlm.nih.gov/articles/PMC9382220/ https://pmc.ncbi.nlm.nih.gov/articles/PMC9382220/ [4] https://journals.plos.org/plosbiology/article?id=10.1371%2Fjournal.pbio.2000797 https://journals.plos.org/plosbiology/article?id=10.1371%2Fj... [5] https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2018.00037/full https://www.frontiersin.org/journals/human-neuroscience/arti... [6] https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0014331 https://journals.plos.org/plosone/article?id=10.1371%2Fjourn... [7] https://blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment/ https://blog.neurips.cc/2021/12/08/the-neurips-2021-consiste...

bonsai_spool 7mo ago
I am sympathetic to the argument you wish to make, that peer review is no panacea, but the actual evidence you offer has nothing to do with this claim. You are trying to say that high profile journals have more retractions, which is well known as you share. How does that have anything to do with peer review? Are you saying that there is more review or less review in some cases and that influences retraction rate? In what evidence? In what world does the arxiv system moderate this discrepancy?
- D-Machine 7mo ago
  > How does that have anything to do with peer review? I already addressed this. People know peer review can be bad, but some think "good journals" still do good peer review. This is not so clear. > In what world does the arxiv system moderate this discrepancy? Open systems allow the scientific community to figure out ways to properly assess research quality and value more cheaply, and without passing through (often arbitrary and random) small numbers of gatekeepers that don't even do a reliable or good job gatekeeping in the first place.
  - bonsai_spool 7mo ago
    Your argument depends on worse peer review at top journals - but fundamentally, you fail to show how doing any peer review is strictly worse than doing no peer review. I understand that we want arxiv to exist, and it does, and it’s growing. That doesn’t mean we don’t want Nature or Science to triage the most compelling stories. Importantly, we can already begin the search for these ‘cheaper’ review strategies while not losing the helpful information filter we get by seeing where things are presented/published
    - D-Machine 7mo ago
      > Your argument depends on worse peer review at top journals - but fundamentally, you fail to show how doing any peer review is strictly worse than doing no peer review. No, it doesn't. The argument is that peer review is incompetent gatekeeping in general, and so slows things down and makes thing expensive. Also, I am countering the argument "we need journals because journals do peer review" by arguing "peer review by journals isn't clearly actually good", I am not saying "peer review in general is unneeded", as I support review by the entire scientific community, rather than journal gatekeepers. > you fail to show how doing any peer review is strictly worse than doing no peer review I wasn't trying to show that. I have provided plenty of arguments to show why killing journal-based peer review could definitely speed things up and so potentially make things better. I want actual organic review by the community, not by tiny groups of gatekeepers.

bonsai_spool 7mo ago

I am sympathetic to the argument you wish to make, that peer review is no panacea, but the actual evidence you offer has nothing to do with this claim. You are trying to say that high profile journals have more retractions, which is well known as you share. How does that have anything to do with peer review? Are you saying that there is more review or less review in some cases and that influences retraction rate? In what evidence? In what world does the arxiv system moderate this discrepancy?

D-Machine 7mo ago
> How does that have anything to do with peer review? I already addressed this. People know peer review can be bad, but some think "good journals" still do good peer review. This is not so clear. > In what world does the arxiv system moderate this discrepancy? Open systems allow the scientific community to figure out ways to properly assess research quality and value more cheaply, and without passing through (often arbitrary and random) small numbers of gatekeepers that don't even do a reliable or good job gatekeeping in the first place.
- bonsai_spool 7mo ago
  Your argument depends on worse peer review at top journals - but fundamentally, you fail to show how doing any peer review is strictly worse than doing no peer review. I understand that we want arxiv to exist, and it does, and it’s growing. That doesn’t mean we don’t want Nature or Science to triage the most compelling stories. Importantly, we can already begin the search for these ‘cheaper’ review strategies while not losing the helpful information filter we get by seeing where things are presented/published
  - D-Machine 7mo ago
    > Your argument depends on worse peer review at top journals - but fundamentally, you fail to show how doing any peer review is strictly worse than doing no peer review. No, it doesn't. The argument is that peer review is incompetent gatekeeping in general, and so slows things down and makes thing expensive. Also, I am countering the argument "we need journals because journals do peer review" by arguing "peer review by journals isn't clearly actually good", I am not saying "peer review in general is unneeded", as I support review by the entire scientific community, rather than journal gatekeepers. > you fail to show how doing any peer review is strictly worse than doing no peer review I wasn't trying to show that. I have provided plenty of arguments to show why killing journal-based peer review could definitely speed things up and so potentially make things better. I want actual organic review by the community, not by tiny groups of gatekeepers.