4 ms·

A paper that is not peer reviewed is basically like a blog article. There are great blog articles, but also a lot with basically nonsense written by people who

A paper that is not peer reviewed is basically like a blog article. There are great blog articles, but also a lot with basically nonsense written by people who think they know something but don't. There's a lot of noise. How is someone going to know, before they read it, whether they should spend time on it? It's a bar to clear. It helps to increase the signal to noise ratio. It's not perfect - bad articles get published, good articles get rejected. But imperfection is part of life. One thing is that knowing your paper is going to be peer reviewed might discourage a lot of cranks from trying to publish. Also, I believe editors have the ability to outright reject papers before they even go out to peer review. This cuts down on completely irrelevant and terrible articles.

SpicyLemonZest 4y ago
I guess they'd know the same way as they do for blog articles? I'm confused about the premise here. As the author says, all the academics I know already do spend lots of their research time reading blog posts and preprints other sources which haven't been vetted for quality.
- Vinnl 4y ago
  And one can come up with plenty of alternative ways of discovering relevant research without blocking or delaying publication or having researchers spend many hours bending over backwards for all kinds of minutiae that don't meaningfully help with that goal. (Disclosure: I volunteer for one attempt at such an alternative, https://plaudit.pub https://plaudit.pub.)
timr 4y ago
> A paper that is not peer reviewed is basically like a blog article. The one thing has nothing to do with the other. I have seen many "peer-reviewed" articles regarding Covid in 2020-2022 that were no better than blog posts. It isn't a panacea, and the last two years have drawn into sharp relief one of the major problems of the system: confirmation bias. Peer review is worth no more than the "peers" who review it, their objectivity, personal investment in the claim, and the level of effort they put into the review. For example, if the "peers" are a group of people who are all seeking to confirm their prior beliefs about a controversial topic, the peer review process is almost completely useless as a signal of quality. Papers that confirm groupthink are routinely shunted to the high-status members of the group for like-minded "review" and rubber-stamp approval. Papers that challenge groupthink are almost invariably nit-picked and/or dismissed entirely. Many never even make it to peer review, because the journal editor rejects the submission. Throw in issues like reviewer bias toward "established" authors and fields, as well as lack of blinding and selection bias amongst the reviewer pool, and you have a recipe for a formalized system of groupthink.
- soundnote 4y ago
  Apart from active confirmation bias, simple blind spots can also cause a lot of trouble. As one example heuristics and biases researchers Stanovich and Toplak found that their Actively Open-minded Thinking (AOT) instrument had a bug. It stood for twenty years because the bug was consonant with a secular-liberal worldview so no one really thought it might be wrong. https://doi.org/10.1016/j.cognition.2019.03.006 https://doi.org/10.1016/j.cognition.2019.03.006 This, of course, when the scientists themselves are honest, acting in good faith and eager to fix their mistakes as Stanovich and Toplak were. In openly political fields like sociology and whateverstudies, you get "researchers" blatantly manufacturing their conclusions in scale construction. As one example of some of the gems of scientific integrity in these fields is the finding that conservatives are higher in hostile sexism. Will any liberal check this? Of course not, because it conforms to their expectations that their political opponents are bad. Even if they are honest, if they just glance at a paper they won't question it, and it won't feel weird to read in a quick report on the research. How was the sausage made? In at least one study, the entire hostile sexism scale was two items long, and one item was, not kidding, "feminists are making entirely reasonable demands of me." And in the greatest surprise ever people dislike their political opponents. Bam, conservatives bad, because we're measuring their attitudes towards politically partisan activists, not the fairer sex. (The question also presumes that feminists' demands are reasonable. If the activist set are insane, a low score on the question would be an indicator of sanity, not misogyny or simple political partisanship. As an example, Simone de Beauvoir: "No, we don't believe that any woman should have this choice. No woman should be authorized to stay at home to raise children. Society should be totally different. Women should not have that choice, precisely because if there is such a choice, too many women will make that one. It is a way of forcing women in a certain direction.") There's also the famous case of Bob Altemeyer's Right-wing Authoritarianism scale which is explicitly constructed so that leftists will score lower on it than rightists. Cue decades of results that authoritarianism is more prevalent on the right. Quelle surprise.
- sseagull 4y ago
  > It isn't a panacea, and the last two years have drawn into sharp relief one of the major problems of the system: confirmation bias. I believe I was explicit in saying it wasn’t perfect. What alternative is there that doesn’t have something resembling peer review, isn’t curated by third parties, doesn’t suffer from confirmation bias/groupthink, and doesn’t leave it up to authors to sift through 100s of papers of unknown quality a day to learn about what’s going on in the field?

SpicyLemonZest 4y ago

I guess they'd know the same way as they do for blog articles? I'm confused about the premise here. As the author says, all the academics I know already do spend lots of their research time reading blog posts and preprints other sources which haven't been vetted for quality.

Vinnl 4y ago
And one can come up with plenty of alternative ways of discovering relevant research without blocking or delaying publication or having researchers spend many hours bending over backwards for all kinds of minutiae that don't meaningfully help with that goal. (Disclosure: I volunteer for one attempt at such an alternative, https://plaudit.pub https://plaudit.pub.)

timr 4y ago

> A paper that is not peer reviewed is basically like a blog article. The one thing has nothing to do with the other. I have seen many "peer-reviewed" articles regarding Covid in 2020-2022 that were no better than blog posts. It isn't a panacea, and the last two years have drawn into sharp relief one of the major problems of the system: confirmation bias. Peer review is worth no more than the "peers" who review it, their objectivity, personal investment in the claim, and the level of effort they put into the review. For example, if the "peers" are a group of people who are all seeking to confirm their prior beliefs about a controversial topic, the peer review process is almost completely useless as a signal of quality. Papers that confirm groupthink are routinely shunted to the high-status members of the group for like-minded "review" and rubber-stamp approval. Papers that challenge groupthink are almost invariably nit-picked and/or dismissed entirely. Many never even make it to peer review, because the journal editor rejects the submission. Throw in issues like reviewer bias toward "established" authors and fields, as well as lack of blinding and selection bias amongst the reviewer pool, and you have a recipe for a formalized system of groupthink.

soundnote 4y ago
Apart from active confirmation bias, simple blind spots can also cause a lot of trouble. As one example heuristics and biases researchers Stanovich and Toplak found that their Actively Open-minded Thinking (AOT) instrument had a bug. It stood for twenty years because the bug was consonant with a secular-liberal worldview so no one really thought it might be wrong. https://doi.org/10.1016/j.cognition.2019.03.006 https://doi.org/10.1016/j.cognition.2019.03.006 This, of course, when the scientists themselves are honest, acting in good faith and eager to fix their mistakes as Stanovich and Toplak were. In openly political fields like sociology and whateverstudies, you get "researchers" blatantly manufacturing their conclusions in scale construction. As one example of some of the gems of scientific integrity in these fields is the finding that conservatives are higher in hostile sexism. Will any liberal check this? Of course not, because it conforms to their expectations that their political opponents are bad. Even if they are honest, if they just glance at a paper they won't question it, and it won't feel weird to read in a quick report on the research. How was the sausage made? In at least one study, the entire hostile sexism scale was two items long, and one item was, not kidding, "feminists are making entirely reasonable demands of me." And in the greatest surprise ever people dislike their political opponents. Bam, conservatives bad, because we're measuring their attitudes towards politically partisan activists, not the fairer sex. (The question also presumes that feminists' demands are reasonable. If the activist set are insane, a low score on the question would be an indicator of sanity, not misogyny or simple political partisanship. As an example, Simone de Beauvoir: "No, we don't believe that any woman should have this choice. No woman should be authorized to stay at home to raise children. Society should be totally different. Women should not have that choice, precisely because if there is such a choice, too many women will make that one. It is a way of forcing women in a certain direction.") There's also the famous case of Bob Altemeyer's Right-wing Authoritarianism scale which is explicitly constructed so that leftists will score lower on it than rightists. Cue decades of results that authoritarianism is more prevalent on the right. Quelle surprise.
sseagull 4y ago
> It isn't a panacea, and the last two years have drawn into sharp relief one of the major problems of the system: confirmation bias. I believe I was explicit in saying it wasn’t perfect. What alternative is there that doesn’t have something resembling peer review, isn’t curated by third parties, doesn’t suffer from confirmation bias/groupthink, and doesn’t leave it up to authors to sift through 100s of papers of unknown quality a day to learn about what’s going on in the field?