3 ms·

> A paper that is not peer reviewed is basically like a blog article. The one thing has nothing to do with the other. I have seen many "peer-reviewed" article

by timr 4y ago

> A paper that is not peer reviewed is basically like a blog article. The one thing has nothing to do with the other. I have seen many "peer-reviewed" articles regarding Covid in 2020-2022 that were no better than blog posts. It isn't a panacea, and the last two years have drawn into sharp relief one of the major problems of the system: confirmation bias. Peer review is worth no more than the "peers" who review it, their objectivity, personal investment in the claim, and the level of effort they put into the review. For example, if the "peers" are a group of people who are all seeking to confirm their prior beliefs about a controversial topic, the peer review process is almost completely useless as a signal of quality. Papers that confirm groupthink are routinely shunted to the high-status members of the group for like-minded "review" and rubber-stamp approval. Papers that challenge groupthink are almost invariably nit-picked and/or dismissed entirely. Many never even make it to peer review, because the journal editor rejects the submission. Throw in issues like reviewer bias toward "established" authors and fields, as well as lack of blinding and selection bias amongst the reviewer pool, and you have a recipe for a formalized system of groupthink.

soundnote 4y ago
Apart from active confirmation bias, simple blind spots can also cause a lot of trouble. As one example heuristics and biases researchers Stanovich and Toplak found that their Actively Open-minded Thinking (AOT) instrument had a bug. It stood for twenty years because the bug was consonant with a secular-liberal worldview so no one really thought it might be wrong. https://doi.org/10.1016/j.cognition.2019.03.006 https://doi.org/10.1016/j.cognition.2019.03.006 This, of course, when the scientists themselves are honest, acting in good faith and eager to fix their mistakes as Stanovich and Toplak were. In openly political fields like sociology and whateverstudies, you get "researchers" blatantly manufacturing their conclusions in scale construction. As one example of some of the gems of scientific integrity in these fields is the finding that conservatives are higher in hostile sexism. Will any liberal check this? Of course not, because it conforms to their expectations that their political opponents are bad. Even if they are honest, if they just glance at a paper they won't question it, and it won't feel weird to read in a quick report on the research. How was the sausage made? In at least one study, the entire hostile sexism scale was two items long, and one item was, not kidding, "feminists are making entirely reasonable demands of me." And in the greatest surprise ever people dislike their political opponents. Bam, conservatives bad, because we're measuring their attitudes towards politically partisan activists, not the fairer sex. (The question also presumes that feminists' demands are reasonable. If the activist set are insane, a low score on the question would be an indicator of sanity, not misogyny or simple political partisanship. As an example, Simone de Beauvoir: "No, we don't believe that any woman should have this choice. No woman should be authorized to stay at home to raise children. Society should be totally different. Women should not have that choice, precisely because if there is such a choice, too many women will make that one. It is a way of forcing women in a certain direction.") There's also the famous case of Bob Altemeyer's Right-wing Authoritarianism scale which is explicitly constructed so that leftists will score lower on it than rightists. Cue decades of results that authoritarianism is more prevalent on the right. Quelle surprise.
sseagull 4y ago
> It isn't a panacea, and the last two years have drawn into sharp relief one of the major problems of the system: confirmation bias. I believe I was explicit in saying it wasn’t perfect. What alternative is there that doesn’t have something resembling peer review, isn’t curated by third parties, doesn’t suffer from confirmation bias/groupthink, and doesn’t leave it up to authors to sift through 100s of papers of unknown quality a day to learn about what’s going on in the field?
- timr 4y ago
  It's not just "not perfect"...it's pretty much useless as a signal of quality. Certainly, it doesn't come close to the level of credibility that you've attributed to it (i.e. papers without peer review are somehow "blog posts"). > What alternative is there that doesn’t have something resembling peer review, isn’t curated by third parties, doesn’t suffer from confirmation bias/groupthink, and doesn’t leave it up to authors to sift through 100s of papers of unknown quality a day to learn about what’s going on in the field? There's no free lunch. Nobody serious in any scientific field relies on peer review to determine paper quality. You have to read the paper in front of you and use your experience, which means you must read not just "100s of papers", but literally THOUSANDS of papers, in order to have any reasonable chance of being a credible evaluator. If you don't have this experience, you can't know the quality of a piece of scientific literature. That's just how it is.

by timr 4y ago

soundnote 4y ago

Apart from active confirmation bias, simple blind spots can also cause a lot of trouble. As one example heuristics and biases researchers Stanovich and Toplak found that their Actively Open-minded Thinking (AOT) instrument had a bug. It stood for twenty years because the bug was consonant with a secular-liberal worldview so no one really thought it might be wrong. https://doi.org/10.1016/j.cognition.2019.03.006 https://doi.org/10.1016/j.cognition.2019.03.006 This, of course, when the scientists themselves are honest, acting in good faith and eager to fix their mistakes as Stanovich and Toplak were. In openly political fields like sociology and whateverstudies, you get "researchers" blatantly manufacturing their conclusions in scale construction. As one example of some of the gems of scientific integrity in these fields is the finding that conservatives are higher in hostile sexism. Will any liberal check this? Of course not, because it conforms to their expectations that their political opponents are bad. Even if they are honest, if they just glance at a paper they won't question it, and it won't feel weird to read in a quick report on the research. How was the sausage made? In at least one study, the entire hostile sexism scale was two items long, and one item was, not kidding, "feminists are making entirely reasonable demands of me." And in the greatest surprise ever people dislike their political opponents. Bam, conservatives bad, because we're measuring their attitudes towards politically partisan activists, not the fairer sex. (The question also presumes that feminists' demands are reasonable. If the activist set are insane, a low score on the question would be an indicator of sanity, not misogyny or simple political partisanship. As an example, Simone de Beauvoir: "No, we don't believe that any woman should have this choice. No woman should be authorized to stay at home to raise children. Society should be totally different. Women should not have that choice, precisely because if there is such a choice, too many women will make that one. It is a way of forcing women in a certain direction.") There's also the famous case of Bob Altemeyer's Right-wing Authoritarianism scale which is explicitly constructed so that leftists will score lower on it than rightists. Cue decades of results that authoritarianism is more prevalent on the right. Quelle surprise.

sseagull 4y ago

> It isn't a panacea, and the last two years have drawn into sharp relief one of the major problems of the system: confirmation bias. I believe I was explicit in saying it wasn’t perfect. What alternative is there that doesn’t have something resembling peer review, isn’t curated by third parties, doesn’t suffer from confirmation bias/groupthink, and doesn’t leave it up to authors to sift through 100s of papers of unknown quality a day to learn about what’s going on in the field?

timr 4y ago
It's not just "not perfect"...it's pretty much useless as a signal of quality. Certainly, it doesn't come close to the level of credibility that you've attributed to it (i.e. papers without peer review are somehow "blog posts"). > What alternative is there that doesn’t have something resembling peer review, isn’t curated by third parties, doesn’t suffer from confirmation bias/groupthink, and doesn’t leave it up to authors to sift through 100s of papers of unknown quality a day to learn about what’s going on in the field? There's no free lunch. Nobody serious in any scientific field relies on peer review to determine paper quality. You have to read the paper in front of you and use your experience, which means you must read not just "100s of papers", but literally THOUSANDS of papers, in order to have any reasonable chance of being a credible evaluator. If you don't have this experience, you can't know the quality of a piece of scientific literature. That's just how it is.