6 ms·
Double-anonymous peer review reduces reviewer bias, finds three-year trial
- Qem 4y agoThat should be the default for a long time already.
- bilbo0s 4y agoMy cynical view is that the reason it isn't is because so many papers would not survive the review process. And all the consequences of that, up to and including impacts on getting degrees and even tenure.
- Red_Leaves_Flyy 4y agoThese people do crap work so instead of holding them, and ourselves, accountable for our crappy work, we’re going to lower our standards and hope the public doesn’t notice? What happens when the public starts noticing the consequences of bad research having tangible impacts on the real world? Just look outside, the anti science rhetoric is loud among many groups.
- SubiculumCode 4y agomeh. Experts in the field often know who is pursuing what lines of inquiry.
- m000 4y agoExperts may indeed "de-anonymize" a paper if they want. But: (a) not all of your assigned reviewers may be such experts, (b) experts often delegate some reviews to their PhDs/postdocs who are more susceptible to being influenced by author's name/affiliation.
- bilbo0s 4y ago(b) is especially prevalent.
- SubiculumCode 4y agohonestly feel like post-docs / grad students would be less influenced. They often don't have personal connections with senior research fellows at other institutes, and don't gaf.
- Overtonwindow 4y agoThis is an excellent step in the right direction, but will it help reduce the absurd volume of studies and the "publish or perish" paradigm?
- anamexis 4y agoI don't see why it would. It doesn't change the incentives.
- xiphias2 4y agoBy the time that problem is solved, GPT-8 will be out, and even GPT-4 can be used to help reduce the number of studies (making a table of the studies and asking questions about it). Reproducibility is getting better, but still a huge problem (we can't forget the billions of dollars of funding that went to Alzheimer for a scam paper)
- mike_hearn 4y agoIs reproducibility getting better? The last semi-rigorous treatment of the topic I saw suggested there was almost no change over the past decade, at least in the social sciences. It did however use a proxy for reproducibility.
- xiphias2 4y agoSocial sciences are inherently political: facebook, tiktok, tinder and onlyfans have all the resources and data to help people getting less depressed (for onlyfans it would be just shutting themselves down, as it's hard for me to think that there's any honest real human connection going on there, which makes both sides depressed over long term). Social workers are just trying to reverse some of the damage companies and governments are doing, but I don't think it will help, even I honestly think they are generally good people doing their best. Maybe disrupting the ad model that just started recently can help, but I believe social studies have no chance to fight against these things.
- mike_hearn 4y ago
- xhkkffbf 4y agoSome fields like Computer Science have had double blind review for some time. Others, not so much. Still, smart reviewers know who is working in which area. They can usually identify the other person pretty quickly. It's not always 100% accurate, but the people who you want doing the reviewing will be the ones who know the others in the field pretty well.
- floren 4y agoOr a review will say "Why didn't you cite Foo, an important work in the field?" and then you look at the committee list on the conference site and see that Dr Johnson worked on Foo. It's happened to me, but what can you do.
- quelltext 4y agoI don't know. I can understand that as a reviewer, if you see work that relates closely to your own prior work, you would be inclined to point the authors to that work. They may simply have missed it or failed to realize the relationship before so blindly saying "it's fine I will just ignore that they skipped over my own work" when it is in fact related doesn't seem like the right approach. If it really is unrelated I doubt anyone feels the need to point to their own research.
- scythe 4y agoWhen I worked for MDPI (please lower your pitchforks), we had a guideline, IIRC, that the reviewers of a paper were supposed to not be collaborators with each other, in addition to the usual requirement that they not be collaborators with the authors. ("Collaborators" meant coauthorship within five years ago). In principle, this means that if a citation was necessary and you were an author on it, you could hope that your co-reviewer(s) would recommend it. In practice, reviewers did sometimes request citations of their own works. As an editor you could reject the whole review report, but you were on deadlines and quotas and you typically had a hard enough time finding reviewers anyway. In my experience, most of the practices that researchers gripe about in MDPI are results of onerous working conditions rather than any kind of laxity in the culture; everyone was regularly encouraged to be serious about scientific integrity and seek out the best reviewers, and practically all of the people I worked with were quite intelligent and concerned about doing a good job, but it was also an unspoken expectation to work after hours, Saturdays, to always be sending an email, etc; exhausted people make mistakes.
- superb-owl 4y agoA friend of mine was sexually assaulted by a man who would be a very likely reviewer for her papers. It basically forced her out of the field. This would be an important step forward.
- flangola7 4y agoThat's horrible
- hedora 4y agoThat’s horrible, but she could have just listed him as a conflict when submitting (at least in CS; I assume all fields allow authors to disclose reviewers with conflicts). In the worst case, a PC chair would occasionally ask about it, so it would follow him for years.
- saghm 4y agoGiven that this person already has shown they don't care about harming others, it seems reasonable to be worried about retaliation if they found out they were being listed as a conflict. I'm not sure about anyone else, but I don't think I'd personally be willing to bet my safety on the hope that putting someone down as a conflict wouldn't leak somehow; all it would take is one person to not realize the severity of the reason they were listed to get curious and ask their colleague in their field if they knew why they were listed as a "conflict".
- flangola7 4y agoMaybe if she was "just" willing to risk her physical safety.
- mike_hearn 4y agoI scan read the paper. • It seems OK. Large sample size, fairly simple statistical treatment, clear and (mostly) honest writeup of their results. • About half of reviewers could correctly guess who the authors were despite the double blinding. • The motivation was the widespread belief in their field that women are discriminated against by peer review. In reality they found a slight bias in favor of women. (Their abstract and press release says they found no bias, but on p11 they say differently). • They also found no racism. Review results for people from non-western countries didn't change. • The difference is all due to a strong positive bias towards researchers from rich countries. This is probably "prestige bias", but can also just be that people go softer on people they know already because the reviewers are mostly western. • Optional double blinding will therefore be ineffective, it has to be mandatory.
- WFHRenaissance 4y agoOkay, but now do it for PRs.
- entropicdrifter 4y ago"This isn't in the code style of <Reviewer number 4>. Please redesign from first principles and rewrite your entire project to reflect the preferred coding style"