15 ms·

We shouldn’t take peer review as the ‘gold standard’

224 points by danso 7y ago

pacbard 7y ago
What would be the alternative to peer review? White papers on “pop” topics? Opinion pieces by think tanks? The ideal of peer review is pretty much the gold standard of scientific work. By that, I mean the idea that your work should undergo rigorous scrutiny by your peers before it gets accepted for publications. As any other human system, though, the process is still subject to politics and biases. My hope is that a truly open review system might one day democratize science and ameliorate the issues with the current review system. For example, it would be cool to put all the revisions, reviews and authors’ comments in the open as an online appendix. That will show how the paper changes as it moves trough the process, the issues that the reviewers brought up, and how the authors responded to them. It will also help show if there is systemic bias against certain researchers or topics.
- buboard 7y ago
  E-life is already doing peer review right. Now you need get others on board
- nine_k 7y ago
  I think that publication should not be gated by peer review. Release early, release often, publish experimental setup and what you are looking for prior to the actual experiment, etc. We have arxiv now; the whole process could open up even more, slightly closer to research in the open, like open-source development in the open. This is to make peer review easier. Then what is currently "publication" becomes what it should be, in the sense of adding value: curation, filtering the most interesting, least dubious results.
  - deleted 7y ago
    [deleted]
  - chrisseaton 7y ago
    arxiv is actually less open than most traditional venues. Anyone can submit to most conferences and journals. Anyone at all! No PhD or reputation needed and it’s blind. All that matters is the quality of the research. To submit to arxiv you need to be approved by someone as a legitimate researcher, or beg for reviews from people you’ve never met without any anonymity. It’s somewhat a step backwards in some ways.
    - rocqua 7y ago
      I wonder which is more likely though. An unpublished and non-PhD author being published in a journal after submission, or that same author being allowed to publish on arXiv. My guess would be on arXiv. If you really have a decent breakthrough, I'm pretty sure your best bet is to directly contact some relevant people in the field.
    - joshuamorton 7y ago
      Which is more likely: you get a paper, yet unreviewed for formatting and style by any expert in the field, accepted into a conference or journal, or you get someone with field expertise to review for format and style, and they let you post to arxiv?
    - human_scientist 7y ago
      > To submit to arxiv you need to be approved by someone as a legitimate researcher, or beg for reviews from people you’ve never met without any anonymity. I did not have to do something like this.
      - chrisseaton 7y ago
        Well that’s good for you isn’t it! What about everyone else? Not everyone is automatically approved because of things like their email address. That’s my point - they focus on reputation and credentials rather than the blind value of the work. It’s less accessible.
        317070 7y ago
  - amelius 7y ago
    Yes all results should get published. Peer review should only affect ranking/discoverability.
  - unholiness 7y ago
    You're not the only one to comment this, but from what I know per review is a long iterative process, wher reviewers raise objections and submitted respond with edits. So, what you're suggesting seems a lot like shipping code before code review, and seems to have many of the same problems. What do you do in the long period before critical issues have been addressed? What do you do if the submitter just says "fuck you, my code works, I have more important things to do"?
    - Fomite 7y ago
      This. I have definitely dug my heels in on some peer reviewed papers where I'm rather glad I was "gatekeeping".
- tptacek 7y ago
  The article doesn't propose we do away with peer review, but rather that lay audiences stop pretending that peer review is a "gold standard" by which the quality of research can be judged. It's "gold standards" the author proposes to eliminate.
  - pacbard 7y ago
    I get that. Sorry my reply wasn’t very clear. The article gives examples of how politicians and policymakers have “weaponized” peer review to silence critics. The author proposes moving away from peer review (and the gold standard it sometimes represents) as a way to solve this issue. In my reply, I was just pointing out that peer review still plays an important role in the development of knowledge and I was wondering if alternative systems that could replace it would actually be feasible. That lead me somewhere off topic. By the way, the “weaponization” of peer review is nothing new. You can see it used in the past to discredit research linking smoking or lead to health issues. You can also see of the possible risks or benefits (depending on your point of view) of rejecting peer reviewed research in the anti-vax movement.
    - tptacek 7y ago
      The author does not propose moving away from peer review and, in the conclusion, says it's a necessity for science.
- tomp 7y ago
  Personally, I view any study in the less scientific fields (i.e. those that routinely accept p < 0.05 as "proven") as suspect, unless it's preregistered, has full data available for the public to check, and has p < 0.01. (I'm neither aware of any such study, nor have I looked for it... I'm just saying that that should be the minimal acceptable standard for e.g. psychology and medicine.)
- JumpCrisscross 7y ago
  > What would be the alternative to peer review? Replication. If science has a gold standard, it is this.
  - tptacek 7y ago
    Replication is slow and can be extremely expensive. It's a good thing, but it doesn't serve the same function peer review does.
    - tomlue 7y ago
      It does not have to be slow in all domains. In my field there are many modeling papers. Journals could require that all data and code be open access, or at least define a submission type where this is the case. You could even automate the process of running submitted containers / packages.
      - petschge 7y ago
        All that would prove is that the the code that implements a buggy version of a model gives consistently wrong results. Actual replication requires that somebody else implements the code independently. And that both implementations are checked over a reasonably wide range of parameters over which the model should be valid. Actual replication is much more effort and consequently slower than just a "docker run".
      - dagw 7y ago
        While I agree that this should be mandatory in many cases, it doesn't prove too much. I've experienced many cases where the open code worked fine with the test data provided, but failed completely when I tried it with my own real world data.
    - JumpCrisscross 7y ago
      > Replication is slow and can be extremely expensive Gold standards are identified as such because they’re the best. Not everything needs to meet the gold standard to be debatable. Peer review is adequate for further research, but perhaps not policy initiatives. An unreviewed paper is enough to start simple inquiries. Et cetera
- SubiculumCode 7y ago
  Some journals do this. It is an interesting approach. I am however ambivalent, or even perhaps against allowing public comments directly into a paper. We've all experienced, no doubt, the a naive but loud internet com-mentor spreading FUD on comment sections and social media on topics they simply know nothing about, and do so prolifically. It is nice to think that good arguments will float to the top, but often it is the loudest and least nuanced argument, right or wrong, which gets taken up.
- aeternus 7y ago
  I would love to see an open review system like that. Comments out in the open, along with the experiments and experimental data which supports or does not support the work. Science is all about experimentation, yet we currently measure papers by number of citations rather than by experimental support.
- o09rdk 7y ago
  I'm an editor of a journal that has one of (the?) highest IFs in its field, among other things. Lots of experience with scientific publishing. I've had lengthy discussions about this with my colleagues, and I don't think the problem is with peer review per se, it's with how peer review, and by extension, scientific literature, is interpreted and utilized. It's not just with the lay community, it's among scientists themselves, as well as scientific consumers as in engineering, healthcare, etc. People now treat science, and scientific publications, as if it's a farm or factory. The produced commodity is the peer-reviewed paper. Once research, via a paper, is peer-reviewed, it's blessed as the truth, and if it's not, it's worthless. The goal of many institutions is now not to produce sound research per se, but to receive money (largely due to indirect cost income) to produce the commodity of peer-reviewed paper. Once a paper is accepted as in press, it can be cited as if it were truth, which provides a building block for something else. The truth per se doesn't matter, only the citability of it as peer-reviewed. I realize this sounds a bit conspiratorial but I do think this is basically how it works at this point in time, at least in certain fields. Peer-review is overvalued, as are single papers, and contributions of particular researchers in many cases. That doesn't mean they don't have any value, just that their real value is probably much less than we assume. I agree that peer review can be improved a lot, and will likely always have an important role in some form, but I don't think fundamental problems in the field will go away unless there's a change in perspective on the real scientific process as a whole, to something more nuanced and gradual.
buboard 7y ago
Another paywall ad. Can we please be done with these? They're worse than tracking ads.
- rolltiide 7y ago
  Truely, I'm really curious who here subscribes to Washington Post, NY Times, LA Times, Medium etc Is this really commonplace?
  - ekc 7y ago
    Medium, The New York Times, and Washington Post all have easily circumventable paywalls, though I can't vouch for the other one you listed.
    - rolltiide 7y ago
      Is that really the point? They've started detecting incognito mode and won't show content, all you have to do is just use a completely different browser program like Safari if you normally use Chrome, but its different and changes for every site
      - jaredsohn 7y ago
        You can also use a different Chrome profile. FYI, Chrome is making changes to make it harder to detect incognito mode. https://nakedsecurity.sophos.com/2019/07/22/chrome-76-blocks-websites-from-detecting-incognito-mode/ https://nakedsecurity.sophos.com/2019/07/22/chrome-76-blocks...
    - buboard 7y ago
      This website asked me to disable my ad blocker, then showed me an ad for their subscription. I guess they follow the footsteps of porn sites again ...
  - bootlooped 7y ago
    I subscribe to two of the sites you listed because it doesn't cost that much and it is more convenient to pay for it than deal with fidgeting with incognito mode, erasing cookies, using a 3rd party site to read, etc...
elihu 7y ago
Test results are very easy to fake. The "gold standard" is whether outside groups can run the same tests and reproduce the same results. Peer review doesn't mean much more than "this paper seems interesting and is well-written and properly formatted."
- Retric 7y ago
  Self consistent is another major check. A lot of math and similar errors are caught in peer review.
  - petschge 7y ago
    Peer review isn't (and never was meant to be) a golden seal declaring a result to be unchangeable truth. It is a process that makes sure that at least one competent colleague has read it and caught all the embarrassing Monday morning mistakes that might have been in there. This is done before wider publication to avoid that the 100 people in your field all stumble over the same stupid little mistake and can spent their time thinking about the actual content. Think of it as the type checking by the compiler or a code review by somebody on your team, NOT a papal announcement of unchangeable dogma.
- dnautics 7y ago
  Depends on the journal, for example https://en.m.wikipedia.org/wiki/Organic_Syntheses https://en.m.wikipedia.org/wiki/Organic_Syntheses
- mxcrossb 7y ago
  I disagree. The gold standard is if outside groups subject your findings to new experiments and reproduce the same results.
  - tfha 7y ago
    Well, there's replicating the experiment and then there's challenging the conclusion. Those are two separate ideas, and are actually fairly independent.
  - 7y ago
tptacek 7y ago
Just to clarify: this article is against gold standards, not against peer review. It concludes by observing that peer review is a necessity. The argument it raises is that peer review doesn't mean what lay audiences think it means. I think they're clearly correct, and that even on HN, in a community that (roughly) prides itself on being scientifically literate, there are broad misunderstandings of what peer review means (during the bogus "Sokal Squared" hoax, for instance, many commenters implied that peer review prior to publication was meant to encompass replication). Also, while I'm not a "scientist", I've gotten to do some peer review work for ACM and Usenix, and even in the little bit of review I did, I seen some shit. There is much less formality and oversight to review than you might expect.
- ZhuanXia 7y ago
  Is there any proof peer review has increased scientific productivity? It is relatively new, and I don't see any reason prima facie why it would work better than an editorial teaming willing to delegate when necessary. Anecdotally, I see lots of academics complain about very important results being unable to make it through peer review. Ralph Merkel's early groundbreaking cryptography work was suppressed for years by unthinking peer review. From Merklel: >My first paper (and, in fact, the first paper) on public key cryptography was submitted in 1974 and initially rejected by an unknown "cryptography expert" because it was "...not in the main stream of present cryptography thinking...."
  - sokoloff 7y ago
    Isn’t peer review at most journals done by “an editorial team delegating as necessary”? I married into the field, didn’t work in it, but that’s what I’ve observed from the side.
    - tptacek 7y ago
      The ACM and Usenix peer review work I did involved rotating teams of volunteers and very little guidance from the publication/venue.
tomlue 7y ago
I'm a recent PhD graduate that focused heavily on publication and collected ~20 publications. I think the situation with peer review is worse than discussed here. Peer review: 1. dramatically reduces the pace of progress 2. exacerbates publication bias 3. creates a false sense of accomplishment 4. creates a false meritocracy 5. creates many perverse incentives (3) and (4) unfortunately wrap into grant financing as well. The merit of your research isn't measured on its impact on the human condition, but on it's 'impact' factor (a measure derived from your publications). The gatekeeper problem here is pernicious as well. If you become a highly cited author your ability to get/maintain financing improves. You also become a gatekeeper as a peer reviewer. Which means that you are now strongly incentivized to accept papers from people who cite your work or align themselves with you, and reject everything else. What is absolutely amazing here, is that the peer review process is opaque. It is my belief that if you knew who reviewed which papers you would quickly discover that mild to severe abuse of peer review is the norm, not action by a handful of bad actors. This is because the entire academic reward system is wrapped into the process. Getting your name on a big paper can have lifelong ramifications on your ability to get grants, start companies, do consulting work, etc. Peer reviewers should probably get paid for the work. If they don't get paid then their incentive to do review must come from somewhere else, vague notions of improving the field don't cut it. Peer review should obviously be transparent. Some people might be uncomfortable signing their name to a paper rejection, but its time to get over that. A small payment might help reviewers overcome this discomfort. It is bizarre that peer reviewers don't get paid. Peer review is valuable work if done right, and without payment all the reward of being a peer reviewer comes for the wrong reasons. Finally, I agree with some other comments. Publication should not be contingent on peer review, it should come first. This would increase the pace of progress, reduce publication bias, reduce the false meritocracy, reduce the ability for bad actors to censor research, and more. The cost would be a larger number of publications, but perhaps this would help people realize that many of the publications coming out right now are of little value. Transparency is badly needed in many facets of academic research. My company made a site that helps bring transparency into literature review (sysrev.com).
- sgillen 7y ago
  I totally agree with the transparency thing. I would love to see something like open review (https://openreview.net/forum?id=ryQu7f-RZ https://openreview.net/forum?id=ryQu7f-RZ) become more common, and even have something like this replace the traditional peer review system. We would all have to think long and hard about how to align incentives to get the desired results.As a first pass I think hiring some amount of people to review every paper, and then allowing anyone who wants to come by and leave their comments as well might be a place to start. I think some sort of reputation system similar to stack overflow might be a motivator as well.
wtvanhest 7y ago
I'm going to propose something that is unconventional today, but I believe will be standard practice at somepoint. Overtime we have shown that many mistakes are made in labs and we know that many experiments are not replicable. I think its time that we ask researchers using public funds to film their entire experiment from beginning to end. Every pipette, every prep etc. Peer review should also have a video audit. Then a replication in another lab. The cost of ensuring results are what they say they are would be offset by less wasted research dollars on false rabbit holes.
- sgillen 7y ago
  I do actually agree that we should apply more funding to replicating results. I disagree that a video log is an efficient way to ensure replicatability. In many disciplines this is not useful at all (math, high energy physics). And in most others, like your biology example, I think it adds an undue burden without much gain.
- fwip 7y ago
  What evidence do you have that suggests that the cost of doing a video audit (an expert carefully watching hundreds of hours of experimentation) and a replication study for every submitted article would be cheaper than the occasional paper that leads to a second experiment that doesn't pan out?
- Nemo_bis 7y ago
  Are you trying to re-invent https://en.wikipedia.org/wiki/Reproducibility_Project https://en.wikipedia.org/wiki/Reproducibility_Project ?
  - wtvanhest 7y ago
    No. Reproducibility requires double the investment and may also have the same or different mistakes. A video log would help researchers find errors and confirm results. Video is simply a better solution.
dan-robertson 7y ago
A curious thing that’s happened in some parts of mathematics is that formal peer review, needing coordination of journals, editors, and reviewers, can happen after many people have already read the paper (effectively informally peer reviewing it but with fewer comments), and even after people have cited the paper. This peer review then seems to not serve much purpose. An example is the recent first finite prime gap paper where the result was quickly strengthened several times by other authors long before the paper made it into any journal. Another example here: https://mobile.twitter.com/adrianprw/status/1156534906618597377 https://mobile.twitter.com/adrianprw/status/1156534906618597...
mathgenius 7y ago
> A 2015 study of 1,000 medical journal submissions found that of the papers that were eventually published, the 14 that became the most frequently cited were initially rejected. Groundbreaking studies by Sir Frank MacFarlane Burnet, Rosalind Yalow, Baruch Blumberg and others were rejected by peer reviewers, yet later led to Nobel Prizes. Yup, if you are actually doing good work it is likely to be so different to what is going on that people have a hard time evaluating it.
- dmurray 7y ago
  If you are actually doing ground-breaking work. There's plenty of scope for doing good work that is boring and mainstream and still high quality enough to be published.
  - analog31 7y ago
    Indeed, my work was of the latter type. It was rejected by Physical Review Letters but accepted by another journal. One feature of peer review that's overlooked, is that there are different journals with different standards, and you can choose a journal that fits what you're trying to accomplish.
- magicalhippo 7y ago
  I can't read the article, but being rejected doesn't have to mean it's bad work. My mom has helped many PhD students take their degree over the years, and in the process helped with their papers. Several have been rejected, some multiple times. Frequently it was down to presentation issues (the student had not explained the certain things clearly enough or similar), or there was disagreement about how strong conclusions one could draw from the results. For example, recently a paper was rejected because the experimenters had forgotten or overlooked an important detail which meant they couldn't control for a certain variable. One of the reviewers picked this up, and rejected the paper because the findings could possibly be explained by this uncontrolled variable. So they had to resubmit with a weaker conclusion (this is when my mom got involved). In this case the review process was harsh but fair. The experimenters had goofed, and the reviewer caught it. Of course when doing ground breaking experiments, I guess the process might not be optimal. But the majority of scientists are not doing that.
scythe 7y ago
We don’t. Peer review is a very helpful component of a system that slowly filters and refines academic work. It’s like the first air intake stage of a jet turbine.
cassowary37 7y ago
For everyone complaining about peer review, another alternative to open review is paid reviewers who focus on a particular topic. For example, I'm an editor at a journal where statistical reviewers are paid and evaluated, while the remaining (content) reviewers are not. This ensures that we have a rigorous review of the statistical methods, at least. I do think payment increases the reviewers' responsibility and conscientiousness, and the overall quality of the subsequent articles. (nb This probably has been tested somewhere.) BUT - paid reviews are not feasible for many journals, especially open access/low-cost journals where margins may be thin. (Conversely, Elsevier should be able to pay their reviewers in gold bullion, rather than taking advantage of the scientific community's altruism, but that's another conversation).
jedberg 7y ago
Somewhat tangential, but are there any journals that will only accept a paper if someone unrelated to the author can replicate the study? Or a journal of only replicated studies?
- Nemo_bis 7y ago
  Sure. My favourite is IPOL Journal · Image Processing On Line https://www.ipol.im/ https://www.ipol.im/, which is doing it since 2011: «IPOL is a research journal of image processing and image analysis which emphasizes the role of mathematics as a source for algorithm design and the reproducibility of the research. Each article contains a text on an algorithm and its source code, with an online demonstration facility and an archive of experiments. Text and source code are peer-reviewed and the demonstration is controlled. IPOL is an Open Science and Reproducible Research journal». More recently, some megajournals have joined the trend and are trying to expand it, particularly eLife: https://elifesciences.org/labs/ad58f08d/introducing-elife-s-first-computationally-reproducible-article https://elifesciences.org/labs/ad58f08d/introducing-elife-s-... (also covered in https://www.nature.com/articles/d41586-019-00724-7 https://www.nature.com/articles/d41586-019-00724-7 ). For some context see: * https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis * https://www.frontiersin.org/articles/10.3389/fninf.2017.00069/full https://www.frontiersin.org/articles/10.3389/fninf.2017.0006...
neonate 7y ago
http://archive.is/UCSof http://archive.is/UCSof
typon 7y ago
I think the peer review system is ripe for, as SV people call it, "disruption". Even something like an app that you submit papers to, which then anonymizes them, standardizes formatting and presents them to randomly selected experts for review in a certain time period would go a long way compared to the current process. In my lab at a prestigious Canadian university, the papers' authors names were visible as well as the reviewers names because any back and forth post review would happen through email. I think journals and universities need to buy into a system that standardizes this process so at least these basic requirements of impartiality can be met.
- tastroder 7y ago
  While I'd apreciate a little fresh air in the process as well... > an app that you submit papers to, which then anonymizes them, standardizes formatting and presents them to randomly selected experts for review in a certain time period Except for that it's not an app (that would be kind of pointless, why would I want that on a phone instead of a webapp), that's pretty much the standard in my field. Anonymization is done through the authors (if the conference/journal wants it to be) and ensured by the conference submission system. Formatting is standardized anyway. The problem is the "randomly selected experts" part, for many (most?) subfields suitable reviewers are a scarce resource, to the point that in quite a few double-blind processes I've seen one or both parties could recognize each other's work or review. I've always found openreview [1] a nice way to at least make the process public, especially when anonymity is an illusion anyway. https://openreview.net/ https://openreview.net/
  - typon 7y ago
    By app I meant software not a phone app. Secondly, I think the problem of having few experts in the field is a legitimate problem but I don't think the status quo is a good solution either. I think it erodes trust in the scientific community which is very hard to build back up.
    - tastroder 7y ago
      I think we need an overall shift in incentives in academia before that will change. Popular conferences see an all time high in submissions, though I do not have data on that I'm sure Journals face similar problems these days. I feel like that's not an aspect that will get better in the short term. On the app/phone app thing: Ah, fair enough, sorry about that.
chr1 7y ago
The whole scientific publishing field is still stuck in previous century. We desperately need a github for science, where any reader would be a reviwer, who would be able to rate the article and comment on it, and then journals would use these ratings combined with credibility rankings of reviewers to create indexes of interesting new article-repositories.
- HNisCurated 7y ago
  I have people citing research with useless AMA standard. If you want both parties to agree, they need to see the Data. Most studies become uninteresting after you read their terrible data.
  - chr1 7y ago
    Absolutely agree, articles without data and without the sourcecode, are not science, and that's another reason why we need github for science.
- Nemo_bis 7y ago
  More on this idea: http://bjoern.brembs.net/2015/04/what-should-a-modern-scientific-infrastructure-look-like/ http://bjoern.brembs.net/2015/04/what-should-a-modern-scient... http://bjoern.brembs.net/2018/01/why-academic-journals-need-to-go/ http://bjoern.brembs.net/2018/01/why-academic-journals-need-...
p1necone 7y ago
If peer review in Academia is anything like code review in corporate software dev it's probably next to worthless.
- jdnenej 7y ago
  Peer review never seems to catch actual bugs but does give you helpful tips like "we already have a function that does this" or "the language has a feature for this"
  - p1necone 7y ago
    Oh code review done properly is really great - I'm just jaded by repeatedly seeing code go up and getting that green tick with no comments and knowing that the reviewer barely even attempted to read it.
fastaguy88 7y ago
The idea that in scientific publication, more is better, makes little sense to me as a practicing scientist. The scientific literature is already filled with papers with conclusions that are not well supported by the data, and that is after 50-80% of submissions are rejected. To argue that publication should have fewer restrictions, one needs to show that the modest number of papers that are improperly rejected by every journal (not just the first submission) equals or outnumbers the overwhelming number of submissions that should not be published because they reflect a misunderstanding of their field or a misinterpretation of the data. For many (most) scientists, the problem is not that important results are unpublished, the problem is that it is almost impossible to keep up with the current gated literature, particularly when that literature is full of mistakes. While it is certainly true that some reviewers are biased and some ill-informed, most of the time papers are rejected because the authors did not communicate well (for papers that should be published) or because the authors did not understand that their data did not support their conclusion (papers that should be rejected). The scientific literature needs a better signal to noise ratio, not more noise.
- Nemo_bis 7y ago
  The rejection of a submission does cancel a work, it merely shifts elsewhere. The question is whether peer review gives the right incentives; many suspect that publication venues with an ostensibly stricter peer review actually encourage the production of more noise, not less. See https://www.frontiersin.org/articles/10.3389/fnhum.2013.00291/full https://www.frontiersin.org/articles/10.3389/fnhum.2013.0029...
alexmlamb 7y ago
"Studies have shown that journal editors prefer reviewers of the same gender, that women are underrepresented in the peer review process, and that reviewers tend to be influenced by demographic factors like the author’s gender or institutional affiliation. " Most venues are double blind, so reviewers wouldn't know about the latter things. Having fewer women in the reviewer pool is unfortunate, but it's hard to see how it would effect the mean review quality in a direct way assuming that men and women are equally good at reviewing (maybe in an indirect way it could harm reviews by increasing the load per-reviewer).
- bmc7505 7y ago
  Not sure if this is the case, but I'd like to play devils advocate. While most venues are double blind, it is still possible there is a subconscious gender bias. Many studies have shown that gender can be readily detected from anonymized authors based style alone. https://www.aclweb.org/anthology/W11-0310 https://www.aclweb.org/anthology/W11-0310 http://cs224d.stanford.edu/reports/BartleAric.pdf http://cs224d.stanford.edu/reports/BartleAric.pdf https://www.sciencedirect.com/science/article/pii/S1877050916326849 https://www.sciencedirect.com/science/article/pii/S187705091...
- xioxox 7y ago
  Whether venues are double blind is extremely field dependent. In my field no journals are to my knowledge. However, many people use an initial rather than a first name, so it's difficult to assign gender unless you already know the person.
- abdullahkhalids 7y ago
  Depends on the field. In my field, I know pretty much the twenty groups who have the expertise to review my work and every paper of mine I could identify with high certainity from the referee's comment which group they belonged to.
known 7y ago
Is https://en.wikipedia.org/wiki/Wisdom_of_the_crowd https://en.wikipedia.org/wiki/Wisdom_of_the_crowd better?

224 points by danso 7y ago

What would be the alternative to peer review? White papers on “pop” topics? Opinion pieces by think tanks? The ideal of peer review is pretty much the gold standard of scientific work. By that, I mean the idea that your work should undergo rigorous scrutiny by your peers before it gets accepted for publications. As any other human system, though, the process is still subject to politics and biases. My hope is that a truly open review system might one day democratize science and ameliorate the issues with the current review system. For example, it would be cool to put all the revisions, reviews and authors’ comments in the open as an online appendix. That will show how the paper changes as it moves trough the process, the issues that the reviewers brought up, and how the authors responded to them. It will also help show if there is systemic bias against certain researchers or topics.

buboard 7y ago
E-life is already doing peer review right. Now you need get others on board
nine_k 7y ago
I think that publication should not be gated by peer review. Release early, release often, publish experimental setup and what you are looking for prior to the actual experiment, etc. We have arxiv now; the whole process could open up even more, slightly closer to research in the open, like open-source development in the open. This is to make peer review easier. Then what is currently "publication" becomes what it should be, in the sense of adding value: curation, filtering the most interesting, least dubious results.
- deleted 7y ago
  [deleted]
- chrisseaton 7y ago
  arxiv is actually less open than most traditional venues. Anyone can submit to most conferences and journals. Anyone at all! No PhD or reputation needed and it’s blind. All that matters is the quality of the research. To submit to arxiv you need to be approved by someone as a legitimate researcher, or beg for reviews from people you’ve never met without any anonymity. It’s somewhat a step backwards in some ways.
  - rocqua 7y ago
    I wonder which is more likely though. An unpublished and non-PhD author being published in a journal after submission, or that same author being allowed to publish on arXiv. My guess would be on arXiv. If you really have a decent breakthrough, I'm pretty sure your best bet is to directly contact some relevant people in the field.
  - joshuamorton 7y ago
    Which is more likely: you get a paper, yet unreviewed for formatting and style by any expert in the field, accepted into a conference or journal, or you get someone with field expertise to review for format and style, and they let you post to arxiv?
  - human_scientist 7y ago
    > To submit to arxiv you need to be approved by someone as a legitimate researcher, or beg for reviews from people you’ve never met without any anonymity. I did not have to do something like this.
    - chrisseaton 7y ago
      Well that’s good for you isn’t it! What about everyone else? Not everyone is automatically approved because of things like their email address. That’s my point - they focus on reputation and credentials rather than the blind value of the work. It’s less accessible.
      - 317070 7y ago
- amelius 7y ago
  Yes all results should get published. Peer review should only affect ranking/discoverability.
- unholiness 7y ago
  You're not the only one to comment this, but from what I know per review is a long iterative process, wher reviewers raise objections and submitted respond with edits. So, what you're suggesting seems a lot like shipping code before code review, and seems to have many of the same problems. What do you do in the long period before critical issues have been addressed? What do you do if the submitter just says "fuck you, my code works, I have more important things to do"?
  - Fomite 7y ago
    This. I have definitely dug my heels in on some peer reviewed papers where I'm rather glad I was "gatekeeping".
tptacek 7y ago
The article doesn't propose we do away with peer review, but rather that lay audiences stop pretending that peer review is a "gold standard" by which the quality of research can be judged. It's "gold standards" the author proposes to eliminate.
- pacbard 7y ago
  I get that. Sorry my reply wasn’t very clear. The article gives examples of how politicians and policymakers have “weaponized” peer review to silence critics. The author proposes moving away from peer review (and the gold standard it sometimes represents) as a way to solve this issue. In my reply, I was just pointing out that peer review still plays an important role in the development of knowledge and I was wondering if alternative systems that could replace it would actually be feasible. That lead me somewhere off topic. By the way, the “weaponization” of peer review is nothing new. You can see it used in the past to discredit research linking smoking or lead to health issues. You can also see of the possible risks or benefits (depending on your point of view) of rejecting peer reviewed research in the anti-vax movement.
  - tptacek 7y ago
    The author does not propose moving away from peer review and, in the conclusion, says it's a necessity for science.
tomp 7y ago
Personally, I view any study in the less scientific fields (i.e. those that routinely accept p < 0.05 as "proven") as suspect, unless it's preregistered, has full data available for the public to check, and has p < 0.01. (I'm neither aware of any such study, nor have I looked for it... I'm just saying that that should be the minimal acceptable standard for e.g. psychology and medicine.)
JumpCrisscross 7y ago
> What would be the alternative to peer review? Replication. If science has a gold standard, it is this.
- tptacek 7y ago
  Replication is slow and can be extremely expensive. It's a good thing, but it doesn't serve the same function peer review does.
  - tomlue 7y ago
    It does not have to be slow in all domains. In my field there are many modeling papers. Journals could require that all data and code be open access, or at least define a submission type where this is the case. You could even automate the process of running submitted containers / packages.
    - petschge 7y ago
      All that would prove is that the the code that implements a buggy version of a model gives consistently wrong results. Actual replication requires that somebody else implements the code independently. And that both implementations are checked over a reasonably wide range of parameters over which the model should be valid. Actual replication is much more effort and consequently slower than just a "docker run".
    - dagw 7y ago
      While I agree that this should be mandatory in many cases, it doesn't prove too much. I've experienced many cases where the open code worked fine with the test data provided, but failed completely when I tried it with my own real world data.
  - JumpCrisscross 7y ago
    > Replication is slow and can be extremely expensive Gold standards are identified as such because they’re the best. Not everything needs to meet the gold standard to be debatable. Peer review is adequate for further research, but perhaps not policy initiatives. An unreviewed paper is enough to start simple inquiries. Et cetera
SubiculumCode 7y ago
Some journals do this. It is an interesting approach. I am however ambivalent, or even perhaps against allowing public comments directly into a paper. We've all experienced, no doubt, the a naive but loud internet com-mentor spreading FUD on comment sections and social media on topics they simply know nothing about, and do so prolifically. It is nice to think that good arguments will float to the top, but often it is the loudest and least nuanced argument, right or wrong, which gets taken up.
aeternus 7y ago
I would love to see an open review system like that. Comments out in the open, along with the experiments and experimental data which supports or does not support the work. Science is all about experimentation, yet we currently measure papers by number of citations rather than by experimental support.
o09rdk 7y ago
I'm an editor of a journal that has one of (the?) highest IFs in its field, among other things. Lots of experience with scientific publishing. I've had lengthy discussions about this with my colleagues, and I don't think the problem is with peer review per se, it's with how peer review, and by extension, scientific literature, is interpreted and utilized. It's not just with the lay community, it's among scientists themselves, as well as scientific consumers as in engineering, healthcare, etc. People now treat science, and scientific publications, as if it's a farm or factory. The produced commodity is the peer-reviewed paper. Once research, via a paper, is peer-reviewed, it's blessed as the truth, and if it's not, it's worthless. The goal of many institutions is now not to produce sound research per se, but to receive money (largely due to indirect cost income) to produce the commodity of peer-reviewed paper. Once a paper is accepted as in press, it can be cited as if it were truth, which provides a building block for something else. The truth per se doesn't matter, only the citability of it as peer-reviewed. I realize this sounds a bit conspiratorial but I do think this is basically how it works at this point in time, at least in certain fields. Peer-review is overvalued, as are single papers, and contributions of particular researchers in many cases. That doesn't mean they don't have any value, just that their real value is probably much less than we assume. I agree that peer review can be improved a lot, and will likely always have an important role in some form, but I don't think fundamental problems in the field will go away unless there's a change in perspective on the real scientific process as a whole, to something more nuanced and gradual.

buboard 7y ago

Another paywall ad. Can we please be done with these? They're worse than tracking ads.

rolltiide 7y ago
Truely, I'm really curious who here subscribes to Washington Post, NY Times, LA Times, Medium etc Is this really commonplace?
- ekc 7y ago
  Medium, The New York Times, and Washington Post all have easily circumventable paywalls, though I can't vouch for the other one you listed.
  - rolltiide 7y ago
    Is that really the point? They've started detecting incognito mode and won't show content, all you have to do is just use a completely different browser program like Safari if you normally use Chrome, but its different and changes for every site
    - jaredsohn 7y ago
      You can also use a different Chrome profile. FYI, Chrome is making changes to make it harder to detect incognito mode. https://nakedsecurity.sophos.com/2019/07/22/chrome-76-blocks-websites-from-detecting-incognito-mode/ https://nakedsecurity.sophos.com/2019/07/22/chrome-76-blocks...
  - buboard 7y ago
    This website asked me to disable my ad blocker, then showed me an ad for their subscription. I guess they follow the footsteps of porn sites again ...
- bootlooped 7y ago
  I subscribe to two of the sites you listed because it doesn't cost that much and it is more convenient to pay for it than deal with fidgeting with incognito mode, erasing cookies, using a 3rd party site to read, etc...

elihu 7y ago

Test results are very easy to fake. The "gold standard" is whether outside groups can run the same tests and reproduce the same results. Peer review doesn't mean much more than "this paper seems interesting and is well-written and properly formatted."

Retric 7y ago
Self consistent is another major check. A lot of math and similar errors are caught in peer review.
- petschge 7y ago
  Peer review isn't (and never was meant to be) a golden seal declaring a result to be unchangeable truth. It is a process that makes sure that at least one competent colleague has read it and caught all the embarrassing Monday morning mistakes that might have been in there. This is done before wider publication to avoid that the 100 people in your field all stumble over the same stupid little mistake and can spent their time thinking about the actual content. Think of it as the type checking by the compiler or a code review by somebody on your team, NOT a papal announcement of unchangeable dogma.
dnautics 7y ago
Depends on the journal, for example https://en.m.wikipedia.org/wiki/Organic_Syntheses https://en.m.wikipedia.org/wiki/Organic_Syntheses
mxcrossb 7y ago
I disagree. The gold standard is if outside groups subject your findings to new experiments and reproduce the same results.
- tfha 7y ago
  Well, there's replicating the experiment and then there's challenging the conclusion. Those are two separate ideas, and are actually fairly independent.
- 7y ago

tptacek 7y ago

Just to clarify: this article is against gold standards, not against peer review. It concludes by observing that peer review is a necessity. The argument it raises is that peer review doesn't mean what lay audiences think it means. I think they're clearly correct, and that even on HN, in a community that (roughly) prides itself on being scientifically literate, there are broad misunderstandings of what peer review means (during the bogus "Sokal Squared" hoax, for instance, many commenters implied that peer review prior to publication was meant to encompass replication). Also, while I'm not a "scientist", I've gotten to do some peer review work for ACM and Usenix, and even in the little bit of review I did, I seen some shit. There is much less formality and oversight to review than you might expect.

ZhuanXia 7y ago
Is there any proof peer review has increased scientific productivity? It is relatively new, and I don't see any reason prima facie why it would work better than an editorial teaming willing to delegate when necessary. Anecdotally, I see lots of academics complain about very important results being unable to make it through peer review. Ralph Merkel's early groundbreaking cryptography work was suppressed for years by unthinking peer review. From Merklel: >My first paper (and, in fact, the first paper) on public key cryptography was submitted in 1974 and initially rejected by an unknown "cryptography expert" because it was "...not in the main stream of present cryptography thinking...."
- sokoloff 7y ago
  Isn’t peer review at most journals done by “an editorial team delegating as necessary”? I married into the field, didn’t work in it, but that’s what I’ve observed from the side.
  - tptacek 7y ago
    The ACM and Usenix peer review work I did involved rotating teams of volunteers and very little guidance from the publication/venue.

tomlue 7y ago

I'm a recent PhD graduate that focused heavily on publication and collected ~20 publications. I think the situation with peer review is worse than discussed here. Peer review: 1. dramatically reduces the pace of progress 2. exacerbates publication bias 3. creates a false sense of accomplishment 4. creates a false meritocracy 5. creates many perverse incentives (3) and (4) unfortunately wrap into grant financing as well. The merit of your research isn't measured on its impact on the human condition, but on it's 'impact' factor (a measure derived from your publications). The gatekeeper problem here is pernicious as well. If you become a highly cited author your ability to get/maintain financing improves. You also become a gatekeeper as a peer reviewer. Which means that you are now strongly incentivized to accept papers from people who cite your work or align themselves with you, and reject everything else. What is absolutely amazing here, is that the peer review process is opaque. It is my belief that if you knew who reviewed which papers you would quickly discover that mild to severe abuse of peer review is the norm, not action by a handful of bad actors. This is because the entire academic reward system is wrapped into the process. Getting your name on a big paper can have lifelong ramifications on your ability to get grants, start companies, do consulting work, etc. Peer reviewers should probably get paid for the work. If they don't get paid then their incentive to do review must come from somewhere else, vague notions of improving the field don't cut it. Peer review should obviously be transparent. Some people might be uncomfortable signing their name to a paper rejection, but its time to get over that. A small payment might help reviewers overcome this discomfort. It is bizarre that peer reviewers don't get paid. Peer review is valuable work if done right, and without payment all the reward of being a peer reviewer comes for the wrong reasons. Finally, I agree with some other comments. Publication should not be contingent on peer review, it should come first. This would increase the pace of progress, reduce publication bias, reduce the false meritocracy, reduce the ability for bad actors to censor research, and more. The cost would be a larger number of publications, but perhaps this would help people realize that many of the publications coming out right now are of little value. Transparency is badly needed in many facets of academic research. My company made a site that helps bring transparency into literature review (sysrev.com).

sgillen 7y ago
I totally agree with the transparency thing. I would love to see something like open review (https://openreview.net/forum?id=ryQu7f-RZ https://openreview.net/forum?id=ryQu7f-RZ) become more common, and even have something like this replace the traditional peer review system. We would all have to think long and hard about how to align incentives to get the desired results.As a first pass I think hiring some amount of people to review every paper, and then allowing anyone who wants to come by and leave their comments as well might be a place to start. I think some sort of reputation system similar to stack overflow might be a motivator as well.

wtvanhest 7y ago

I'm going to propose something that is unconventional today, but I believe will be standard practice at somepoint. Overtime we have shown that many mistakes are made in labs and we know that many experiments are not replicable. I think its time that we ask researchers using public funds to film their entire experiment from beginning to end. Every pipette, every prep etc. Peer review should also have a video audit. Then a replication in another lab. The cost of ensuring results are what they say they are would be offset by less wasted research dollars on false rabbit holes.

sgillen 7y ago
I do actually agree that we should apply more funding to replicating results. I disagree that a video log is an efficient way to ensure replicatability. In many disciplines this is not useful at all (math, high energy physics). And in most others, like your biology example, I think it adds an undue burden without much gain.
fwip 7y ago
What evidence do you have that suggests that the cost of doing a video audit (an expert carefully watching hundreds of hours of experimentation) and a replication study for every submitted article would be cheaper than the occasional paper that leads to a second experiment that doesn't pan out?
Nemo_bis 7y ago
Are you trying to re-invent https://en.wikipedia.org/wiki/Reproducibility_Project https://en.wikipedia.org/wiki/Reproducibility_Project ?
- wtvanhest 7y ago
  No. Reproducibility requires double the investment and may also have the same or different mistakes. A video log would help researchers find errors and confirm results. Video is simply a better solution.

dan-robertson 7y ago

A curious thing that’s happened in some parts of mathematics is that formal peer review, needing coordination of journals, editors, and reviewers, can happen after many people have already read the paper (effectively informally peer reviewing it but with fewer comments), and even after people have cited the paper. This peer review then seems to not serve much purpose. An example is the recent first finite prime gap paper where the result was quickly strengthened several times by other authors long before the paper made it into any journal. Another example here: https://mobile.twitter.com/adrianprw/status/1156534906618597377 https://mobile.twitter.com/adrianprw/status/1156534906618597...

mathgenius 7y ago

> A 2015 study of 1,000 medical journal submissions found that of the papers that were eventually published, the 14 that became the most frequently cited were initially rejected. Groundbreaking studies by Sir Frank MacFarlane Burnet, Rosalind Yalow, Baruch Blumberg and others were rejected by peer reviewers, yet later led to Nobel Prizes. Yup, if you are actually doing good work it is likely to be so different to what is going on that people have a hard time evaluating it.

dmurray 7y ago
If you are actually doing ground-breaking work. There's plenty of scope for doing good work that is boring and mainstream and still high quality enough to be published.
- analog31 7y ago
  Indeed, my work was of the latter type. It was rejected by Physical Review Letters but accepted by another journal. One feature of peer review that's overlooked, is that there are different journals with different standards, and you can choose a journal that fits what you're trying to accomplish.
magicalhippo 7y ago
I can't read the article, but being rejected doesn't have to mean it's bad work. My mom has helped many PhD students take their degree over the years, and in the process helped with their papers. Several have been rejected, some multiple times. Frequently it was down to presentation issues (the student had not explained the certain things clearly enough or similar), or there was disagreement about how strong conclusions one could draw from the results. For example, recently a paper was rejected because the experimenters had forgotten or overlooked an important detail which meant they couldn't control for a certain variable. One of the reviewers picked this up, and rejected the paper because the findings could possibly be explained by this uncontrolled variable. So they had to resubmit with a weaker conclusion (this is when my mom got involved). In this case the review process was harsh but fair. The experimenters had goofed, and the reviewer caught it. Of course when doing ground breaking experiments, I guess the process might not be optimal. But the majority of scientists are not doing that.

scythe 7y ago

We don’t. Peer review is a very helpful component of a system that slowly filters and refines academic work. It’s like the first air intake stage of a jet turbine.

cassowary37 7y ago

For everyone complaining about peer review, another alternative to open review is paid reviewers who focus on a particular topic. For example, I'm an editor at a journal where statistical reviewers are paid and evaluated, while the remaining (content) reviewers are not. This ensures that we have a rigorous review of the statistical methods, at least. I do think payment increases the reviewers' responsibility and conscientiousness, and the overall quality of the subsequent articles. (nb This probably has been tested somewhere.) BUT - paid reviews are not feasible for many journals, especially open access/low-cost journals where margins may be thin. (Conversely, Elsevier should be able to pay their reviewers in gold bullion, rather than taking advantage of the scientific community's altruism, but that's another conversation).

jedberg 7y ago

Somewhat tangential, but are there any journals that will only accept a paper if someone unrelated to the author can replicate the study? Or a journal of only replicated studies?

Nemo_bis 7y ago
Sure. My favourite is IPOL Journal · Image Processing On Line https://www.ipol.im/ https://www.ipol.im/, which is doing it since 2011: «IPOL is a research journal of image processing and image analysis which emphasizes the role of mathematics as a source for algorithm design and the reproducibility of the research. Each article contains a text on an algorithm and its source code, with an online demonstration facility and an archive of experiments. Text and source code are peer-reviewed and the demonstration is controlled. IPOL is an Open Science and Reproducible Research journal». More recently, some megajournals have joined the trend and are trying to expand it, particularly eLife: https://elifesciences.org/labs/ad58f08d/introducing-elife-s-first-computationally-reproducible-article https://elifesciences.org/labs/ad58f08d/introducing-elife-s-... (also covered in https://www.nature.com/articles/d41586-019-00724-7 https://www.nature.com/articles/d41586-019-00724-7 ). For some context see: * https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis * https://www.frontiersin.org/articles/10.3389/fninf.2017.00069/full https://www.frontiersin.org/articles/10.3389/fninf.2017.0006...

neonate 7y ago

http://archive.is/UCSof http://archive.is/UCSof

typon 7y ago

I think the peer review system is ripe for, as SV people call it, "disruption". Even something like an app that you submit papers to, which then anonymizes them, standardizes formatting and presents them to randomly selected experts for review in a certain time period would go a long way compared to the current process. In my lab at a prestigious Canadian university, the papers' authors names were visible as well as the reviewers names because any back and forth post review would happen through email. I think journals and universities need to buy into a system that standardizes this process so at least these basic requirements of impartiality can be met.

tastroder 7y ago
While I'd apreciate a little fresh air in the process as well... > an app that you submit papers to, which then anonymizes them, standardizes formatting and presents them to randomly selected experts for review in a certain time period Except for that it's not an app (that would be kind of pointless, why would I want that on a phone instead of a webapp), that's pretty much the standard in my field. Anonymization is done through the authors (if the conference/journal wants it to be) and ensured by the conference submission system. Formatting is standardized anyway. The problem is the "randomly selected experts" part, for many (most?) subfields suitable reviewers are a scarce resource, to the point that in quite a few double-blind processes I've seen one or both parties could recognize each other's work or review. I've always found openreview [1] a nice way to at least make the process public, especially when anonymity is an illusion anyway. https://openreview.net/ https://openreview.net/
- typon 7y ago
  By app I meant software not a phone app. Secondly, I think the problem of having few experts in the field is a legitimate problem but I don't think the status quo is a good solution either. I think it erodes trust in the scientific community which is very hard to build back up.
  - tastroder 7y ago
    I think we need an overall shift in incentives in academia before that will change. Popular conferences see an all time high in submissions, though I do not have data on that I'm sure Journals face similar problems these days. I feel like that's not an aspect that will get better in the short term. On the app/phone app thing: Ah, fair enough, sorry about that.

chr1 7y ago

The whole scientific publishing field is still stuck in previous century. We desperately need a github for science, where any reader would be a reviwer, who would be able to rate the article and comment on it, and then journals would use these ratings combined with credibility rankings of reviewers to create indexes of interesting new article-repositories.

HNisCurated 7y ago
I have people citing research with useless AMA standard. If you want both parties to agree, they need to see the Data. Most studies become uninteresting after you read their terrible data.
- chr1 7y ago
  Absolutely agree, articles without data and without the sourcecode, are not science, and that's another reason why we need github for science.
Nemo_bis 7y ago
More on this idea: http://bjoern.brembs.net/2015/04/what-should-a-modern-scientific-infrastructure-look-like/ http://bjoern.brembs.net/2015/04/what-should-a-modern-scient... http://bjoern.brembs.net/2018/01/why-academic-journals-need-to-go/ http://bjoern.brembs.net/2018/01/why-academic-journals-need-...

p1necone 7y ago

If peer review in Academia is anything like code review in corporate software dev it's probably next to worthless.

jdnenej 7y ago
Peer review never seems to catch actual bugs but does give you helpful tips like "we already have a function that does this" or "the language has a feature for this"
- p1necone 7y ago
  Oh code review done properly is really great - I'm just jaded by repeatedly seeing code go up and getting that green tick with no comments and knowing that the reviewer barely even attempted to read it.

fastaguy88 7y ago

The idea that in scientific publication, more is better, makes little sense to me as a practicing scientist. The scientific literature is already filled with papers with conclusions that are not well supported by the data, and that is after 50-80% of submissions are rejected. To argue that publication should have fewer restrictions, one needs to show that the modest number of papers that are improperly rejected by every journal (not just the first submission) equals or outnumbers the overwhelming number of submissions that should not be published because they reflect a misunderstanding of their field or a misinterpretation of the data. For many (most) scientists, the problem is not that important results are unpublished, the problem is that it is almost impossible to keep up with the current gated literature, particularly when that literature is full of mistakes. While it is certainly true that some reviewers are biased and some ill-informed, most of the time papers are rejected because the authors did not communicate well (for papers that should be published) or because the authors did not understand that their data did not support their conclusion (papers that should be rejected). The scientific literature needs a better signal to noise ratio, not more noise.

Nemo_bis 7y ago
The rejection of a submission does cancel a work, it merely shifts elsewhere. The question is whether peer review gives the right incentives; many suspect that publication venues with an ostensibly stricter peer review actually encourage the production of more noise, not less. See https://www.frontiersin.org/articles/10.3389/fnhum.2013.00291/full https://www.frontiersin.org/articles/10.3389/fnhum.2013.0029...

alexmlamb 7y ago

"Studies have shown that journal editors prefer reviewers of the same gender, that women are underrepresented in the peer review process, and that reviewers tend to be influenced by demographic factors like the author’s gender or institutional affiliation. " Most venues are double blind, so reviewers wouldn't know about the latter things. Having fewer women in the reviewer pool is unfortunate, but it's hard to see how it would effect the mean review quality in a direct way assuming that men and women are equally good at reviewing (maybe in an indirect way it could harm reviews by increasing the load per-reviewer).

bmc7505 7y ago
Not sure if this is the case, but I'd like to play devils advocate. While most venues are double blind, it is still possible there is a subconscious gender bias. Many studies have shown that gender can be readily detected from anonymized authors based style alone. https://www.aclweb.org/anthology/W11-0310 https://www.aclweb.org/anthology/W11-0310 http://cs224d.stanford.edu/reports/BartleAric.pdf http://cs224d.stanford.edu/reports/BartleAric.pdf https://www.sciencedirect.com/science/article/pii/S1877050916326849 https://www.sciencedirect.com/science/article/pii/S187705091...
xioxox 7y ago
Whether venues are double blind is extremely field dependent. In my field no journals are to my knowledge. However, many people use an initial rather than a first name, so it's difficult to assign gender unless you already know the person.
abdullahkhalids 7y ago
Depends on the field. In my field, I know pretty much the twenty groups who have the expertise to review my work and every paper of mine I could identify with high certainity from the referee's comment which group they belonged to.

known 7y ago

Is https://en.wikipedia.org/wiki/Wisdom_of_the_crowd https://en.wikipedia.org/wiki/Wisdom_of_the_crowd better?