I have been reviewer in this field (ACL-related conferences) for a couple years now. Based on the results of the yearly surveys: One of the biggest challenges

I have been reviewer in this field (ACL-related conferences) for a couple years now. Based on the results of the yearly surveys: One of the biggest challenges was to decide what to do about ArXiv and pre-prints in general - most researchers keep track of new papers on their area, and therefore it is likely that they will be assigned as reviewers for a paper that they have already seen (and for which they already know who the authors are). This issue came up for a vote a couple years ago, and the chosen answer was to not disqualify papers for this outright but to kindly ask authors to refer from doing so. At the same time, "88% consider double-blind reviewing at ACL conferences to be important", and only 14% of reviewers would penalize a paper for not including a relevant pre-print. This seems to me like a fair compromise. There is also work in progress to either talk to ArXiv to allow for author masking, and/or to create a new pre-print server that allows it. There is a very detailed report on the results of the survey in [1]. A second issue is whether the reviews should be made public, because it could compromise the double-blind process. The general result of the survey [2] was against it, where "support for public review tended to be inversely correlated with reviewing experience (and) female respondents were less likely to support public review than male respondents". [1] https://www.aclweb.org/portal/sites/default/files/SurveyReport2017.pdf https://www.aclweb.org/portal/sites/default/files/SurveyRepo... [2] http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019/07/ReportACL2019ReviewingSurvey.pdf http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019...