3 ms·

> My field (#NLProc) is working to maintain fully anonymous review. In such conditions, it’s important to be careful when tweeting about work (your own or other

by twic 6y ago

> My field (#NLProc) is working to maintain fully anonymous review. In such conditions, it’s important to be careful when tweeting about work (your own or others) that is under review. Interesting! I wasn't aware that anyone was doing double-anonymous peer review. I would love to know how that is working out.

probably_wrong 6y ago
I have been reviewer in this field (ACL-related conferences) for a couple years now. Based on the results of the yearly surveys: One of the biggest challenges was to decide what to do about ArXiv and pre-prints in general - most researchers keep track of new papers on their area, and therefore it is likely that they will be assigned as reviewers for a paper that they have already seen (and for which they already know who the authors are). This issue came up for a vote a couple years ago, and the chosen answer was to not disqualify papers for this outright but to kindly ask authors to refer from doing so. At the same time, "88% consider double-blind reviewing at ACL conferences to be important", and only 14% of reviewers would penalize a paper for not including a relevant pre-print. This seems to me like a fair compromise. There is also work in progress to either talk to ArXiv to allow for author masking, and/or to create a new pre-print server that allows it. There is a very detailed report on the results of the survey in [1]. A second issue is whether the reviews should be made public, because it could compromise the double-blind process. The general result of the survey [2] was against it, where "support for public review tended to be inversely correlated with reviewing experience (and) female respondents were less likely to support public review than male respondents". [1] https://www.aclweb.org/portal/sites/default/files/SurveyReport2017.pdf https://www.aclweb.org/portal/sites/default/files/SurveyRepo... [2] http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019/07/ReportACL2019ReviewingSurvey.pdf http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019...
mkeeter 6y ago
SIGGRAPH switched from single-blind (reviewers know the authors but not vice versa) to double-blind, starting in the 2019 conference: https://blog.siggraph.org/2018/10/the-magic-of-technical-papers.html/ https://blog.siggraph.org/2018/10/the-magic-of-technical-pap...

by twic 6y ago

probably_wrong 6y ago

I have been reviewer in this field (ACL-related conferences) for a couple years now. Based on the results of the yearly surveys: One of the biggest challenges was to decide what to do about ArXiv and pre-prints in general - most researchers keep track of new papers on their area, and therefore it is likely that they will be assigned as reviewers for a paper that they have already seen (and for which they already know who the authors are). This issue came up for a vote a couple years ago, and the chosen answer was to not disqualify papers for this outright but to kindly ask authors to refer from doing so. At the same time, "88% consider double-blind reviewing at ACL conferences to be important", and only 14% of reviewers would penalize a paper for not including a relevant pre-print. This seems to me like a fair compromise. There is also work in progress to either talk to ArXiv to allow for author masking, and/or to create a new pre-print server that allows it. There is a very detailed report on the results of the survey in [1]. A second issue is whether the reviews should be made public, because it could compromise the double-blind process. The general result of the survey [2] was against it, where "support for public review tended to be inversely correlated with reviewing experience (and) female respondents were less likely to support public review than male respondents". [1] https://www.aclweb.org/portal/sites/default/files/SurveyReport2017.pdf https://www.aclweb.org/portal/sites/default/files/SurveyRepo... [2] http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019/07/ReportACL2019ReviewingSurvey.pdf http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019...

mkeeter 6y ago

SIGGRAPH switched from single-blind (reviewers know the authors but not vice versa) to double-blind, starting in the 2019 conference: https://blog.siggraph.org/2018/10/the-magic-of-technical-papers.html/ https://blog.siggraph.org/2018/10/the-magic-of-technical-pap...