3 ms·

Sounds like a movement toward crowdsourcing, long overdue for our increasingly broken scientific process. If the HN experiment has worked for so long, surely,

Sounds like a movement toward crowdsourcing, long overdue for our increasingly broken scientific process. If the HN experiment has worked for so long, surely, with proper controls on participation and interaction, there's no need to worry about any kind of "Wild Wild West," as the author put it. What I wonder is whether anonymity would be better or worse for some kind of semi-public peer review forum.

kickout 9y ago
Anonymity would be better in my opinion (to start off with). Way too much name recognition, and "good old boys club" dynamics happening in way too many areas of science i follow (genetics, agriculture, prediction)
- neuromantik8086 9y ago
  Pseudonymity might also be a good option (although it also has its pitfalls): https://twitter.com/jaipelai/status/900513036552859648 https://twitter.com/jaipelai/status/900513036552859648 https://news.ycombinator.com/item?id=15918851 https://news.ycombinator.com/item?id=15918851
mlspector 9y ago
I'm one of the main developers of OpenReview.net, and we're trying to do something similar to APPRAISE as described in the OP. Right now we mainly focus on the machine learning research community, and one conference in particular - ICLR - has been experimenting with varying degrees of anonymity in their peer review process. We've had some preliminary discussions about evaluating the impact of anonymity on peer review "quality", but so far the proper way to analyze the data isn't quite clear to us. We do have an open API that allows the general public to access the data, but I'm embarrassed to say that our documentation is very sparse right now. If anyone reading this is interested in exploring the data, shoot us an email at info@openreview.net and we can help you get started. https://openreview.net/about https://openreview.net/about
forapurpose 9y ago
> If the HN experiment has worked for so long ... HN is relatively good for an Internet forum, not for scientific literature. It's not even good professional IT work product - that's not an insult; it's not what the forum is intended to be. I'd be very concerned if the quality of scientific research and review dropped to this level.

kickout 9y ago

Anonymity would be better in my opinion (to start off with). Way too much name recognition, and "good old boys club" dynamics happening in way too many areas of science i follow (genetics, agriculture, prediction)

neuromantik8086 9y ago
Pseudonymity might also be a good option (although it also has its pitfalls): https://twitter.com/jaipelai/status/900513036552859648 https://twitter.com/jaipelai/status/900513036552859648 https://news.ycombinator.com/item?id=15918851 https://news.ycombinator.com/item?id=15918851

mlspector 9y ago

I'm one of the main developers of OpenReview.net, and we're trying to do something similar to APPRAISE as described in the OP. Right now we mainly focus on the machine learning research community, and one conference in particular - ICLR - has been experimenting with varying degrees of anonymity in their peer review process. We've had some preliminary discussions about evaluating the impact of anonymity on peer review "quality", but so far the proper way to analyze the data isn't quite clear to us. We do have an open API that allows the general public to access the data, but I'm embarrassed to say that our documentation is very sparse right now. If anyone reading this is interested in exploring the data, shoot us an email at info@openreview.net and we can help you get started. https://openreview.net/about https://openreview.net/about

forapurpose 9y ago

> If the HN experiment has worked for so long ... HN is relatively good for an Internet forum, not for scientific literature. It's not even good professional IT work product - that's not an insult; it's not what the forum is intended to be. I'd be very concerned if the quality of scientific research and review dropped to this level.