3 ms·
This is very cool, but I'm having trouble following the method behind the results. Say concretely, the result that single-blind reviewers give higher reviews t
by bo1024 9y ago
This is very cool, but I'm having trouble following the method behind the results.
Say concretely, the result that single-blind reviewers give higher reviews to papers from prestigious sources. Can anyone help me understand? https://arxiv.org/pdf/1702.00502.pdf https://arxiv.org/pdf/1702.00502.pdf
They give each paper a "blind" score bqps from the blind reviewers only. Then they fit the vector Theta to the model in 5.1 and present Theta in Table 5. They also present a p-value for each entry of Theta. In particular, for com (top company), uni (top university), and fam (famous author) the p-values are below 0.05.
Bu I didn't see how these p-values are computed or what they mean, or what the entries of Theta are supposed to mean. What's the null hypothesis? If the single-blind authors and double-blind authors acted the same way, would we expect the entries of Theta to all be zero except for the coefficient of bqps and the constant? (If so, why?)
...
In other words, I don't understand how the exponential odds-ratio model at the beginning of 5.1 maps to the claim in the introduction "We find that the likelihood for single-blind reviewers to enter a positive review is significantly higher for papers" from say top companies.
(Edit:) One thing that would help me is if the authors fit the same model in the reverse direction, i.e. computed bqps from the single-blind reviews and fit a Theta' in predicting the double-blind reviews. I am tempted to say that all their result shows is that, conditioned on the double-blind reviews, famousness is useful in predicting single-blind reviews and the trend is positive; but this could conceivably be true in the reverse case as well.
- pacbard 9y agoIt looks like they fit a (multinomial?) logistic regression. Unfortunately, the interpretation of the model coefficients is difficult because each coefficient is a log likelihood - therefore on a weird metric. I would suggest to look at the odds ratio column for a coefficient that is more interpretable. An odds ratio of 1 represent equal odds of being accepted. A coefficient greater than one represents better odds of acceptance (e.g., a paper from a university is 1.5 times as likely as a paper not from a university to be accepted or, said in another way, a paper for a university has is 50% more likely than a paper not from a university to be accepted). Coefficients less than 1 have a similar but opposite interpretation. For example, women are 0.78 times as likely as men to be accepted (or 22% less likely). I wish they had reported marginal probabilities of acceptance because odds ratios can be misleading when marginal probabilities are small or large (e.g., if group 1 has a marginal probability of acceptance of 1% and group 2 has 3%, their odds ratio will be 3 but the marginal difference is basically negligible).
- bo1024 9y agoThanks for the help. The odds ratios are easier to interpret, but the big question I still have is how this shows a (statistically significant) difference between double-blind and single-blind reviews....