3 ms·

1) You must therefore pay the reviewers. Probably a good starting point. 2) It is in fact, hard to publish things already in top journals especially if you're

1) You must therefore pay the reviewers. Probably a good starting point. 2) It is in fact, hard to publish things already in top journals especially if you're not established. Given the fact you also want the pool of reviewers to shrink, you're also making difficulty of publishing works even harder. Yes, the issue is paper mills, got it, but you're making it even harder for sincere scientists who aren't part of a mill in doing this. 3) The flood of work that will fall to the smaller, more accountable group will require culling, which of course, the top journals will cull the horde based on the biases they already have: pedigree, which will further calcify the issues in science and research already, or even intensify it.

godelski 3y ago
1) I mostly agree. There are at least other rewards that can also be offered, such as conference discounts. There's also the token system (pay tokens to submit works, receive tokens for reviewing). But some incentive structure needs to happen, agreed. > especially if you're not established This is commonly stated but I don't think many internalize what it means. I don't think a meritocracy can exist -- this indicator supports that -- and shows that there's strong factors influencing acceptance beyond the quality of work. 2) I do think we need to be very careful about how we sell the prestige of publications/venues. As fields get more popular and our metrics rely more on them (publish or perish) then this not only encourages paper mills, but cheating in general. As two simple examples, look at how ML publishes works where papers use proprietary datasets/models (reveals authors' lab and thus violates ethics), or how it is status quo to tune hyperparameters on test data results (information leakage). You're are a huge disadvantage if you don't cheat. We have a similar problem in schools and this is why students cheat. If it is hard to catch and bad actors aren't punished (high risk because high false positive) then you actively reward cheaters. This needs to be a serious conversation and I don't think we are having this (this is several domains, even outside academia). 3) There's a coupling effect here though, that is pretty destructive and can have bigger social ramifications. As the noise in venue publication increases, the trust in the venue decreases. But I think it increases first, as people are metric chasing. But I suspect it'll be like a rubber band, and snap back hard. The larger ramification is social trust around science, where only the large venues matter and not the small unknown journals. We already have a growing distrust as the "just asking questions" anti-science strategy has been growing, and we need to be pretty careful and think beyond a local (spatially and temporally) window.

godelski 3y ago

1) I mostly agree. There are at least other rewards that can also be offered, such as conference discounts. There's also the token system (pay tokens to submit works, receive tokens for reviewing). But some incentive structure needs to happen, agreed. > especially if you're not established This is commonly stated but I don't think many internalize what it means. I don't think a meritocracy can exist -- this indicator supports that -- and shows that there's strong factors influencing acceptance beyond the quality of work. 2) I do think we need to be very careful about how we sell the prestige of publications/venues. As fields get more popular and our metrics rely more on them (publish or perish) then this not only encourages paper mills, but cheating in general. As two simple examples, look at how ML publishes works where papers use proprietary datasets/models (reveals authors' lab and thus violates ethics), or how it is status quo to tune hyperparameters on test data results (information leakage). You're are a huge disadvantage if you don't cheat. We have a similar problem in schools and this is why students cheat. If it is hard to catch and bad actors aren't punished (high risk because high false positive) then you actively reward cheaters. This needs to be a serious conversation and I don't think we are having this (this is several domains, even outside academia). 3) There's a coupling effect here though, that is pretty destructive and can have bigger social ramifications. As the noise in venue publication increases, the trust in the venue decreases. But I think it increases first, as people are metric chasing. But I suspect it'll be like a rubber band, and snap back hard. The larger ramification is social trust around science, where only the large venues matter and not the small unknown journals. We already have a growing distrust as the "just asking questions" anti-science strategy has been growing, and we need to be pretty careful and think beyond a local (spatially and temporally) window.