4 ms·
As long as there is some quantitative criteria on which jobs and promotions depend, there will be people gaming the system. One solution is to couple this quan
by j7ake 5y ago
As long as there is some quantitative criteria on which jobs and promotions depend, there will be people gaming the system.
One solution is to couple this quantitative criteria with independent committees that assess people beyond the metrics, but that requires a lot of human effort and not scalable.
Assessing people in ways that don’t scale seem to be the way to avoid this gaming trap in academia.
- cycomanic 5y agoI'd argue that it's not just that the metrics don't scale but the problem is that we are trying to find quantitative metrics for something that can't be easily quantified. The worst outcome is not even the forgeries and fakes as in this article, but more that even the vast majority of ethical academics are being pushed into a direction that is detrimental to longterm scientific progress, in particular short term outcomes instead of longterm progress.
- lutorm 5y agoeven the vast majority of ethical academics are being pushed into a direction that is detrimental to longterm scientific progress, in particular short term outcomes instead of longterm progress I agree. The egregious fraud is just the high-sigma wing. It's a symptom, but the real problem is how it affects the majority.
- rossdavidh 5y agoInteresting point; it is much like the problems of trying to assess programmer productivity.
- _tom_ 5y agoI was thinking it's much like Google trying to deal with SEO. Most people optimize for high google ranking, not quality content. Google periodically changes the evaluation, in theory to reward good content and penalize bad, but people still try to game the system, rather than improving content.
- _tom_ 5y agoAnd non-quantitive evaluations are prone to favoritism and prejudice. AKA people gaming the system. I doubt there's an easy answer. Trying to better align the short term objectives with longer term ones could help, but that just makes it harder to game, doesn't eliminate it.
- j7ake 5y agoIt’s why one needs both. You need both undeniable productivity by quantitative metrics, as well as glowing reviews from independent panels that are not influenced by favoritism (almost like an audit).