4 ms·

Fascinating how we (society) have let KPIs get into research/education and healthcare.

oefrha 3y ago
Research inherently requires attention and attention is inherently scarce, so unfortunately people do and will compete for attention however they can, metrics or not.
JumpCrisscross 3y ago
> let KPIs get into research/education and healthcare Research, sure, that’s a creative profession. But education and healthcare have obviously measurable outcomes, some of which are predictably desired ex ante.
- jrimbault 3y ago
  The implicit here is "business/for profit" KPIs.
godelski 3y ago
This is why I've started to argue against "peer review." Because no one can decide a paper's correctness/legitimacy by sitting down and reading the work. The method just doesn't scale and certainly doesn't in settings that are zero sum like conferences (see the absolute shitshow that is ML peer review). It's become a entire waste of time and money. The alternative I suggest is just submitting works to Open Review so there can be open discussions on the works. It removes the hostility from the setting and self biases towards reviewers/commenters being parties that are experts in that sub-niche as they're going to tend to be the only ones reading those papers, and especially reading closely. Metrics are fucking hard and we don't improve systems by removing nuance from metrics. Yet as complexity has increased in our world, this has seemed to be the direction we've gone. It's bad for science and the current methods have far too many false negatives (improper rejects) and false positives (improper accepts) to validate it as a meaningful signal. We've tried so many ways to fix it while maintaining the institution that it's time to just try letting go of it. I honestly cannot find a great argument for keeping these journals around (yes, there are arguments, but not many ,,strong,, arguments, and specifically ones that lead to better science).

Research inherently requires attention and attention is inherently scarce, so unfortunately people do and will compete for attention however they can, metrics or not.

JumpCrisscross 3y ago

> let KPIs get into research/education and healthcare Research, sure, that’s a creative profession. But education and healthcare have obviously measurable outcomes, some of which are predictably desired ex ante.

jrimbault 3y ago
The implicit here is "business/for profit" KPIs.

godelski 3y ago

This is why I've started to argue against "peer review." Because no one can decide a paper's correctness/legitimacy by sitting down and reading the work. The method just doesn't scale and certainly doesn't in settings that are zero sum like conferences (see the absolute shitshow that is ML peer review). It's become a entire waste of time and money. The alternative I suggest is just submitting works to Open Review so there can be open discussions on the works. It removes the hostility from the setting and self biases towards reviewers/commenters being parties that are experts in that sub-niche as they're going to tend to be the only ones reading those papers, and especially reading closely. Metrics are fucking hard and we don't improve systems by removing nuance from metrics. Yet as complexity has increased in our world, this has seemed to be the direction we've gone. It's bad for science and the current methods have far too many false negatives (improper rejects) and false positives (improper accepts) to validate it as a meaningful signal. We've tried so many ways to fix it while maintaining the institution that it's time to just try letting go of it. I honestly cannot find a great argument for keeping these journals around (yes, there are arguments, but not many ,,strong,, arguments, and specifically ones that lead to better science).