3 ms·

science output is not directly proportional to effort. Instead of ranking scientists purely on their publication records, credit should be given for all sorts o

science output is not directly proportional to effort. Instead of ranking scientists purely on their publication records, credit should be given for all sorts of activities that don’t necessarily come across in traditional metrics. While I agree there is more to science output than publications, candidates for post-doc/faculty positions shouldn't be judged on "effort", but on making real contributions that bring progress to their field of research. Judging that on anything else than quality publications becomes difficult, but I look forward to what people may do about it. So far I've heard mostly talk and little action.

_delirium 16y ago
When you quantify output, though, it tends to reward conservatism: work on smallish, high-probability incremental improvements, not big things, because big things have too high variance. It also rewards being good at overselling work, although that's a bigger problem in some areas than others. I think people actually are already pretty good at judging research quality on something like "effort", or more to the point: how good a scientist is so-and-so? But it requires familiarity with the person over some time, and tends not to scale well. If you ask a scientist about the quality of research of one of their colleagues they've been in the same department with for 10 years, they'll rarely refer to publications, and they often won't even refer to discoveries. Instead, it's inputs rather than outputs to the research process that they evaluate: how well is the person managing his/her lab, how insightful is he/she in picking problems and approaches, how good at mentoring students, how well does he/she really understand where this field is and where it's going / should be going, etc. There is correlation, of course, in that the good scientists most often also produce good results. But at least among the researchers I know, if you have to pick one as the "real" one, inputs rather than outputs are it. If you observe someone for a while, you can tell whether they're a good scientist or not, and you can also tell the places where it doesn't correlate with outputs--- someone's been doing everything exactly right on an important problem but just hasn't gotten a break (or maybe did get it, but it came a month after they were scooped by another group), or someone else is kind of mediocre but got lucky with a big result. Neither's the common case, but it's not that rare either; it's not like research is just a crank you turn and results come out. =] I guess the main issue is how to scale it. I think the above is pretty much how people already do make decisions when they're familiar with someone's work. If someone is up for tenure, my impression is that profs in that person's area will rarely even bother looking at publication lists when deciding whether to vote yes/no: after working with the person for 6 years they already have a good idea of whether the person is a good researcher they want to keep. It's mainly the people in other areas (in a large department), or in administrative roles, who seem to have no choice but to use the CV as the main basis of their tenure decision.

_delirium 16y ago

When you quantify output, though, it tends to reward conservatism: work on smallish, high-probability incremental improvements, not big things, because big things have too high variance. It also rewards being good at overselling work, although that's a bigger problem in some areas than others. I think people actually are already pretty good at judging research quality on something like "effort", or more to the point: how good a scientist is so-and-so? But it requires familiarity with the person over some time, and tends not to scale well. If you ask a scientist about the quality of research of one of their colleagues they've been in the same department with for 10 years, they'll rarely refer to publications, and they often won't even refer to discoveries. Instead, it's inputs rather than outputs to the research process that they evaluate: how well is the person managing his/her lab, how insightful is he/she in picking problems and approaches, how good at mentoring students, how well does he/she really understand where this field is and where it's going / should be going, etc. There is correlation, of course, in that the good scientists most often also produce good results. But at least among the researchers I know, if you have to pick one as the "real" one, inputs rather than outputs are it. If you observe someone for a while, you can tell whether they're a good scientist or not, and you can also tell the places where it doesn't correlate with outputs--- someone's been doing everything exactly right on an important problem but just hasn't gotten a break (or maybe did get it, but it came a month after they were scooped by another group), or someone else is kind of mediocre but got lucky with a big result. Neither's the common case, but it's not that rare either; it's not like research is just a crank you turn and results come out. =] I guess the main issue is how to scale it. I think the above is pretty much how people already do make decisions when they're familiar with someone's work. If someone is up for tenure, my impression is that profs in that person's area will rarely even bother looking at publication lists when deciding whether to vote yes/no: after working with the person for 6 years they already have a good idea of whether the person is a good researcher they want to keep. It's mainly the people in other areas (in a large department), or in administrative roles, who seem to have no choice but to use the CV as the main basis of their tenure decision.