6 ms·

The only alternatives I see: 1. National research funding agencies and universities need to stop doing their assessments based on publication counting. Of cour

by 13415 6y ago

The only alternatives I see: 1. National research funding agencies and universities need to stop doing their assessments based on publication counting. Of course, publications need to be taken into account somehow, but at the very least these indicators should be weighed by curated rankings of journals. 2. Existing publications can be used as an indicator for how much output is to be expected in the future, but the goals for research output should be kept reasonably low. You cannot control the output of researchers, you can only control which ones you hire and (to some extent) the type of research they conduct. 3. Hiring in academia needs to be based more often on screening candidates by committees of outside experts in the field instead of local staff and people who don't know the area well. Projects also always need to be evaluated by experts on project's topic. (As strange as this may sound, this is often not the case!) It might even help to prune away irrelevant indicators, e.g. ask candidates to submit only 3-5 of their best publications and completely ignore the rest. And by "completely" I really mean completely. The reason is that if you have two researchers and one of them has 20 publications with five really good ones, and the other has 40 mediocre publications and not a single good one, then it is very hard for a normal assessment committee to justify taking the first one, but it's always easy to converge on the second one, even if that is exactly the wrong choice. 4. Assessment committees need to be told to evaluate the quality and originality of the research only. If you really want or need a certain output quantity, then make it part of the formal hiring criteria, not part of the scientific evaluation. 5. Increase funding for risky projects and risky individual grants, elimination of criteria that exclude unusual CVs (e.g. allow long time after PhD, people from other areas, people with time spent in business, unusual research suggestions). Originality should be one of the highest ranking criteria. There should also be a focus on versatility and hard skills. Even in the humanities, never hire anyone who says anything disparaging about mathematical methods and statistics, and never leverage people who don't know the tools of their trade into positions of power. Basically, you cannot push scientists to anything. Once someone is hired at the postdoc level, you cannot steer much, micromanagement and constant evaluation are highly counter-productive. You need to hire scientists that do interesting research. Treat them like an investment in a startup: Most of them will fail but some of them will succeed. Give them a second chance, maybe even a third one, but not indefinitely many. The hiring policies and processes at universities are often bad. Candidates are not evaluated by experts, there is plenty of favoritism, boring high producers are favored over interesting researchers who want to built something up, because the local staff doesn't like scientists who "shake things up", and so on. There is a lot of inertia to overcome in Academia, being a good scientist can sometimes even be harmful to your career. Funding agencies need to steer against this. Edit: Much of what I mention can also be achieved by getting an absolute top researcher in an area, give him or her an institute or research unit and ton of money, and let them do their thing. They know what to do and whom to hire. But it's expensive.

asciident 6y ago
From my experience, most of your ideas have been implemented in different ways in the academic procedures as best as they can. 1. Funding agencies and universities don't make decisions. Only individuals in those institutions do. And I don't know any individuals who admit to doing assessments based on publication counts. They'd be openly mocked for such an archaic way of thinking. The only people who seem to openly obsess over publication counts are Ph.D. students. 2. Agreed, but output can be measured in different ways besides traditional academic publications. 3. It seems like this is already done, through recommendation letters when people apply, and in external letters during the tenure process. I see the opposite outcomes, whenever I've seen someone with fewer but better publications compared against someone with more but weaker publications, the first is always preferred. Because it's easy for someone to make the case "Person 1 has fewer paper but they're more impactful and they choose riskier problems, an excellent trait" versus someone to make the case for "Person 2 has more papers in quantity so are more productive." Unless of course it's not clear that person 1's papers are actually higher quality, in which case the original premise doesn't hold. 4. This is already the primary factor from what I've seen. That's the whole point of job talks, publications listed on CVs, etc. I've never seen a tenure-track hire without people reading some of the papers listed on the CV, and discussing the presented work in the job talk. 5. Half of the funding agencies now say they're looking for risky ideas, the word "transformative" has been a common part of NSF vocabulary for decades. Originality is one of the NIH core criteria factors, as is criteria about the Investigator themselves. How panelists interpret the criteria is more social and political than a technical thing. Anyways, maybe we probably just live in different academic worlds, but I don't see as much blatant expression of the biases you are describing, and feel like the major goals of your ideas have already been implemented which has led us to the current situation. Maybe not everyone buys into them, but as you say there isn't a way to get a bunch of individuals to follow orders, so the only change that's possible is changes to process which has already happened. I don't agree that we need to add additional bureaucracy like rating applicants on "did they disparage mathematics", which I find extremely problematic. Even the last idea in your edit is already done in every way possible. The MacArthur award, Turing Award, Nobel Prize, various prizes, etc. all give an absolutely top researcher money and prestige, so they can do their thing. I think the best way forward are social solutions, rather than procedural. Convince minds, rather than add bureaucracy.

by 13415 6y ago

asciident 6y ago

From my experience, most of your ideas have been implemented in different ways in the academic procedures as best as they can. 1. Funding agencies and universities don't make decisions. Only individuals in those institutions do. And I don't know any individuals who admit to doing assessments based on publication counts. They'd be openly mocked for such an archaic way of thinking. The only people who seem to openly obsess over publication counts are Ph.D. students. 2. Agreed, but output can be measured in different ways besides traditional academic publications. 3. It seems like this is already done, through recommendation letters when people apply, and in external letters during the tenure process. I see the opposite outcomes, whenever I've seen someone with fewer but better publications compared against someone with more but weaker publications, the first is always preferred. Because it's easy for someone to make the case "Person 1 has fewer paper but they're more impactful and they choose riskier problems, an excellent trait" versus someone to make the case for "Person 2 has more papers in quantity so are more productive." Unless of course it's not clear that person 1's papers are actually higher quality, in which case the original premise doesn't hold. 4. This is already the primary factor from what I've seen. That's the whole point of job talks, publications listed on CVs, etc. I've never seen a tenure-track hire without people reading some of the papers listed on the CV, and discussing the presented work in the job talk. 5. Half of the funding agencies now say they're looking for risky ideas, the word "transformative" has been a common part of NSF vocabulary for decades. Originality is one of the NIH core criteria factors, as is criteria about the Investigator themselves. How panelists interpret the criteria is more social and political than a technical thing. Anyways, maybe we probably just live in different academic worlds, but I don't see as much blatant expression of the biases you are describing, and feel like the major goals of your ideas have already been implemented which has led us to the current situation. Maybe not everyone buys into them, but as you say there isn't a way to get a bunch of individuals to follow orders, so the only change that's possible is changes to process which has already happened. I don't agree that we need to add additional bureaucracy like rating applicants on "did they disparage mathematics", which I find extremely problematic. Even the last idea in your edit is already done in every way possible. The MacArthur award, Turing Award, Nobel Prize, various prizes, etc. all give an absolutely top researcher money and prestige, so they can do their thing. I think the best way forward are social solutions, rather than procedural. Convince minds, rather than add bureaucracy.