6 ms·

This is a bit surprising, but hardly the biggest problem. I’ve posted before about it, but academic has to get away from the infantilism of using KPIs to judge

This is a bit surprising, but hardly the biggest problem. I’ve posted before about it, but academic has to get away from the infantilism of using KPIs to judge creative intellectual work. Part of the problem is that people seem to want an objective evaluation of a piece of research, and measure the value for taxpayer money. Well, you can’t get an objective evaluation. It’s all subjective. And “impact” as a quantifiable entity in general is nonsense, for one thing the timescales prohibit its measurement at the point where it may be useful. The solution is to use management. Lots of people object here and say “but nepotism, favouritism” and yep that’s a problem, but it is less of a problem that the decline of western universities. You can circumvent it somewhat by rotation, by involving external figures, by a hierarchy that ensures people are not colluding, but ultimately you just have to trust people and accept some wastage. People aren’t in academia for the money. It’s a vocation. You’re not going to have many people milking the system. Things went pretty well before the metric culture invaded the academy. They can go well again.

pfortuny 5y ago
This is a laudable opinion BUT. (Speaking about Spain). The KPIs are there because OFFICIALLY (and this is strictly so) you are not even allowed to get a tenured position without an absurd number of (in Maths) JCR papers IN THE FIRST QUARTILE. This is so stupid it is not even funny but how can you fight that when your PhD students depend on those metrics?
- dheera 5y ago
  Personally I think we need to do away with tenure. Academic institutions should just hire good researchers as employees and do good research. There was a time in history when tenure made sense, but today the tenure track process forces a lot of people to go after low-hanging fruit that has a high probability of being accepted for publication, instead of trying things that are meaningful to try but may fail.
  - pfortuny 5y ago
    That is precisely why tenure is important: you can think without time pressures.
    - j7ake 5y ago
      Having tenure does not mean you do not have time pressure. You still have students and postdocs under you and you still need to win grants to do research and pay salaries. You are constantly under evaluation whether you have tenure or not.
      - burntoutfire 5y ago
        > You are constantly under evaluation whether you have tenure or not. And what happens if you're poorly evaluated? They can't fire you anyway. Will they move you to a boiler room as a punishment?
        austinjp 5y ago
        Presumably this depends on the country and the institution, but you certainly could be "overlooked" and end up with work that is tedious and unlikely to garner plaudits. I'm sure that after a while this could lead to something approximating firing, being "asked to move on" or similar.
        deleted 5y ago
        [deleted]
    - jcelerier 5y ago
      Ahah what. All my tenured professors had constant time pressures.
    - inglor_cz 5y ago
      Or without political pressures. Provided that you get tenure in the first place.
    - austinjp 5y ago
- einpoklum 5y ago
  With respect, you're saying metrics are used because they are used. Officially or semi-officially, doesn't matter. There needs to be a collective and individual process of disowning the metrics.
YetAnotherNick 5y ago
The issue is everyone's subjective measure varies by a lot. The NIPS experiment showed that the papers selected for publication by peer review is more on the random side. Now assume you got this much randomness but on your career progression. And it would be even worse as in the peer review system works on more niche topics then university panel judging the work could be so their review will be even more subjective and changing with time. In short, I think it is definitely clear neither the citation nor the journal brand is the best proxy for the worthiness but the system you are proposing is worse while still reliant on subjective judgement.
burntoutfire 5y ago
> People aren’t in academia for the money. It’s a vocation. You’re not going to have many people milking the system. Before introducing the KPIs, a majority of polish science was basically people milking the system and doing barely any (valueable) research. It was seen as an easy, safe and ok paying job where the only major hassle is having to teach the students. You often needed connections to get in. It was partially like that because of the communist legacy, where playing ball with the communist party was the most important merit for promotion, which, over the course of 45 years (the span of communism in Poland), filled the academia management ranks with conformist mediocrities. Now, after a series of major reforms, there's a ton of KPIs, and people are now doing plenty of makework research to collect the required points, but still little valueable work gets done. Also, people interested in doing genuine science who would be doing it under the old system are now discouraged from joining academia, because in the system they're expected to game the points system and not to do real work. What is the lesson from this is? Creating institutionalized science is hard? It requires a long tradition and scientific cultural standards and can't be just wished into place by bureaucrats? Also, perhaps it's good to be doing the science for some purpose, which in the US case are often DoD grants, where the military expects some practical application. This application may be extremely distant, vague and uncertain (they fund pure math research!), but still, they're the client and they expect results. Whereas the (unstated) goal of science in Poland seems to be just to increase the prestige of Polish science and its Universities by getting papers into prestigious journals, whereas the actual science being done doesn't matter at all - basically state-level navel gazing.
- lkozma 5y ago
  I recognize the symptoms you describe, but I find the picture you painted a bit too pessimistic, given that in the field that I'm familiar with (theoretical CS), groups in Poland have been among the strongest and most visible in Europe in the past 10 years.
  - ed_balls 5y ago
    because it's fresh blood and there is a skill set filter which reduces nepotism and other forms of corruption. Looking at other departments, say, sociology* it's a dumpster. Full of cronies, retired politicians and relatives. *Just an example which doesn't require "hard" skills like math.
kungito 5y ago
I'd say the problem is there is simply too many people wanting to get a position compared to how many open positions there are. In my college PhD's are practically handed out in return for working on some projects which generate revenue for the Uni + doing the teaching assistant work. After that everyone goes back to the industry because it's practically impossible to get a professor title. And no one ever loses their job so there are more professors barely doing anything than those investing their life into it. They should make it so that you can only be a professor for max 15 years then you have to go. Use that expertise in the industry and let someone else try being a professor.
prof-dr-ir 5y ago
> The solution is to use management. You comment does not seem to contain any explanation as to why 'using management' would solve the problem you allude to. Can you elaborate?
jonathanstrange 5y ago
Wow, I couldn't disagree more. I can assure you that if you mean by "management" subjective evaluation by local administrative staff or "higher ups" like full professors, then it's the worst system you could possibly have. It invariably leads to corruption, favouritism, and brown nosing. Funding authorities have been fighting these rotten structures for the past 20 years where I work (Southern Europe), but they cannot do too much because the university system is not under their control, only the funding for national research projects. There is only one reasonable way to evaluate research and researchers, that's to evaluate the content of their work and publications by external evaluation panels and tell these panels explicitly that they should not base their assessments solely on indicator counting, but on the overall merits, originality, and prospects of the research according to their subjective opinion. Metrics shouldn't even be used as a tie-breaker, they should only ever be used as weak indicators, and this must be explicitly mentioned in the guidelines. In addition, you need a few other guidelines and laws. For example, it must be prohibited that someone becomes a postdoc at the same place where they obtained their Ph.D. We have people who study at university X, do their exams at university X, do their Ph.D. at university X under the same professors they always knew, then become postdocs working for their former supervisors and being exploited by them (teaching the same bad seminar their former supervisor teaches the past 20 years), and then get tenure in a rigged call. And the worst thing about it is that they feel entitled to all of this. You've got to break this vicious cycle, but with your suggestion of using a methodology that worked in 1950s (with an order of magnitude less candidates) this could never be achieved.
- cinntaile 5y ago
  But optimizing for originality is one of the reasons of the current mess so why is this an appropriate metric? Reproducing research is very valuable and a way to combat fake results.
- kungito 5y ago
  But many less well off EU countries have only 1 good Uni and many people don't like moving away. That would mean forcing people to move out of their homeland just for tenure. I'd rather we stop having lifetime professors and limit the tenure to 10-15 years
mschuster91 5y ago
> Things went pretty well before the metric culture invaded the academy. They can go well again. That won't happen, mostly because employers have outsourced education and vetting (in the form of requiring bachelor/master degrees) to universities (and the associated costs to governments and/or students who pay tuitions) instead of the old style vocational training/apprenticeship system where the employers had to pay. Want to restore academia to only those actually interested in science? Make employers pay a substantial tax on jobs requiring academic degrees.
miltondts 5y ago
"The solution is to use management." Not sure what you mean by that, but KPIs are generally put in place by high-level management. Or do you want more micromanagement? Either way, I think the solution is not more control, quite the opposite. I think the solution is just to remove the extrinsic incentives. Some people say UBI will cause people to do nothing and that is probably true, but the flip side is that the output of the remaining people will likely be many times higher both in volume and quality (with the total volume much lower but higher quality). Not having their energy completely destroyed by all the busy work necessary to show they are working.
derbOac 5y ago
> People aren't in academia for the money. I agree overall with what you wrote, but have to comment here, because I think this is already not the case in many settings. I can only speak for the US, but in my experience with some other places overseas similar issues are developing. There are many legitimate hypotheses for why this is the case, but in general at many universities, as far as climbing the academic ladder is concerned, publication metrics are no longer relevant. That is, some baseline is required, but beyond that, most of the focus is on money and grant sizes. I've been in promotion meetings discussing junior faculty that are not publishing and this is brushed aside because they have large grants. I've also repeatedly heard sentiments to the effect of "papers are a dime a dozen, grant dollars distinguish good from bad research." Again, there's lots of reasonable opinions about this, but I've come to a place where I've decided this is incentivizing corruption. Good research is only weakly correlated with its grant reimbursement, and regardless, it's lead to a focus on something only weakly associated with research quality. Discussions with university staff where you're openly pressured to artificially inflate costs to bring in more indirect funds should raise questions. Just as it's apparent that incentivizing (relatively) superficial bibliometric indices like publication count or h factors leads to superficial science, incentivizing research through grant money has the same effect, but differently. So yes, going into academics is not the way to make money if that's what you want. However, I think nowadays in the US, it's very much all about the money for large segments, who are milking the system right now at this moment. Also, in theory, yes, management is the solution, but really management is how we've gotten into this mess. Good management, yes, bad management no. But how do you insure the former? Fixing this mess academics has slid down (in my perception, maybe everything really is fine) will require a lot of changes that will be controversial and painful to many, and I don't think there's a single magic bullet cure. Eliminating indirect funds is probably one thing, funding research through different mechanisms is another, maybe lotteries, probably opening up grant review processes to the general public. Maybe dissociating scientific research from the university system even more so than has been the case is also necessary. Maybe incentivizing a change in university administration structures. Probably all of the above, plus a lot else. How to get things to go well again is achievable in theory but how to get there is less clear given the amount of change involved.
Beldin 5y ago
> academic has to get away from the infantilism of using KPIs Well, science doesn't care that much for KPIs, per se. It's more that the managers want numbers to steer by. In academia, getting promoted means more management tasks. So higher up academics have been indoctrinated to want numbers. Is the scourge of management. As such: not a big fan of your solution.
jltsiren 5y ago
The problem is not about judging creative intellectual work. We already know how to do that, and the academia usually manages to do it just fine. It's not really about judging people either. When it comes to choosing which people to reward with jobs, promotions, grants, and prizes, we already know the solution: expert panels that spend nontrivial time with each application. Sometimes there are political or administrative reasons that override academic excellence, but in general the academia has figured out how to evaluate shortlisted people. The real problem is shortlisting people. For example, when a reputable university has an open faculty position, it typically gets tens of great applications. Because the people evaluating the applications are busy professors, they need a way of throwing away most of the applications with minimal effort. From the applicant's perspective, this means you only get a chance if the first impression makes you stand out among other great applicants. And that's why it matters that you publish in prestigious conferences/journals, come from a prestigious university, and have prestigious PhD/postdoc supervisors.

pfortuny 5y ago

This is a laudable opinion BUT. (Speaking about Spain). The KPIs are there because OFFICIALLY (and this is strictly so) you are not even allowed to get a tenured position without an absurd number of (in Maths) JCR papers IN THE FIRST QUARTILE. This is so stupid it is not even funny but how can you fight that when your PhD students depend on those metrics?

dheera 5y ago
Personally I think we need to do away with tenure. Academic institutions should just hire good researchers as employees and do good research. There was a time in history when tenure made sense, but today the tenure track process forces a lot of people to go after low-hanging fruit that has a high probability of being accepted for publication, instead of trying things that are meaningful to try but may fail.
- pfortuny 5y ago
  That is precisely why tenure is important: you can think without time pressures.
  - j7ake 5y ago
    Having tenure does not mean you do not have time pressure. You still have students and postdocs under you and you still need to win grants to do research and pay salaries. You are constantly under evaluation whether you have tenure or not.
    - burntoutfire 5y ago
      > You are constantly under evaluation whether you have tenure or not. And what happens if you're poorly evaluated? They can't fire you anyway. Will they move you to a boiler room as a punishment?
      - austinjp 5y ago
        Presumably this depends on the country and the institution, but you certainly could be "overlooked" and end up with work that is tedious and unlikely to garner plaudits. I'm sure that after a while this could lead to something approximating firing, being "asked to move on" or similar.
      - deleted 5y ago
        [deleted]
  - jcelerier 5y ago
    Ahah what. All my tenured professors had constant time pressures.
  - inglor_cz 5y ago
    Or without political pressures. Provided that you get tenure in the first place.
  - austinjp 5y ago
einpoklum 5y ago
With respect, you're saying metrics are used because they are used. Officially or semi-officially, doesn't matter. There needs to be a collective and individual process of disowning the metrics.

YetAnotherNick 5y ago

The issue is everyone's subjective measure varies by a lot. The NIPS experiment showed that the papers selected for publication by peer review is more on the random side. Now assume you got this much randomness but on your career progression. And it would be even worse as in the peer review system works on more niche topics then university panel judging the work could be so their review will be even more subjective and changing with time. In short, I think it is definitely clear neither the citation nor the journal brand is the best proxy for the worthiness but the system you are proposing is worse while still reliant on subjective judgement.

burntoutfire 5y ago

> People aren’t in academia for the money. It’s a vocation. You’re not going to have many people milking the system. Before introducing the KPIs, a majority of polish science was basically people milking the system and doing barely any (valueable) research. It was seen as an easy, safe and ok paying job where the only major hassle is having to teach the students. You often needed connections to get in. It was partially like that because of the communist legacy, where playing ball with the communist party was the most important merit for promotion, which, over the course of 45 years (the span of communism in Poland), filled the academia management ranks with conformist mediocrities. Now, after a series of major reforms, there's a ton of KPIs, and people are now doing plenty of makework research to collect the required points, but still little valueable work gets done. Also, people interested in doing genuine science who would be doing it under the old system are now discouraged from joining academia, because in the system they're expected to game the points system and not to do real work. What is the lesson from this is? Creating institutionalized science is hard? It requires a long tradition and scientific cultural standards and can't be just wished into place by bureaucrats? Also, perhaps it's good to be doing the science for some purpose, which in the US case are often DoD grants, where the military expects some practical application. This application may be extremely distant, vague and uncertain (they fund pure math research!), but still, they're the client and they expect results. Whereas the (unstated) goal of science in Poland seems to be just to increase the prestige of Polish science and its Universities by getting papers into prestigious journals, whereas the actual science being done doesn't matter at all - basically state-level navel gazing.

lkozma 5y ago
I recognize the symptoms you describe, but I find the picture you painted a bit too pessimistic, given that in the field that I'm familiar with (theoretical CS), groups in Poland have been among the strongest and most visible in Europe in the past 10 years.
- ed_balls 5y ago
  because it's fresh blood and there is a skill set filter which reduces nepotism and other forms of corruption. Looking at other departments, say, sociology* it's a dumpster. Full of cronies, retired politicians and relatives. *Just an example which doesn't require "hard" skills like math.

kungito 5y ago

I'd say the problem is there is simply too many people wanting to get a position compared to how many open positions there are. In my college PhD's are practically handed out in return for working on some projects which generate revenue for the Uni + doing the teaching assistant work. After that everyone goes back to the industry because it's practically impossible to get a professor title. And no one ever loses their job so there are more professors barely doing anything than those investing their life into it. They should make it so that you can only be a professor for max 15 years then you have to go. Use that expertise in the industry and let someone else try being a professor.

prof-dr-ir 5y ago

> The solution is to use management. You comment does not seem to contain any explanation as to why 'using management' would solve the problem you allude to. Can you elaborate?

jonathanstrange 5y ago

Wow, I couldn't disagree more. I can assure you that if you mean by "management" subjective evaluation by local administrative staff or "higher ups" like full professors, then it's the worst system you could possibly have. It invariably leads to corruption, favouritism, and brown nosing. Funding authorities have been fighting these rotten structures for the past 20 years where I work (Southern Europe), but they cannot do too much because the university system is not under their control, only the funding for national research projects. There is only one reasonable way to evaluate research and researchers, that's to evaluate the content of their work and publications by external evaluation panels and tell these panels explicitly that they should not base their assessments solely on indicator counting, but on the overall merits, originality, and prospects of the research according to their subjective opinion. Metrics shouldn't even be used as a tie-breaker, they should only ever be used as weak indicators, and this must be explicitly mentioned in the guidelines. In addition, you need a few other guidelines and laws. For example, it must be prohibited that someone becomes a postdoc at the same place where they obtained their Ph.D. We have people who study at university X, do their exams at university X, do their Ph.D. at university X under the same professors they always knew, then become postdocs working for their former supervisors and being exploited by them (teaching the same bad seminar their former supervisor teaches the past 20 years), and then get tenure in a rigged call. And the worst thing about it is that they feel entitled to all of this. You've got to break this vicious cycle, but with your suggestion of using a methodology that worked in 1950s (with an order of magnitude less candidates) this could never be achieved.

cinntaile 5y ago
But optimizing for originality is one of the reasons of the current mess so why is this an appropriate metric? Reproducing research is very valuable and a way to combat fake results.
kungito 5y ago
But many less well off EU countries have only 1 good Uni and many people don't like moving away. That would mean forcing people to move out of their homeland just for tenure. I'd rather we stop having lifetime professors and limit the tenure to 10-15 years

mschuster91 5y ago

> Things went pretty well before the metric culture invaded the academy. They can go well again. That won't happen, mostly because employers have outsourced education and vetting (in the form of requiring bachelor/master degrees) to universities (and the associated costs to governments and/or students who pay tuitions) instead of the old style vocational training/apprenticeship system where the employers had to pay. Want to restore academia to only those actually interested in science? Make employers pay a substantial tax on jobs requiring academic degrees.

miltondts 5y ago

"The solution is to use management." Not sure what you mean by that, but KPIs are generally put in place by high-level management. Or do you want more micromanagement? Either way, I think the solution is not more control, quite the opposite. I think the solution is just to remove the extrinsic incentives. Some people say UBI will cause people to do nothing and that is probably true, but the flip side is that the output of the remaining people will likely be many times higher both in volume and quality (with the total volume much lower but higher quality). Not having their energy completely destroyed by all the busy work necessary to show they are working.

derbOac 5y ago

> People aren't in academia for the money. I agree overall with what you wrote, but have to comment here, because I think this is already not the case in many settings. I can only speak for the US, but in my experience with some other places overseas similar issues are developing. There are many legitimate hypotheses for why this is the case, but in general at many universities, as far as climbing the academic ladder is concerned, publication metrics are no longer relevant. That is, some baseline is required, but beyond that, most of the focus is on money and grant sizes. I've been in promotion meetings discussing junior faculty that are not publishing and this is brushed aside because they have large grants. I've also repeatedly heard sentiments to the effect of "papers are a dime a dozen, grant dollars distinguish good from bad research." Again, there's lots of reasonable opinions about this, but I've come to a place where I've decided this is incentivizing corruption. Good research is only weakly correlated with its grant reimbursement, and regardless, it's lead to a focus on something only weakly associated with research quality. Discussions with university staff where you're openly pressured to artificially inflate costs to bring in more indirect funds should raise questions. Just as it's apparent that incentivizing (relatively) superficial bibliometric indices like publication count or h factors leads to superficial science, incentivizing research through grant money has the same effect, but differently. So yes, going into academics is not the way to make money if that's what you want. However, I think nowadays in the US, it's very much all about the money for large segments, who are milking the system right now at this moment. Also, in theory, yes, management is the solution, but really management is how we've gotten into this mess. Good management, yes, bad management no. But how do you insure the former? Fixing this mess academics has slid down (in my perception, maybe everything really is fine) will require a lot of changes that will be controversial and painful to many, and I don't think there's a single magic bullet cure. Eliminating indirect funds is probably one thing, funding research through different mechanisms is another, maybe lotteries, probably opening up grant review processes to the general public. Maybe dissociating scientific research from the university system even more so than has been the case is also necessary. Maybe incentivizing a change in university administration structures. Probably all of the above, plus a lot else. How to get things to go well again is achievable in theory but how to get there is less clear given the amount of change involved.

Beldin 5y ago

> academic has to get away from the infantilism of using KPIs Well, science doesn't care that much for KPIs, per se. It's more that the managers want numbers to steer by. In academia, getting promoted means more management tasks. So higher up academics have been indoctrinated to want numbers. Is the scourge of management. As such: not a big fan of your solution.

jltsiren 5y ago

The problem is not about judging creative intellectual work. We already know how to do that, and the academia usually manages to do it just fine. It's not really about judging people either. When it comes to choosing which people to reward with jobs, promotions, grants, and prizes, we already know the solution: expert panels that spend nontrivial time with each application. Sometimes there are political or administrative reasons that override academic excellence, but in general the academia has figured out how to evaluate shortlisted people. The real problem is shortlisting people. For example, when a reputable university has an open faculty position, it typically gets tens of great applications. Because the people evaluating the applications are busy professors, they need a way of throwing away most of the applications with minimal effort. From the applicant's perspective, this means you only get a chance if the first impression makes you stand out among other great applicants. And that's why it matters that you publish in prestigious conferences/journals, come from a prestigious university, and have prestigious PhD/postdoc supervisors.