4 ms·

The answer is simple, but it's also one that's completely useless for anyone reading this post. What needs to happen is a gradual realignment in both the fundin

The answer is simple, but it's also one that's completely useless for anyone reading this post. What needs to happen is a gradual realignment in both the funding and hiring criteria across all of academia. There needs to be less emphasis on quantitative metrics like citation counts, h-indices and other trivially gameable nonsense, and more emphasis on human judgment along with a set of basic criteria to provide a floor to research quality across the board (things that every good scientist should do, as opposed to things that you should relentlessly maximize). This would require the non-scientific managerial class to abrogate some of the power they now wield in the academia and relevant funding institutions, therefore is very unlikely to happen. And - probably, most annoyingly to the HN crowd - there is nothing meaningful that technological or business disruption can accomplish here. These are systemic social problems that have been continuously and systemically building up since (approximately) WW2. At this point the only entities large enough to make a difference are probably the NIH and NSF in the U.S. and the relevant EU funding agencies in Europe. In my judgment, these are precisely the agencies least likely to institute such change, so here we are.

nradov 7y ago
If we put more weight on human judgment then we'll get more favoritism, nepotism, and focus on politically popular topics. So we might end up just exchanging one set of problems for another.
- inimino 7y ago
  At least favoritism has some variability in it. The problem with metrics, besides being gamed, is that they are so uniform.
- keldaris 7y ago
  You won't get much more focus on politically (in the context of the academia) popular topics, the existing system already imposes that particular bias exceedingly well (how do you think you get citations?). But yes, you'll get more everyday human issues along with the judgment. In my view, that's far preferable to the existing situation, if only for the simple reason that the human problems will vary from place to place, whereas the existing system distorts virtually the entirety of academic research in roughly the same ways everywhere, to the uniform detriment of all.
- TeMPOraL 7y ago
  I used to be more worried about favoritism, nepotism et al. but these days, I'm more worried about quantifiable metrics. The former are situational, and there's only so much nepotism you can engage with before your peers in other places start considering it to be in bad taste. The latter, however, can be ruthlessly and efficiently optimized, to the full exclusion of any value that's not captured in the metrics.
cgiles 7y ago
At my institute, there is only one metric that matters: dollars you bring in through grants. There are a variety of ways to get to the cash: you can do it with a small number of high-impact papers, a larger number of low-impact papers, by finding a valuable niche, or by chasing fads. So I would say that fundamentally the problem is with study sections and how they think. They are incredibly conservative, but in all the wrong ways. They highly value institution of origin, preliminary data from pilots, no matter how shitty, and the sort of hypothesis that seems reasonable from prior literature. There are also a lot of political games on study sections. This means that if you repeat a lie often enough in the literature, and you know people, then voila, you now have support for the same hypothesis in future grants. Peer review is not working for funding. I would say the NIH (in my case) needs to hire completely independent full-time reviewers, and needs to place a real, high emphasis on reproducibility. It is in the NIH's interest to do so, because as I said, its funding will be cut if this continues.
- haecceity 7y ago
  Is this the case in elite institutions? I thought researchers there have more freedom in what they do.
  - cgiles 7y ago
    Haha, no. The more elite the institution, the more money you are expected to bring in. It is compensated somewhat by the fact that a big name institution will give you an advantage in getting grants. There is a bit of a difference between working directly for a university and working for a research nonprofit, like Dana Farber or MD Anderson or whatever. If you work for a university, and you don't pull in a lot of money, what will happen is that you will be expected to teach more hours to "make up for it". You are less likely to be fired. At a nonprofit, if you don't cover your own salary and expenses through grants, you will quickly become unemployed.
inimino 7y ago
There may be some tiny influence that the HN crowd can have through the impact of hiring influences in industry on respective incentives in academia, but overall, I agree completely. Solutions are easy to state but as always the difficulty is finding a lever and a place to stand to put them in practice.
buboard 7y ago
> At this point the only entities large enough to make a difference are probably the NIH and NSF in the U.S. and the relevant EU funding agencies in Europe. I m really not sure about that as the publishing bubble coincides with the rise in funding by public bodies [1]. Funding safe projects is a way to disperse funds to as many people as possible, and that's not necessarily good. At least in EU it feels that way. The politics of funding seem to naturally have gravitated to the current situation. I doubt the model of funding can change with the same stakeholders. [1] https://phoenixrising.me/wp-content/uploads/NIH_Total-Funding_87_2010.jpg https://phoenixrising.me/wp-content/uploads/NIH_Total-Fundin... , https://www.researchgate.net/publication/327917254/figure/fig1/AS:677815078887426@1538615116716/The-annual-rate-of-publications-in-the-biomedical-domain-as-indexed-by-PubMed-The.png https://www.researchgate.net/publication/327917254/figure/fi...
larnmar 7y ago
I wish people would stop using the word “simple” to describe problems like massively realigning all the incentives in a huge decentralised system against the wishes of the existing power-holders in that system. That’s not simple, it’s probably the most complex problem that exists.

nradov 7y ago

If we put more weight on human judgment then we'll get more favoritism, nepotism, and focus on politically popular topics. So we might end up just exchanging one set of problems for another.

inimino 7y ago
At least favoritism has some variability in it. The problem with metrics, besides being gamed, is that they are so uniform.
keldaris 7y ago
You won't get much more focus on politically (in the context of the academia) popular topics, the existing system already imposes that particular bias exceedingly well (how do you think you get citations?). But yes, you'll get more everyday human issues along with the judgment. In my view, that's far preferable to the existing situation, if only for the simple reason that the human problems will vary from place to place, whereas the existing system distorts virtually the entirety of academic research in roughly the same ways everywhere, to the uniform detriment of all.
TeMPOraL 7y ago
I used to be more worried about favoritism, nepotism et al. but these days, I'm more worried about quantifiable metrics. The former are situational, and there's only so much nepotism you can engage with before your peers in other places start considering it to be in bad taste. The latter, however, can be ruthlessly and efficiently optimized, to the full exclusion of any value that's not captured in the metrics.

cgiles 7y ago

At my institute, there is only one metric that matters: dollars you bring in through grants. There are a variety of ways to get to the cash: you can do it with a small number of high-impact papers, a larger number of low-impact papers, by finding a valuable niche, or by chasing fads. So I would say that fundamentally the problem is with study sections and how they think. They are incredibly conservative, but in all the wrong ways. They highly value institution of origin, preliminary data from pilots, no matter how shitty, and the sort of hypothesis that seems reasonable from prior literature. There are also a lot of political games on study sections. This means that if you repeat a lie often enough in the literature, and you know people, then voila, you now have support for the same hypothesis in future grants. Peer review is not working for funding. I would say the NIH (in my case) needs to hire completely independent full-time reviewers, and needs to place a real, high emphasis on reproducibility. It is in the NIH's interest to do so, because as I said, its funding will be cut if this continues.

haecceity 7y ago
Is this the case in elite institutions? I thought researchers there have more freedom in what they do.
- cgiles 7y ago
  Haha, no. The more elite the institution, the more money you are expected to bring in. It is compensated somewhat by the fact that a big name institution will give you an advantage in getting grants. There is a bit of a difference between working directly for a university and working for a research nonprofit, like Dana Farber or MD Anderson or whatever. If you work for a university, and you don't pull in a lot of money, what will happen is that you will be expected to teach more hours to "make up for it". You are less likely to be fired. At a nonprofit, if you don't cover your own salary and expenses through grants, you will quickly become unemployed.

inimino 7y ago

There may be some tiny influence that the HN crowd can have through the impact of hiring influences in industry on respective incentives in academia, but overall, I agree completely. Solutions are easy to state but as always the difficulty is finding a lever and a place to stand to put them in practice.

buboard 7y ago

> At this point the only entities large enough to make a difference are probably the NIH and NSF in the U.S. and the relevant EU funding agencies in Europe. I m really not sure about that as the publishing bubble coincides with the rise in funding by public bodies [1]. Funding safe projects is a way to disperse funds to as many people as possible, and that's not necessarily good. At least in EU it feels that way. The politics of funding seem to naturally have gravitated to the current situation. I doubt the model of funding can change with the same stakeholders. [1] https://phoenixrising.me/wp-content/uploads/NIH_Total-Funding_87_2010.jpg https://phoenixrising.me/wp-content/uploads/NIH_Total-Fundin... , https://www.researchgate.net/publication/327917254/figure/fig1/AS:677815078887426@1538615116716/The-annual-rate-of-publications-in-the-biomedical-domain-as-indexed-by-PubMed-The.png https://www.researchgate.net/publication/327917254/figure/fi...

larnmar 7y ago

I wish people would stop using the word “simple” to describe problems like massively realigning all the incentives in a huge decentralised system against the wishes of the existing power-holders in that system. That’s not simple, it’s probably the most complex problem that exists.