5 ms·

How to fix?

aalleavitch 7y ago
Science shouldn't be funded this way
keldaris 7y ago
The answer is simple, but it's also one that's completely useless for anyone reading this post. What needs to happen is a gradual realignment in both the funding and hiring criteria across all of academia. There needs to be less emphasis on quantitative metrics like citation counts, h-indices and other trivially gameable nonsense, and more emphasis on human judgment along with a set of basic criteria to provide a floor to research quality across the board (things that every good scientist should do, as opposed to things that you should relentlessly maximize). This would require the non-scientific managerial class to abrogate some of the power they now wield in the academia and relevant funding institutions, therefore is very unlikely to happen. And - probably, most annoyingly to the HN crowd - there is nothing meaningful that technological or business disruption can accomplish here. These are systemic social problems that have been continuously and systemically building up since (approximately) WW2. At this point the only entities large enough to make a difference are probably the NIH and NSF in the U.S. and the relevant EU funding agencies in Europe. In my judgment, these are precisely the agencies least likely to institute such change, so here we are.
- nradov 7y ago
  If we put more weight on human judgment then we'll get more favoritism, nepotism, and focus on politically popular topics. So we might end up just exchanging one set of problems for another.
  - inimino 7y ago
    At least favoritism has some variability in it. The problem with metrics, besides being gamed, is that they are so uniform.
  - keldaris 7y ago
    You won't get much more focus on politically (in the context of the academia) popular topics, the existing system already imposes that particular bias exceedingly well (how do you think you get citations?). But yes, you'll get more everyday human issues along with the judgment. In my view, that's far preferable to the existing situation, if only for the simple reason that the human problems will vary from place to place, whereas the existing system distorts virtually the entirety of academic research in roughly the same ways everywhere, to the uniform detriment of all.
  - TeMPOraL 7y ago
    I used to be more worried about favoritism, nepotism et al. but these days, I'm more worried about quantifiable metrics. The former are situational, and there's only so much nepotism you can engage with before your peers in other places start considering it to be in bad taste. The latter, however, can be ruthlessly and efficiently optimized, to the full exclusion of any value that's not captured in the metrics.
- cgiles 7y ago
  At my institute, there is only one metric that matters: dollars you bring in through grants. There are a variety of ways to get to the cash: you can do it with a small number of high-impact papers, a larger number of low-impact papers, by finding a valuable niche, or by chasing fads. So I would say that fundamentally the problem is with study sections and how they think. They are incredibly conservative, but in all the wrong ways. They highly value institution of origin, preliminary data from pilots, no matter how shitty, and the sort of hypothesis that seems reasonable from prior literature. There are also a lot of political games on study sections. This means that if you repeat a lie often enough in the literature, and you know people, then voila, you now have support for the same hypothesis in future grants. Peer review is not working for funding. I would say the NIH (in my case) needs to hire completely independent full-time reviewers, and needs to place a real, high emphasis on reproducibility. It is in the NIH's interest to do so, because as I said, its funding will be cut if this continues.
  - haecceity 7y ago
    Is this the case in elite institutions? I thought researchers there have more freedom in what they do.
    - cgiles 7y ago
      Haha, no. The more elite the institution, the more money you are expected to bring in. It is compensated somewhat by the fact that a big name institution will give you an advantage in getting grants. There is a bit of a difference between working directly for a university and working for a research nonprofit, like Dana Farber or MD Anderson or whatever. If you work for a university, and you don't pull in a lot of money, what will happen is that you will be expected to teach more hours to "make up for it". You are less likely to be fired. At a nonprofit, if you don't cover your own salary and expenses through grants, you will quickly become unemployed.
- inimino 7y ago
  There may be some tiny influence that the HN crowd can have through the impact of hiring influences in industry on respective incentives in academia, but overall, I agree completely. Solutions are easy to state but as always the difficulty is finding a lever and a place to stand to put them in practice.
- buboard 7y ago
  > At this point the only entities large enough to make a difference are probably the NIH and NSF in the U.S. and the relevant EU funding agencies in Europe. I m really not sure about that as the publishing bubble coincides with the rise in funding by public bodies [1]. Funding safe projects is a way to disperse funds to as many people as possible, and that's not necessarily good. At least in EU it feels that way. The politics of funding seem to naturally have gravitated to the current situation. I doubt the model of funding can change with the same stakeholders. [1] https://phoenixrising.me/wp-content/uploads/NIH_Total-Funding_87_2010.jpg https://phoenixrising.me/wp-content/uploads/NIH_Total-Fundin... , https://www.researchgate.net/publication/327917254/figure/fig1/AS:677815078887426@1538615116716/The-annual-rate-of-publications-in-the-biomedical-domain-as-indexed-by-PubMed-The.png https://www.researchgate.net/publication/327917254/figure/fi...
- larnmar 7y ago
  I wish people would stop using the word “simple” to describe problems like massively realigning all the incentives in a huge decentralised system against the wishes of the existing power-holders in that system. That’s not simple, it’s probably the most complex problem that exists.
geoalchimista 7y ago
Not the OP, but as a working scientist, it seems to me there are two important issues, among many others. (i) Reproducibility is not a criterion for tenure evaluation, therefore it bears little relevance for career advancement for young scientists. This is an overworked and underpaid group in academia. (In the bay area, they earn 1/3--1/2 the salary of an entry level software engineer but probably work 50% more hours.) They simply don't have the luxury to take time off to reproduce someone else's work. (ii) No major funding agency would be willing to support the kind of work that reproduces published studies. When something is published, it is considered "done" and not worth spending more money on it. There needs to be a sea change. But sadly, the academia is at best paying lip services to the structural problems in the "reproducibility crisis". I'd expect it to continue like this for most fields.
inimino 7y ago
Making a significant contribution to your field used to be a reasonable requirement for a Ph.D. However, we are churning out more and more Ph.Ds, and there just aren't that many new discoveries to be made. This is creating a pressure to create more and more splintered fields, and more and more useless publications that nobody will ever read. Furthermore the academic system uses publication to measure the value of professors or researchers, creating yet more useless crap published every year. Only changing the incentives can improve the situation and it seems academia left to its own devices never will. Maybe it's time for governments to take a more active role in basic research again. Maybe we need more "patent clerk" jobs where people have time to think without having to chase structured research grants.
vanniv 7y ago
That's the hundred-billion-dollar question, isn't it. My radical proposal is to separate the research institutions from the universities -- especially the public universities. If you want public research institutes, that's fine -- but they should be their own entities. The university system, with its mix of research and instruction, and its tenure-vs-non-tenure system creates terrible incentives. Top researchers are rarely also the best instructors -- and so it makes little sense for the two occupations to be so intimately tied. The university funding model is terribly destructive to top science labs as well -- the university sees them as a source of prestige, sure, but more importantly as a source of that all-too-critical grant money (entirely too much of which is bogarted by the rest of the university) -- putting many of the labs in this place where they have to keep securing grants at an ever-increasing pace no matter what, to keep the rest of the university flush with cash to spend elsewhere. There's really no good reason why research needs to be done at a university, or why a research institution needs a body of lecturers and students.
- CrazyStat 7y ago
  I absolutely agree. My wife did a PhD at an elite private university, undergrad tuition north of $50k/year not including room and board and other expenses. At one point three senior professors in her department went to the department chair and the dean with a proposal to revamp a notoriously unpopular intro class in their department. Rather than having it taught by adjuncts or postdocs, the three of them would team teach it. They would change the syllabus to bring it up to date with the current methods in the field--it had hardly been touched in two decades. They were told to forget about it. The University said it was a waste of time for them to teach undergrads. Presumably the kids applying to these types of universities are not aware of how little concern the institution has for the quality of their education (the perceived quality, on the other hand, is extremely important--but only loosely connected to the actual quality).
  - vanniv 7y ago
    The current undergraduate university market is almost entirely a reputational marketplace -- people are shopping for reputation/prestige. It turns out that prestige has very little to do with undergraduate education quality. This used to make lots of sense -- for centuries, college was more important as a place to meet other people as it was a place to actually learn useful skills -- and so you were primarily trying to go to the highest-class place you could get yourself in to. In this day and age, though, I don't think that's the right way to approach choosing a school.
BenoitEssiambre 7y ago
Until university positions and research grants stop being given out based on prior research _results_ (which is what journals tend to look for), we won't be able to trust the research performed there. There are millions of dollars on the line for researchers involved. It is the difference between a well paid career and a life of destitution while being a slave to huge student debt. Professorship and research grants should be given out based on criteria that are only incidental to research results. Evaluate profs and grants based on: 1. Domain knowledge (test the applicants) 2. Math skills (test the applicants, makes sure they know about how to avoid p-hacking, preregistration etc.) 3. Motivation and leadership. 4. Prior and current research _proposals_ (but without looking at the results or whether they have been published). 5. Other skills such as communication, interpersonal skills, outstanding achievements etc. Universities should not rely on journals to evaluate their professors. This corrupts the whole system. Journals have different goals. They want to publish well done research with interesting results. Universities should hire researchers that do good quality research with interesting _questions_ regardless of the results. If universities keep giving out jobs based on having generated interesting publishable results, they are going to keep getting researchers that ignore biases and and ignore bad science practices in order to generate publishable (but unreliable and often false) results.
JMTQp8lwXL 7y ago
Historically, the sciences were limited to people that had the wealth to independently research and verify their work. These days, people perform scientific research for a living wage (which in itself, I am not saying is a bad thing --scientific progress is a good thing-- no doubt), but people have to optimize for income producing potential, which means pumping out papers, even if they are of dubious quality, which may or may not intentionally be done. Not all scientists are optimizing for number of publications, but a good number are.
scarejunba 7y ago
It's already fixed to some degree. Any good scientist who can capture economic value spin-off to private labs. It's the good scientist who produces economic value but cannot capture it that we have to support but maybe that person cannot easily be found. The state just has to overfund and accept that some large percent goes to nonsense. In my mind, that's okay, though, since the guys who can capture economic value can do a lot with what we have.
knuteson 7y ago
Reward being right, penalize being wrong, and transfer information in a manner that facilitates determining what is right and what is wrong. One page (https://www.kn-x.com/static/PWFeb17forum.pdf https://www.kn-x.com/static/PWFeb17forum.pdf), two page (https://ssrn.com/abstract=2835131 https://ssrn.com/abstract=2835131), and three page (https://ssrn.com/abstract=2713998 https://ssrn.com/abstract=2713998) versions of this are available. Further details are provided at http://kn-x.com http://kn-x.com. I doubt anyone on this thread is going to like this solution ... but I think most on this thread agree that the underlying problem is the incentives, that fixing the problem requires changing the incentives, and that changing the incentives within the existing system is very hard.

How to fix?

Science shouldn't be funded this way

The answer is simple, but it's also one that's completely useless for anyone reading this post. What needs to happen is a gradual realignment in both the funding and hiring criteria across all of academia. There needs to be less emphasis on quantitative metrics like citation counts, h-indices and other trivially gameable nonsense, and more emphasis on human judgment along with a set of basic criteria to provide a floor to research quality across the board (things that every good scientist should do, as opposed to things that you should relentlessly maximize). This would require the non-scientific managerial class to abrogate some of the power they now wield in the academia and relevant funding institutions, therefore is very unlikely to happen. And - probably, most annoyingly to the HN crowd - there is nothing meaningful that technological or business disruption can accomplish here. These are systemic social problems that have been continuously and systemically building up since (approximately) WW2. At this point the only entities large enough to make a difference are probably the NIH and NSF in the U.S. and the relevant EU funding agencies in Europe. In my judgment, these are precisely the agencies least likely to institute such change, so here we are.

nradov 7y ago
If we put more weight on human judgment then we'll get more favoritism, nepotism, and focus on politically popular topics. So we might end up just exchanging one set of problems for another.
- inimino 7y ago
  At least favoritism has some variability in it. The problem with metrics, besides being gamed, is that they are so uniform.
- keldaris 7y ago
  You won't get much more focus on politically (in the context of the academia) popular topics, the existing system already imposes that particular bias exceedingly well (how do you think you get citations?). But yes, you'll get more everyday human issues along with the judgment. In my view, that's far preferable to the existing situation, if only for the simple reason that the human problems will vary from place to place, whereas the existing system distorts virtually the entirety of academic research in roughly the same ways everywhere, to the uniform detriment of all.
- TeMPOraL 7y ago
  I used to be more worried about favoritism, nepotism et al. but these days, I'm more worried about quantifiable metrics. The former are situational, and there's only so much nepotism you can engage with before your peers in other places start considering it to be in bad taste. The latter, however, can be ruthlessly and efficiently optimized, to the full exclusion of any value that's not captured in the metrics.
cgiles 7y ago
At my institute, there is only one metric that matters: dollars you bring in through grants. There are a variety of ways to get to the cash: you can do it with a small number of high-impact papers, a larger number of low-impact papers, by finding a valuable niche, or by chasing fads. So I would say that fundamentally the problem is with study sections and how they think. They are incredibly conservative, but in all the wrong ways. They highly value institution of origin, preliminary data from pilots, no matter how shitty, and the sort of hypothesis that seems reasonable from prior literature. There are also a lot of political games on study sections. This means that if you repeat a lie often enough in the literature, and you know people, then voila, you now have support for the same hypothesis in future grants. Peer review is not working for funding. I would say the NIH (in my case) needs to hire completely independent full-time reviewers, and needs to place a real, high emphasis on reproducibility. It is in the NIH's interest to do so, because as I said, its funding will be cut if this continues.
- haecceity 7y ago
  Is this the case in elite institutions? I thought researchers there have more freedom in what they do.
  - cgiles 7y ago
    Haha, no. The more elite the institution, the more money you are expected to bring in. It is compensated somewhat by the fact that a big name institution will give you an advantage in getting grants. There is a bit of a difference between working directly for a university and working for a research nonprofit, like Dana Farber or MD Anderson or whatever. If you work for a university, and you don't pull in a lot of money, what will happen is that you will be expected to teach more hours to "make up for it". You are less likely to be fired. At a nonprofit, if you don't cover your own salary and expenses through grants, you will quickly become unemployed.
inimino 7y ago
There may be some tiny influence that the HN crowd can have through the impact of hiring influences in industry on respective incentives in academia, but overall, I agree completely. Solutions are easy to state but as always the difficulty is finding a lever and a place to stand to put them in practice.
buboard 7y ago
> At this point the only entities large enough to make a difference are probably the NIH and NSF in the U.S. and the relevant EU funding agencies in Europe. I m really not sure about that as the publishing bubble coincides with the rise in funding by public bodies [1]. Funding safe projects is a way to disperse funds to as many people as possible, and that's not necessarily good. At least in EU it feels that way. The politics of funding seem to naturally have gravitated to the current situation. I doubt the model of funding can change with the same stakeholders. [1] https://phoenixrising.me/wp-content/uploads/NIH_Total-Funding_87_2010.jpg https://phoenixrising.me/wp-content/uploads/NIH_Total-Fundin... , https://www.researchgate.net/publication/327917254/figure/fig1/AS:677815078887426@1538615116716/The-annual-rate-of-publications-in-the-biomedical-domain-as-indexed-by-PubMed-The.png https://www.researchgate.net/publication/327917254/figure/fi...
larnmar 7y ago
I wish people would stop using the word “simple” to describe problems like massively realigning all the incentives in a huge decentralised system against the wishes of the existing power-holders in that system. That’s not simple, it’s probably the most complex problem that exists.

geoalchimista 7y ago

Not the OP, but as a working scientist, it seems to me there are two important issues, among many others. (i) Reproducibility is not a criterion for tenure evaluation, therefore it bears little relevance for career advancement for young scientists. This is an overworked and underpaid group in academia. (In the bay area, they earn 1/3--1/2 the salary of an entry level software engineer but probably work 50% more hours.) They simply don't have the luxury to take time off to reproduce someone else's work. (ii) No major funding agency would be willing to support the kind of work that reproduces published studies. When something is published, it is considered "done" and not worth spending more money on it. There needs to be a sea change. But sadly, the academia is at best paying lip services to the structural problems in the "reproducibility crisis". I'd expect it to continue like this for most fields.

inimino 7y ago

Making a significant contribution to your field used to be a reasonable requirement for a Ph.D. However, we are churning out more and more Ph.Ds, and there just aren't that many new discoveries to be made. This is creating a pressure to create more and more splintered fields, and more and more useless publications that nobody will ever read. Furthermore the academic system uses publication to measure the value of professors or researchers, creating yet more useless crap published every year. Only changing the incentives can improve the situation and it seems academia left to its own devices never will. Maybe it's time for governments to take a more active role in basic research again. Maybe we need more "patent clerk" jobs where people have time to think without having to chase structured research grants.

vanniv 7y ago

That's the hundred-billion-dollar question, isn't it. My radical proposal is to separate the research institutions from the universities -- especially the public universities. If you want public research institutes, that's fine -- but they should be their own entities. The university system, with its mix of research and instruction, and its tenure-vs-non-tenure system creates terrible incentives. Top researchers are rarely also the best instructors -- and so it makes little sense for the two occupations to be so intimately tied. The university funding model is terribly destructive to top science labs as well -- the university sees them as a source of prestige, sure, but more importantly as a source of that all-too-critical grant money (entirely too much of which is bogarted by the rest of the university) -- putting many of the labs in this place where they have to keep securing grants at an ever-increasing pace no matter what, to keep the rest of the university flush with cash to spend elsewhere. There's really no good reason why research needs to be done at a university, or why a research institution needs a body of lecturers and students.

CrazyStat 7y ago
I absolutely agree. My wife did a PhD at an elite private university, undergrad tuition north of $50k/year not including room and board and other expenses. At one point three senior professors in her department went to the department chair and the dean with a proposal to revamp a notoriously unpopular intro class in their department. Rather than having it taught by adjuncts or postdocs, the three of them would team teach it. They would change the syllabus to bring it up to date with the current methods in the field--it had hardly been touched in two decades. They were told to forget about it. The University said it was a waste of time for them to teach undergrads. Presumably the kids applying to these types of universities are not aware of how little concern the institution has for the quality of their education (the perceived quality, on the other hand, is extremely important--but only loosely connected to the actual quality).
- vanniv 7y ago
  The current undergraduate university market is almost entirely a reputational marketplace -- people are shopping for reputation/prestige. It turns out that prestige has very little to do with undergraduate education quality. This used to make lots of sense -- for centuries, college was more important as a place to meet other people as it was a place to actually learn useful skills -- and so you were primarily trying to go to the highest-class place you could get yourself in to. In this day and age, though, I don't think that's the right way to approach choosing a school.

BenoitEssiambre 7y ago

Until university positions and research grants stop being given out based on prior research _results_ (which is what journals tend to look for), we won't be able to trust the research performed there. There are millions of dollars on the line for researchers involved. It is the difference between a well paid career and a life of destitution while being a slave to huge student debt. Professorship and research grants should be given out based on criteria that are only incidental to research results. Evaluate profs and grants based on: 1. Domain knowledge (test the applicants) 2. Math skills (test the applicants, makes sure they know about how to avoid p-hacking, preregistration etc.) 3. Motivation and leadership. 4. Prior and current research _proposals_ (but without looking at the results or whether they have been published). 5. Other skills such as communication, interpersonal skills, outstanding achievements etc. Universities should not rely on journals to evaluate their professors. This corrupts the whole system. Journals have different goals. They want to publish well done research with interesting results. Universities should hire researchers that do good quality research with interesting _questions_ regardless of the results. If universities keep giving out jobs based on having generated interesting publishable results, they are going to keep getting researchers that ignore biases and and ignore bad science practices in order to generate publishable (but unreliable and often false) results.

JMTQp8lwXL 7y ago

Historically, the sciences were limited to people that had the wealth to independently research and verify their work. These days, people perform scientific research for a living wage (which in itself, I am not saying is a bad thing --scientific progress is a good thing-- no doubt), but people have to optimize for income producing potential, which means pumping out papers, even if they are of dubious quality, which may or may not intentionally be done. Not all scientists are optimizing for number of publications, but a good number are.

scarejunba 7y ago

It's already fixed to some degree. Any good scientist who can capture economic value spin-off to private labs. It's the good scientist who produces economic value but cannot capture it that we have to support but maybe that person cannot easily be found. The state just has to overfund and accept that some large percent goes to nonsense. In my mind, that's okay, though, since the guys who can capture economic value can do a lot with what we have.

knuteson 7y ago

Reward being right, penalize being wrong, and transfer information in a manner that facilitates determining what is right and what is wrong. One page (https://www.kn-x.com/static/PWFeb17forum.pdf https://www.kn-x.com/static/PWFeb17forum.pdf), two page (https://ssrn.com/abstract=2835131 https://ssrn.com/abstract=2835131), and three page (https://ssrn.com/abstract=2713998 https://ssrn.com/abstract=2713998) versions of this are available. Further details are provided at http://kn-x.com http://kn-x.com. I doubt anyone on this thread is going to like this solution ... but I think most on this thread agree that the underlying problem is the incentives, that fixing the problem requires changing the incentives, and that changing the incentives within the existing system is very hard.