5 ms·

I think the suggestion that "universities could change their hiring policies" is a very good recommendation. There are a lot of problems in academia (bad resear

by joppy 6y ago

I think the suggestion that "universities could change their hiring policies" is a very good recommendation. There are a lot of problems in academia (bad research, lack of diversity, pressure to publish) that could be reduced if universities changed their hiring policy to something further away from the metric of (number of papers published since receiving PhD) / (time since receiving PhD). Of course no university hires explicitly on that metric, but many of the metrics they use are not far from that, mixing in quality of journals and norms for the field, etc. Each of the suggested actions, taken together, would seem to have a very positive improvement on the status quo. Would you care to explain why the suggested actions would be "somewhere between ridiculous and useless?"

meow1032 6y ago
Not OP here, but my issue with the recommendations are that they've pretty accurately listed a whole bunch of mostly structural problems with academia, but all of the suggestions boil down to "we all just need to try harder". You can say something like: "journals need to demand higher standards" but what incentive do they actually have to do so? Then you can counter with "scientists could vote with their feet", but what incentives do they have to do that?? You're asking people to consider seriously damaging their career for some nebulous quality metric. Frankly, having worked in academia long enough to see at least a couple shifts in culture, the only thing I can see that comes out of this is a couple more things get added on to the ever growing checklist of publishing a paper/submitting a grant application. I think we need to get away from the sort of thinking where large structural problems can be solved by tiny incremental improvements. If you really want to solve the problem, one or more of [Granting Agencies|Journals|Universities] has to be completely torn down and built back up.
- joppy 6y ago
  It seems to me, still, that a lot of these problems you bring up can be addressed by universities changing their hiring policies. Which makes sense: academics ultimately rely on universities for their income, and so it is the hiring policies which are setting the perverse incentives. And I don’t think changing hiring policies would be an incremental change, it would be a huge change (and not likely to be made by any university any time soon, since students rank universities on similar metrics to how universities hire staff — a prestigious university will lose prestige even if it changes its hiring policies for the better).
  - meow1032 6y ago
    > academics ultimately rely on universities for their income Sort of, a huge portion of income is from grants, particularly after the first few years from being hired. More importantly, a huge portion of the University income is from grants. When a researcher recieves a grant, there is an "overhead" percentage that goes to the University. Universities hire, in part, to maximize those overheads, which means getting the researchers with the best chance at getting big grants. Changing the hiring process may affect how PHD students act, but once they're "in the system", they are subject to all the same problematic incentives.
    - tejtm 6y ago
      > academics ultimately rely on universities for their income In my decades at it (digital side of bioinformatics) the cash flow is in the other direction.
- mitjak 6y ago
  > one or more of [Granting Agencies|Journals|Universities] has to be completely torn down and built back up right, and unless the new institutions are in a financial vacuum, they will remain built on and affected by broader systems, resulting in conflict of interest.
beagle3 6y ago
OP here; I am not saying the suggestions won’t work. I am saying they will not happen because they mostly undermine the power of those who need to take those actions. You are asking the established people in the academia to give up the source of their power and privilege, and in return there’s some vague promise of Bette overall outcome for humanity. Why should they?
jpeloquin 6y ago
> Would you care to explain why the suggested actions would be "somewhere between ridiculous and useless?" Not OP, but the proposed "solutions" not only add more work items to the ever-growing checklist (as mentioned by meow1032), but to be useful everyone must spend even more time checking everyone else's work items: Solution 1, requiring data sharing and preregistration, greatly increases the work of peer review, perhaps by an order of magnitude. Someone needs to check that the data produces the published results and that the final analysis plan matched the preregistration. That is hard, time-consuming volunteer work, with no reward incentive. Current peer review currently trusts the authors did what they said they did, correctly, and it still takes 4-12 hours to review an article. Most reviewers cut corners. If no one does the work to check the open data or preregistrations, "open science" will be merely performative, with no quality improvement. Solution 2, changing hiring policies to "look beyond publication and citation numbers", is pretty much what hiring committees already do. But with ~ 200 applicants per job opening the depth of examination per applicant is somewhat limited. As in solution 1, lack of time for deep checks is a problem. Applicants who are well-networked with good pre-existing reputations (i.e, who are plugged into the web of trust) get hired; everyone else doesn't. From a perspective of research quality, this may be a good thing. Solution 3, funders fund boring / rigorous research, could improve matters in theory. But with only enough money to fund ~ 1 in 10 proposals, projected impact will always be an overwhelming concern. Proposals will include a "research integrity" section or similar and nothing substantive will change. Solution 4, scientists "vote with their feet" (stop participating in the dysfunctional parts of the system), is a call for people to come up with their own solutions or support other proposed solutions, not a solution in its own right. Ironically, it is perhaps the most useful because it pushes back on the idea that poor quality science is inevitable under the current structure. "Perverse incentives" must not become a generally accepted excuse to sacrifice scientific integrity for the benefit of one's own career. Science is meant to discover new information. Without a culture of integrity, that information will always be suspect, regardless of what top-down interventions are attempted. An effective intervention must reduce workload or at least break even, not increase it. Or increase the resources available. Otherwise people will be forced (actually forced, not just incentivized; there are only so many work hours each day) to cut even more corners elsewhere to make up for lost time.

by joppy 6y ago

meow1032 6y ago

Not OP here, but my issue with the recommendations are that they've pretty accurately listed a whole bunch of mostly structural problems with academia, but all of the suggestions boil down to "we all just need to try harder". You can say something like: "journals need to demand higher standards" but what incentive do they actually have to do so? Then you can counter with "scientists could vote with their feet", but what incentives do they have to do that?? You're asking people to consider seriously damaging their career for some nebulous quality metric. Frankly, having worked in academia long enough to see at least a couple shifts in culture, the only thing I can see that comes out of this is a couple more things get added on to the ever growing checklist of publishing a paper/submitting a grant application. I think we need to get away from the sort of thinking where large structural problems can be solved by tiny incremental improvements. If you really want to solve the problem, one or more of [Granting Agencies|Journals|Universities] has to be completely torn down and built back up.

joppy 6y ago
It seems to me, still, that a lot of these problems you bring up can be addressed by universities changing their hiring policies. Which makes sense: academics ultimately rely on universities for their income, and so it is the hiring policies which are setting the perverse incentives. And I don’t think changing hiring policies would be an incremental change, it would be a huge change (and not likely to be made by any university any time soon, since students rank universities on similar metrics to how universities hire staff — a prestigious university will lose prestige even if it changes its hiring policies for the better).
- meow1032 6y ago
  > academics ultimately rely on universities for their income Sort of, a huge portion of income is from grants, particularly after the first few years from being hired. More importantly, a huge portion of the University income is from grants. When a researcher recieves a grant, there is an "overhead" percentage that goes to the University. Universities hire, in part, to maximize those overheads, which means getting the researchers with the best chance at getting big grants. Changing the hiring process may affect how PHD students act, but once they're "in the system", they are subject to all the same problematic incentives.
  - tejtm 6y ago
    > academics ultimately rely on universities for their income In my decades at it (digital side of bioinformatics) the cash flow is in the other direction.
mitjak 6y ago
> one or more of [Granting Agencies|Journals|Universities] has to be completely torn down and built back up right, and unless the new institutions are in a financial vacuum, they will remain built on and affected by broader systems, resulting in conflict of interest.

beagle3 6y ago

OP here; I am not saying the suggestions won’t work. I am saying they will not happen because they mostly undermine the power of those who need to take those actions. You are asking the established people in the academia to give up the source of their power and privilege, and in return there’s some vague promise of Bette overall outcome for humanity. Why should they?

jpeloquin 6y ago

> Would you care to explain why the suggested actions would be "somewhere between ridiculous and useless?" Not OP, but the proposed "solutions" not only add more work items to the ever-growing checklist (as mentioned by meow1032), but to be useful everyone must spend even more time checking everyone else's work items: Solution 1, requiring data sharing and preregistration, greatly increases the work of peer review, perhaps by an order of magnitude. Someone needs to check that the data produces the published results and that the final analysis plan matched the preregistration. That is hard, time-consuming volunteer work, with no reward incentive. Current peer review currently trusts the authors did what they said they did, correctly, and it still takes 4-12 hours to review an article. Most reviewers cut corners. If no one does the work to check the open data or preregistrations, "open science" will be merely performative, with no quality improvement. Solution 2, changing hiring policies to "look beyond publication and citation numbers", is pretty much what hiring committees already do. But with ~ 200 applicants per job opening the depth of examination per applicant is somewhat limited. As in solution 1, lack of time for deep checks is a problem. Applicants who are well-networked with good pre-existing reputations (i.e, who are plugged into the web of trust) get hired; everyone else doesn't. From a perspective of research quality, this may be a good thing. Solution 3, funders fund boring / rigorous research, could improve matters in theory. But with only enough money to fund ~ 1 in 10 proposals, projected impact will always be an overwhelming concern. Proposals will include a "research integrity" section or similar and nothing substantive will change. Solution 4, scientists "vote with their feet" (stop participating in the dysfunctional parts of the system), is a call for people to come up with their own solutions or support other proposed solutions, not a solution in its own right. Ironically, it is perhaps the most useful because it pushes back on the idea that poor quality science is inevitable under the current structure. "Perverse incentives" must not become a generally accepted excuse to sacrifice scientific integrity for the benefit of one's own career. Science is meant to discover new information. Without a culture of integrity, that information will always be suspect, regardless of what top-down interventions are attempted. An effective intervention must reduce workload or at least break even, not increase it. Or increase the resources available. Otherwise people will be forced (actually forced, not just incentivized; there are only so many work hours each day) to cut even more corners elsewhere to make up for lost time.