5 ms·

It should be noted that the scientists themselves, and particularly the top scientists, since the others are not as easily heard or listened to, have largely co

It should be noted that the scientists themselves, and particularly the top scientists, since the others are not as easily heard or listened to, have largely contributed to the development of this particular academic system in which the quality of one's work and academic decisions (recruitment, tenure, salary) are evaluated primarily, but not exclusively, by: total number of publications, citations, grants obtained and their monetary values, number of students, impact factor of the journals in which the research was published, h-index, gender, sexual orientation, ethnicity, and public visibility. Did I miss anything? For once, neither politicians nor administrators were responsible for a system of "perverse incentives," since these gray figures are not involved in either hiring or tenure decisions. Case in point: one of the leading scientists in a field in which I dabble, and former chief scientist of The Nature Conservancy, routinely had his name on dozens of articles each year. Given the volume of his other academic commitments, at most he would be able to take a look at those papers, certainly he was not able to contribute meaningfully to (any of) them. So why does a top scientist, at the end of his career, whose place in the pecking order is established, still feel this need to have his name on papers to which he has contributed nothing, instead of taking a different path, in which he demonstrates, with facts, that unattainable "productivity" is not what scientists should aspire to? Someone might say, "Well, maybe it's not him, but his students or collaborators who want his name, so they can say they published a paper with an important scientist." And why would they want to say that? To impress other scientists, the ones who will give them professorships, appointments, fellowships, etc. A world I am lucky to have left behind.

majormajor 3y ago
I think this in near-inevitable in any org or federation of orgs of people with more than a few hundred people involved (wild-ass-guess, substitute your own). Once you have too much scale for a few people to truly know what's going on, all that's left to keep people working on the right incentives is trust, and inevitably you'll have some bad actors at some point, and after that you put in some metrics and then it gets worse cause even good actors pursuing those metrics may have gone off the rails. The only places I've seen avoid it moderately successfully have had exceptional people managers, like 1-in-100/1000 level, to minimize the appeal of playing the game instead of doing the work.
- borroka 3y ago
  Every system that was later changed for something better was, at one time, considered inevitable, natural, ineluctable.
- Bjartr 3y ago
  > too much scale for a few people to truly know what's going on Somewhat related is Dunbar's number https://en.m.wikipedia.org/wiki/Dunbar%27s_number https://en.m.wikipedia.org/wiki/Dunbar%27s_number
- brnaftr361 3y ago
  I think "bad actors" is a poor framing, this extends well beyond malfeasance. Even in a perfect system, at some scale people need these elementary quantities to be able to model and predict exogenous factors or justify certain other behaviors to operate. Goodhart's law does tend to come into effect - but Goodhart's law may come to dominate these systems through far more than malfeasance. Just phone game-esque decay of information from one actor to the next can cause it, moreover this occurs across generations which I would posit accelerates the decay. One "generation" may have firsthand experience of the why and how, and upon relaying it to the second a loss occurs, and the third may not ever receive it or they're given some highly heuristicised version which artlessly excises nuance. Very quickly the system is reliant on decayed information which informs garbage processes, but this is ossified as the standard operating procedure. I would posit that, especially in the temporal domain, this decay should prompt us to reconsider our tolerance for staple institutions and our willingness to facilitate their development, sustainment, and immortalization and thus limit scale while maintaining the integrity of vision and information while jointly allowing voids to open and be closed by legitimate and ostensibly better suited successors. Right now we're making models of cancer, many of which are metastatic. Nice handle, by the way.
cycomanic 3y ago
I think this is a misrepresentation of reality. Generally there has been strong pressure from government (and the public) on "justifying" their spending through measurable outcomes. Throughout the 90s there were numerous media campaigns about "useless" research and we can still see the same sentiments even here on HN. Yes academics (well typically more the lab leaders etc), we're involved in some of the system that we see now, but it was in reaction to the pressure from politics and society. What do you expect people to do when the places that fund them start to demand measurable outcomes for something that is inherently difficult to measure. You come up with some (often arbitrary) metrics. That whole dynamic then just became an anweful feedback process.
- borroka 3y ago
  I strongly disagree. The public (let's take the 75% of the educational attainment level folks, excluding PhDs, so non-academic people who are educated enough) knows nothing about research, how research is conducted, what a paper is, how a paper is published. Zero clues. You can imagine what the bottom 75% think. Researchers are considered by the public to be little to no more than glorified teachers. It is like saying that bigger and better weapons are invented and built to justify the Defense budget to the public, where for the public a bomb is a bomb is a bomb. How do I know? (1) I was an academic and I often had low-level conversations about my research, in my home country, in the U.S., in Europe, South America, everywhere. Not a single interlocutor outside of academia knew what is the typical process to follow to have some research results published in a scientific journal, or whether a publication in a certain journal can be indicative of more or less cited/quality research. (2) I read comments here and there on newspapers, forums, social media, etc. Letters to the editor of your local newspaper are considered to be equivalent to a publication in Cell. (3) Ask, for fun, like I used to do, your regular guy if (a) an electron is bigger than an atom, (b) how many people live on earth. Re: research is useless. Most academic science is useless, and the uselessness of 95 percent of research is somehow considered inevitable by the "scientific community." But in that 95 percent there is a portion of research that is speculative and ultimately turns out to be useless, which is a good thing to have, and a portion of research (the majority) that serves only to pay the salaries of the researchers involved, is clear to everyone and their cousin that it will go nowhere, and is published in journals that no one, and rightly so, will ever read.
bsder 3y ago
> For once, neither politicians nor administrators were responsible for a system of "perverse incentives," since these gray figures are not involved in either hiring or tenure decisions. This is incorrect. The government and politicians are definitely some of the biggest factors. The collapse of the longer timescale corporate R&D labs meant that research funding transferred to the government. Before that, if universities got too demanding, the academics would leave to the corporate labs. These factors combined with Bayh-Dole and the lottery nature of "intellectual property" to cause universities to start chasing funding of short-term gains rather than long-term research. (Famous example: Katalin Kariko, the woman behind much of the mRNA groundwork, was denied tenure and had a very difficult time getting funding).
- borroka 3y ago
  The comment makes little sense, the way that is worded. When and where were politicians responsible for the collapse of the "the longer timescale corporate R&D labs"? Are you thinking of ATT/Bell Labs? Are you thinking of just the U.S. and just the East Coast of the U.S., ignoring a much larger world? "Before that, if universities got too demanding, the academics would leave to the corporate labs." Academics who? Let's say biologists, who are a massive presence in academia, exactly where in the private sector were they working before the aforementioned collapse of "the longer timescale corporate R&D labs"? Pharma is strong, maybe stronger than ever, but most of the top pharma-adjacent biologists prefer to stay in academia and collaborate with Pharma, more than the opposite. You are just thinking about a bunch of labs (Bell labs?) able to move maybe 1% of the academics at most in their heyday. And it seems to me, like I explained, that the "productivity pressure" is coming from academics themselves, not from outside academia. Before the 70's, it was fairly easy for a moderately successful Ph.D. to get a position in academia, but the number of positions available did not keep up with the increase of Ph.D.'s who wanted to do academic research. And the "productivity" measurements started.
  - bsder 3y ago
    > When and where were politicians responsible for the collapse of the "the longer timescale corporate R&D labs"? I did not say this and am quite mystified how you managed to construe my words to imply that. The fact that those corporate labs had a longer term focus and then disappeared is simply a fact. > Academics who? Let's say biologists Before PCR (1984ish?), biology had such a vanishingly small presence relative to research funding that it almost doesn't count. Every big company up through about 1970-1980 had very large R&D departments that had been in existence for anywhere between half to a whole century. Sure: IBM, Bell, and Xerox are remembered strongly. However, Kodak and Polaroid had big chemical engineering departments. RCA and Motorola were household names. Every big name in steel (US Steel, Bethlehem, etc.) all had lots of research in metallurgy. Corning was a huge driving force behind ceramic chemistry. Dupont similarly for plastics. When those big companies went down, so, too, did their research departments and the funding driving them. This was a problem because companies funded a LOT of the local universities--Pittsburgh universities, for example, were still using equipment even in the 1990s that had been funded decades earlier by the big steel mills. The collapse of the corporate giants meant that the percentage of academic research funding from government suddenly went way up.
wrp 3y ago
> ...the scientists themselves...have largely contributed to the development of this particular academic system... This is a strong claim, which I think needs some references to support it. Watching my dad's generation in academia from the 1950s forward and my own from the 1980s forward, my impression has been that this change in campus culture has been entirely driven by the administrative level.
- ttymck 3y ago
  Who do you think the administrators of physical science colleges are? EDIT: I guess you can omit "physical science", I suppose we are talking about research across all disciplines. My point remains.
  - wrp 3y ago
    Ultimate decision making is largely by people with a background in business, not science. It seems to be poorly understood even among faculty that administrative policy is ultimately set by the board of trustees/directors. These boards are filled with people who gained prominence through social or business activity. My impression is that successful research scientists rarely if ever land on a board of directors. Although scientists are hired to fill administrative positions, their continued employment depends on their following the directives of the board.
    - borroka 3y ago
      But the context was hiring/tenure/grants assignment decisions and how academic productivity became a major factor in those decisions. Business people have no say in any hiring decisions made at universities. The hiring committee is made up of faculties at the department hiring the new assistant(often)/associate(rarely)/full(very rare) professor. Tenure is not decided by any business or administrative person working at universities. Same for grant funding in the sciences.
      - deleted 3y ago
        [deleted]
nico 3y ago
And this is how you get the case of the president of Stanford, who not only benefited tremendously from all the research that he was supposedly a part of or leading, but when it came to light that a lot of it was fraudulent, he was able to claim he wasn’t really involved that closely So, a lot of the benefit, but barely any of the accountability
- borroka 3y ago
  That's more of a case of elite's protection and general, outside very special cases, immunity from, broadly speaking, prosecution. If the same had happened to an assistant professor, non-minority, they would have been kicked out long ago.
n4r9 3y ago
Your "leading scientist" anecdote only proves that the system is rigged to reward scrounging, and that scientists therefore feel a pressure to scrounge, even at the top level. Does that mean scientists are to blame for the system? I don't think it shows that. At most it shows that genuine scientists aren't very optimistic about their power to change the system.
- borroka 3y ago
  Let's take hiring decisions. Who's assessing the number of papers published by candidates, the number of citations those papers received, the impact factors of the journals in which the papers were published? Who's writing the recommendation/reference letters for the candidates? The answer is other scientists, and greater weight is given to the opinion, assessment, letters of "leading scientists". The "system" was set up over time by scientists, for scientists.
  - n4r9 3y ago
    Scientists might feel pressure to take impact factors into account when hiring. That doesn't necessarily mean that they want to promote impact factors, or that they think impact factors are good for science. It could for example mean that the university/institute has made their funding and other perks contingent on achieving certain metrics. Blaming scientists for this is like blaming employees for poor working conditions because they signed a contract. It ignores the fact that they had no choice but to sign the contract, if they want a job.
    - borroka 3y ago
      What you propose is a hypothesis. Fine. The hypothesis is not supported by any clear evidence. I cannot think of a public university administrator who is not a scientist/researcher/faculty who is familiar with impact factors, the relative strength of journals outside the top 5 and then maybe (Nature, Science, PNAS, Cell[maybe], NEJoM[maybe]), and similar metrics. In contrast, scientists talk about impact factors, citations, and journals all the time. A few years ago, my home country started this enabling system for tenure candidates. Who do you think proposed as criteria, for tenure, the year-weighted citations, the impact factor of the journals in which the research was published as a discriminating factor? Faculties. Who now say that the system is, for all intents and purposes, unchangeable.

majormajor 3y ago

I think this in near-inevitable in any org or federation of orgs of people with more than a few hundred people involved (wild-ass-guess, substitute your own). Once you have too much scale for a few people to truly know what's going on, all that's left to keep people working on the right incentives is trust, and inevitably you'll have some bad actors at some point, and after that you put in some metrics and then it gets worse cause even good actors pursuing those metrics may have gone off the rails. The only places I've seen avoid it moderately successfully have had exceptional people managers, like 1-in-100/1000 level, to minimize the appeal of playing the game instead of doing the work.

borroka 3y ago
Every system that was later changed for something better was, at one time, considered inevitable, natural, ineluctable.
Bjartr 3y ago
> too much scale for a few people to truly know what's going on Somewhat related is Dunbar's number https://en.m.wikipedia.org/wiki/Dunbar%27s_number https://en.m.wikipedia.org/wiki/Dunbar%27s_number
brnaftr361 3y ago
I think "bad actors" is a poor framing, this extends well beyond malfeasance. Even in a perfect system, at some scale people need these elementary quantities to be able to model and predict exogenous factors or justify certain other behaviors to operate. Goodhart's law does tend to come into effect - but Goodhart's law may come to dominate these systems through far more than malfeasance. Just phone game-esque decay of information from one actor to the next can cause it, moreover this occurs across generations which I would posit accelerates the decay. One "generation" may have firsthand experience of the why and how, and upon relaying it to the second a loss occurs, and the third may not ever receive it or they're given some highly heuristicised version which artlessly excises nuance. Very quickly the system is reliant on decayed information which informs garbage processes, but this is ossified as the standard operating procedure. I would posit that, especially in the temporal domain, this decay should prompt us to reconsider our tolerance for staple institutions and our willingness to facilitate their development, sustainment, and immortalization and thus limit scale while maintaining the integrity of vision and information while jointly allowing voids to open and be closed by legitimate and ostensibly better suited successors. Right now we're making models of cancer, many of which are metastatic. Nice handle, by the way.

cycomanic 3y ago

I think this is a misrepresentation of reality. Generally there has been strong pressure from government (and the public) on "justifying" their spending through measurable outcomes. Throughout the 90s there were numerous media campaigns about "useless" research and we can still see the same sentiments even here on HN. Yes academics (well typically more the lab leaders etc), we're involved in some of the system that we see now, but it was in reaction to the pressure from politics and society. What do you expect people to do when the places that fund them start to demand measurable outcomes for something that is inherently difficult to measure. You come up with some (often arbitrary) metrics. That whole dynamic then just became an anweful feedback process.

borroka 3y ago
I strongly disagree. The public (let's take the 75% of the educational attainment level folks, excluding PhDs, so non-academic people who are educated enough) knows nothing about research, how research is conducted, what a paper is, how a paper is published. Zero clues. You can imagine what the bottom 75% think. Researchers are considered by the public to be little to no more than glorified teachers. It is like saying that bigger and better weapons are invented and built to justify the Defense budget to the public, where for the public a bomb is a bomb is a bomb. How do I know? (1) I was an academic and I often had low-level conversations about my research, in my home country, in the U.S., in Europe, South America, everywhere. Not a single interlocutor outside of academia knew what is the typical process to follow to have some research results published in a scientific journal, or whether a publication in a certain journal can be indicative of more or less cited/quality research. (2) I read comments here and there on newspapers, forums, social media, etc. Letters to the editor of your local newspaper are considered to be equivalent to a publication in Cell. (3) Ask, for fun, like I used to do, your regular guy if (a) an electron is bigger than an atom, (b) how many people live on earth. Re: research is useless. Most academic science is useless, and the uselessness of 95 percent of research is somehow considered inevitable by the "scientific community." But in that 95 percent there is a portion of research that is speculative and ultimately turns out to be useless, which is a good thing to have, and a portion of research (the majority) that serves only to pay the salaries of the researchers involved, is clear to everyone and their cousin that it will go nowhere, and is published in journals that no one, and rightly so, will ever read.

bsder 3y ago

> For once, neither politicians nor administrators were responsible for a system of "perverse incentives," since these gray figures are not involved in either hiring or tenure decisions. This is incorrect. The government and politicians are definitely some of the biggest factors. The collapse of the longer timescale corporate R&D labs meant that research funding transferred to the government. Before that, if universities got too demanding, the academics would leave to the corporate labs. These factors combined with Bayh-Dole and the lottery nature of "intellectual property" to cause universities to start chasing funding of short-term gains rather than long-term research. (Famous example: Katalin Kariko, the woman behind much of the mRNA groundwork, was denied tenure and had a very difficult time getting funding).

borroka 3y ago
The comment makes little sense, the way that is worded. When and where were politicians responsible for the collapse of the "the longer timescale corporate R&D labs"? Are you thinking of ATT/Bell Labs? Are you thinking of just the U.S. and just the East Coast of the U.S., ignoring a much larger world? "Before that, if universities got too demanding, the academics would leave to the corporate labs." Academics who? Let's say biologists, who are a massive presence in academia, exactly where in the private sector were they working before the aforementioned collapse of "the longer timescale corporate R&D labs"? Pharma is strong, maybe stronger than ever, but most of the top pharma-adjacent biologists prefer to stay in academia and collaborate with Pharma, more than the opposite. You are just thinking about a bunch of labs (Bell labs?) able to move maybe 1% of the academics at most in their heyday. And it seems to me, like I explained, that the "productivity pressure" is coming from academics themselves, not from outside academia. Before the 70's, it was fairly easy for a moderately successful Ph.D. to get a position in academia, but the number of positions available did not keep up with the increase of Ph.D.'s who wanted to do academic research. And the "productivity" measurements started.
- bsder 3y ago
  > When and where were politicians responsible for the collapse of the "the longer timescale corporate R&D labs"? I did not say this and am quite mystified how you managed to construe my words to imply that. The fact that those corporate labs had a longer term focus and then disappeared is simply a fact. > Academics who? Let's say biologists Before PCR (1984ish?), biology had such a vanishingly small presence relative to research funding that it almost doesn't count. Every big company up through about 1970-1980 had very large R&D departments that had been in existence for anywhere between half to a whole century. Sure: IBM, Bell, and Xerox are remembered strongly. However, Kodak and Polaroid had big chemical engineering departments. RCA and Motorola were household names. Every big name in steel (US Steel, Bethlehem, etc.) all had lots of research in metallurgy. Corning was a huge driving force behind ceramic chemistry. Dupont similarly for plastics. When those big companies went down, so, too, did their research departments and the funding driving them. This was a problem because companies funded a LOT of the local universities--Pittsburgh universities, for example, were still using equipment even in the 1990s that had been funded decades earlier by the big steel mills. The collapse of the corporate giants meant that the percentage of academic research funding from government suddenly went way up.

wrp 3y ago

> ...the scientists themselves...have largely contributed to the development of this particular academic system... This is a strong claim, which I think needs some references to support it. Watching my dad's generation in academia from the 1950s forward and my own from the 1980s forward, my impression has been that this change in campus culture has been entirely driven by the administrative level.

ttymck 3y ago
Who do you think the administrators of physical science colleges are? EDIT: I guess you can omit "physical science", I suppose we are talking about research across all disciplines. My point remains.
- wrp 3y ago
  Ultimate decision making is largely by people with a background in business, not science. It seems to be poorly understood even among faculty that administrative policy is ultimately set by the board of trustees/directors. These boards are filled with people who gained prominence through social or business activity. My impression is that successful research scientists rarely if ever land on a board of directors. Although scientists are hired to fill administrative positions, their continued employment depends on their following the directives of the board.
  - borroka 3y ago
    But the context was hiring/tenure/grants assignment decisions and how academic productivity became a major factor in those decisions. Business people have no say in any hiring decisions made at universities. The hiring committee is made up of faculties at the department hiring the new assistant(often)/associate(rarely)/full(very rare) professor. Tenure is not decided by any business or administrative person working at universities. Same for grant funding in the sciences.
    - deleted 3y ago
      [deleted]

nico 3y ago

And this is how you get the case of the president of Stanford, who not only benefited tremendously from all the research that he was supposedly a part of or leading, but when it came to light that a lot of it was fraudulent, he was able to claim he wasn’t really involved that closely So, a lot of the benefit, but barely any of the accountability

borroka 3y ago
That's more of a case of elite's protection and general, outside very special cases, immunity from, broadly speaking, prosecution. If the same had happened to an assistant professor, non-minority, they would have been kicked out long ago.

n4r9 3y ago

Your "leading scientist" anecdote only proves that the system is rigged to reward scrounging, and that scientists therefore feel a pressure to scrounge, even at the top level. Does that mean scientists are to blame for the system? I don't think it shows that. At most it shows that genuine scientists aren't very optimistic about their power to change the system.

borroka 3y ago
Let's take hiring decisions. Who's assessing the number of papers published by candidates, the number of citations those papers received, the impact factors of the journals in which the papers were published? Who's writing the recommendation/reference letters for the candidates? The answer is other scientists, and greater weight is given to the opinion, assessment, letters of "leading scientists". The "system" was set up over time by scientists, for scientists.
- n4r9 3y ago
  Scientists might feel pressure to take impact factors into account when hiring. That doesn't necessarily mean that they want to promote impact factors, or that they think impact factors are good for science. It could for example mean that the university/institute has made their funding and other perks contingent on achieving certain metrics. Blaming scientists for this is like blaming employees for poor working conditions because they signed a contract. It ignores the fact that they had no choice but to sign the contract, if they want a job.
  - borroka 3y ago
    What you propose is a hypothesis. Fine. The hypothesis is not supported by any clear evidence. I cannot think of a public university administrator who is not a scientist/researcher/faculty who is familiar with impact factors, the relative strength of journals outside the top 5 and then maybe (Nature, Science, PNAS, Cell[maybe], NEJoM[maybe]), and similar metrics. In contrast, scientists talk about impact factors, citations, and journals all the time. A few years ago, my home country started this enabling system for tenure candidates. Who do you think proposed as criteria, for tenure, the year-weighted citations, the impact factor of the journals in which the research was published as a discriminating factor? Faculties. Who now say that the system is, for all intents and purposes, unchangeable.