5 ms·

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in

Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.

isaacfrond 4y ago
[citation needed]
- ackbar03 4y ago
  I suggest the author cite the following works which I believe are related to this topic: list of own publications
- bildung 4y ago
  My background is in humanities but the dynamics are the same: Chasing the goal of objectivity, easily measurable datapoints like number of publications and number of citations are used for quantifying impact of a researcher. This incentivizes researching/publishing low hanging fruit, because actually working on deep or unusual approaches means sinking more time into the endevour while risking less or no publishable results. For people trying to earn a living in academic institutions this would be stupid.
- CrazyStat 4y ago
  I can confirm the GP matches my experience in academia (PhD, published a paper based on my dissertation, left academia shortly after). There is immense pressure for early career researchers to work on low-hanging fruit that will give easy guaranteed publications: make a minor change to previous work and write a paper about it; write a paper about what other people wrote papers about (either a review or a meta-analysis); etc. etc. This is how you rack up publications for your tenure review. It's also how you rack up publications to get funding. Once you have tenure you have a bit more freedom, but you still need to worry about funding (generally). And by that time you've been in the habit of grabbing the lowest-hanging fruit for 1-1.5 decades, assuming you were ~well trained~ in graduate school. After so long reinforcing that habit it becomes hard and scary to build a ladder and go after the better, higher fruit.
  - rscho 4y ago
    Habit is not the problem. Once you get tenure, a good part of your job is to ensure that the younger generation in your lab/uni publishes well, i.e encourage them to pursue the next low-hanging fruit... So no, you do not in fact have more freedom.
    - CrazyStat 4y ago
      That's a fair point. I jumped ship before I got that far, obviously. From my perspective as a student it looked a lot like habit on the part of the senior/tenured people in the department, but I should not have been so quick to assert that.
rscho 4y ago
As a clinical researcher, I wholeheartedly agree. Even worse, meta analysis is comparatively so well cited (even shitty ones) that doing anything else, such as original research, will be suboptimal since the reduced time investment allows you to publish nonstop. Truly, meta analysis has become a blood-ducking parasite of medical science. I know many young researchers who've specialized in meta analysis for that reason.
- nerdponx 4y ago
  Isn't there tremendous real value in meta-analysis? The problem seems to lie in how credit is allocated (via citations).
  - n4r9 4y ago
    I'd argue the problem is a combination of citations and short-termism. We don't give researchers the space to do something novel, because institutions demand results in the short term. If you give someone a tangible, quantifiable demand A and a nebulous, low-priority demand B, they will spend nearly 100% of their time chasing A. Truly novel results can take months or years of hard focus; every grant proposal, teaching duty, and routine publication is a distraction.
  - einpoklum 4y ago
    > Isn't there tremendous real value in meta-analysis? There's also tremendous value in doing the dishes and mopping the floor. Without it we'd get diseases and live in filth. And when we do this, we should be respected and appreciated. But - not as brilliant trail-blazers.
ThereIsNoWorry 4y ago
That was my conclusion after publishing a paper during my BSc. Academia is full of petty politics. Professors that gatekeep whole fields on a "like"-basis, reviewers arguing with "not enough experiments done" and questioning uniqueness and novelty in ways that just don't make any sense (to me). It is a very bureaucratic process that cannot possibly be good for innovation and progress. Then you get incentives of fields that don't make any progress to keep the hype train going so their jobs don't become meaningless over night. It just keeps going. So, I obviously declined to do a PhD and went into industry. At the bare minimum, playing politics in industry leads to vastly better (pay) outcomes than in academia.
- godelski 4y ago
  Dog whistles in academic reviewing; phrases that can be valid but are typically being used in an invalid way: - Not novel: Novelty is ill-defined. Most journals and conferences have guidelines on this that come down to "will someone find it useful" which should make "not novel" a rare review, not the most common. Additionally, what's obvious post hoc is not obvious a priori (I actually saw an AC override reviewers because of this!). If something is well written it also comes across as easy and simple. Lack of novelty could be a true lack of novelty or just an indication of a well written paper. - Incremental: All research is incremental. Research can be too incremental (not enough work done) but this is never expanded upon in those types of reviews. This kind of statement is meaningless in isolation. Incrementalism will always exist with a publish or perish paradigm. One cannot publish breakthroughs year over year. Those take significant periods of time or a lucky break. Neither is a common occurrence, by definition. - Not enough experiments: "Money is all you need" has become a popular phrase. There may not be enough experiments and this can be true but perfection is the enemy of good. Especially in research, where the experiment space is non-exhaustive. Can a reviewer be satisfied in a finite amount of time and with finite resources? I've started to change my mind about where blame falls for all this. It starts with the reviewers, but if this becomes the norm then the system is at fault. It has not removed the rotten apples and thus let the barrel spoil. All while we've had decades of discussion about this happening. I think a lot falls on the Area Chairs and Metareviewers, as they should be preventing these types of reviews to pass. But the whole incentive structure of publishing is wrong. There is high pressure to reject and zero pressure to let papers through. This especially hurts our junior researchers (grad students) as it becomes a lottery process for determining if they can graduate. After all, we can't have any wizards without any noobs.
Frost1x 4y ago
My anecdata from the US agrees. The best part is when funding agencies require proposal reviewers to actually send their critiques back on research proposals. I've read through many reviewers replies to proposals designed for specifically set risky/novel research funding and the responses are sometimes so conservatively comical without a truly valid argument you can't help but laugh. I've also worked at research organizations specifically touted as being highly independent with freedom to explore novel paths and the story is basically the same. Ultimately, someone somewhere holds the purse strings and conservatism kicks in. You at minimum need to spin your research to fit popular trend keyword language or make it apply to these areas. Financial structures ultimately dictate the lack of support for novelty. Basically, novel research has freedom only when it's independently funded, which limits the scale of most novel research. No one wants to take risk, they want to market taking high risk while taking low risk and selling mediocrity. I think the fundamental underlying issue we have across multiple societies is that we no longer proportionately reward risk anymore and that's why people seek low risk everywhere. Taking high risk often has limited opportunity for reward, even if you are successful, so why bother? You'll net more success taking low risks and only fools take high risk.
- api 4y ago
  I've thought for a long time that this is why you see so much innovation during wartime. War overrides conservatism and gets risky things funded. The same can apply to war-like scenarios. A lot of novel biological research got funded during COVID. We should expect to see a burst in advancement in biomedical technology as it bears fruit over the next 5-10 years. The core problem is probably that conservatism makes the most sense at the individual level for most people. Even if you had a casino whose net payout was positive, it still may not make sense individually to gamble. If individual rewards are sporadic and concentrated on a power law distribution then statistically most players will lose. An obvious solution is a less power-law type distribution, which tends toward some kind of socialism, but that has its own problems. Nobody's figured out yet how to provide a productive channel in a socialist system for humans' totally normal drives to compete and maximize individual outcome. Trying to suppress it doesn't work; it's like trying to totally suppress sexuality. Extreme socialist societies (e.g. Soviet Communism) often evolve into totalitarian mafia states because if there is no positive outlet for those urges they drive people into crime. Eventually you get a meritocracy concentrated in crime and the mob takes over everything.
a_bonobo 4y ago
>By then, you've probably lost most of your imagination anyway. I've recently left academia after ~10 years to work on more blue-sky-stuff in industry, and this rings so true. It took me a few months to unlearn academia's focus on the minimal project we can do to get the next paper out (still unlearning!). It's hard to get in, it's harder to get out.
- anonymouskimmer 4y ago
  I worked as a permatemp for almost 7 years in industry and didn't have the ability to order my own reagents. It took 3 or 4 years of working in a DOE lab to break that habit and start ordering my own reagents to innovate with. Ingrained work habits are hard to break.
teekert 4y ago
I heard established profs talk about how they ask for funding for something already done or almost successfully done. Then they do something new with the money.
asdff 4y ago
Certain fields are a lot easier to do novel work in. For a grad student in computer science/computational biology, you practically don't need any grant funding at all. You can download public data, work on the university hpc for basically free, publish whatever the hell you want, and pay your stipend with a TAship. Others, where you have to regularly buy reagents or maintain an animal model, you pretty much need grant support to buy these things.

isaacfrond 4y ago

[citation needed]

ackbar03 4y ago
I suggest the author cite the following works which I believe are related to this topic: list of own publications
bildung 4y ago
My background is in humanities but the dynamics are the same: Chasing the goal of objectivity, easily measurable datapoints like number of publications and number of citations are used for quantifying impact of a researcher. This incentivizes researching/publishing low hanging fruit, because actually working on deep or unusual approaches means sinking more time into the endevour while risking less or no publishable results. For people trying to earn a living in academic institutions this would be stupid.
CrazyStat 4y ago
I can confirm the GP matches my experience in academia (PhD, published a paper based on my dissertation, left academia shortly after). There is immense pressure for early career researchers to work on low-hanging fruit that will give easy guaranteed publications: make a minor change to previous work and write a paper about it; write a paper about what other people wrote papers about (either a review or a meta-analysis); etc. etc. This is how you rack up publications for your tenure review. It's also how you rack up publications to get funding. Once you have tenure you have a bit more freedom, but you still need to worry about funding (generally). And by that time you've been in the habit of grabbing the lowest-hanging fruit for 1-1.5 decades, assuming you were ~well trained~ in graduate school. After so long reinforcing that habit it becomes hard and scary to build a ladder and go after the better, higher fruit.
- rscho 4y ago
  Habit is not the problem. Once you get tenure, a good part of your job is to ensure that the younger generation in your lab/uni publishes well, i.e encourage them to pursue the next low-hanging fruit... So no, you do not in fact have more freedom.
  - CrazyStat 4y ago
    That's a fair point. I jumped ship before I got that far, obviously. From my perspective as a student it looked a lot like habit on the part of the senior/tenured people in the department, but I should not have been so quick to assert that.

rscho 4y ago

As a clinical researcher, I wholeheartedly agree. Even worse, meta analysis is comparatively so well cited (even shitty ones) that doing anything else, such as original research, will be suboptimal since the reduced time investment allows you to publish nonstop. Truly, meta analysis has become a blood-ducking parasite of medical science. I know many young researchers who've specialized in meta analysis for that reason.

nerdponx 4y ago
Isn't there tremendous real value in meta-analysis? The problem seems to lie in how credit is allocated (via citations).
- n4r9 4y ago
  I'd argue the problem is a combination of citations and short-termism. We don't give researchers the space to do something novel, because institutions demand results in the short term. If you give someone a tangible, quantifiable demand A and a nebulous, low-priority demand B, they will spend nearly 100% of their time chasing A. Truly novel results can take months or years of hard focus; every grant proposal, teaching duty, and routine publication is a distraction.
- einpoklum 4y ago
  > Isn't there tremendous real value in meta-analysis? There's also tremendous value in doing the dishes and mopping the floor. Without it we'd get diseases and live in filth. And when we do this, we should be respected and appreciated. But - not as brilliant trail-blazers.

ThereIsNoWorry 4y ago

That was my conclusion after publishing a paper during my BSc. Academia is full of petty politics. Professors that gatekeep whole fields on a "like"-basis, reviewers arguing with "not enough experiments done" and questioning uniqueness and novelty in ways that just don't make any sense (to me). It is a very bureaucratic process that cannot possibly be good for innovation and progress. Then you get incentives of fields that don't make any progress to keep the hype train going so their jobs don't become meaningless over night. It just keeps going. So, I obviously declined to do a PhD and went into industry. At the bare minimum, playing politics in industry leads to vastly better (pay) outcomes than in academia.

godelski 4y ago
Dog whistles in academic reviewing; phrases that can be valid but are typically being used in an invalid way: - Not novel: Novelty is ill-defined. Most journals and conferences have guidelines on this that come down to "will someone find it useful" which should make "not novel" a rare review, not the most common. Additionally, what's obvious post hoc is not obvious a priori (I actually saw an AC override reviewers because of this!). If something is well written it also comes across as easy and simple. Lack of novelty could be a true lack of novelty or just an indication of a well written paper. - Incremental: All research is incremental. Research can be too incremental (not enough work done) but this is never expanded upon in those types of reviews. This kind of statement is meaningless in isolation. Incrementalism will always exist with a publish or perish paradigm. One cannot publish breakthroughs year over year. Those take significant periods of time or a lucky break. Neither is a common occurrence, by definition. - Not enough experiments: "Money is all you need" has become a popular phrase. There may not be enough experiments and this can be true but perfection is the enemy of good. Especially in research, where the experiment space is non-exhaustive. Can a reviewer be satisfied in a finite amount of time and with finite resources? I've started to change my mind about where blame falls for all this. It starts with the reviewers, but if this becomes the norm then the system is at fault. It has not removed the rotten apples and thus let the barrel spoil. All while we've had decades of discussion about this happening. I think a lot falls on the Area Chairs and Metareviewers, as they should be preventing these types of reviews to pass. But the whole incentive structure of publishing is wrong. There is high pressure to reject and zero pressure to let papers through. This especially hurts our junior researchers (grad students) as it becomes a lottery process for determining if they can graduate. After all, we can't have any wizards without any noobs.

Frost1x 4y ago

My anecdata from the US agrees. The best part is when funding agencies require proposal reviewers to actually send their critiques back on research proposals. I've read through many reviewers replies to proposals designed for specifically set risky/novel research funding and the responses are sometimes so conservatively comical without a truly valid argument you can't help but laugh. I've also worked at research organizations specifically touted as being highly independent with freedom to explore novel paths and the story is basically the same. Ultimately, someone somewhere holds the purse strings and conservatism kicks in. You at minimum need to spin your research to fit popular trend keyword language or make it apply to these areas. Financial structures ultimately dictate the lack of support for novelty. Basically, novel research has freedom only when it's independently funded, which limits the scale of most novel research. No one wants to take risk, they want to market taking high risk while taking low risk and selling mediocrity. I think the fundamental underlying issue we have across multiple societies is that we no longer proportionately reward risk anymore and that's why people seek low risk everywhere. Taking high risk often has limited opportunity for reward, even if you are successful, so why bother? You'll net more success taking low risks and only fools take high risk.

api 4y ago
I've thought for a long time that this is why you see so much innovation during wartime. War overrides conservatism and gets risky things funded. The same can apply to war-like scenarios. A lot of novel biological research got funded during COVID. We should expect to see a burst in advancement in biomedical technology as it bears fruit over the next 5-10 years. The core problem is probably that conservatism makes the most sense at the individual level for most people. Even if you had a casino whose net payout was positive, it still may not make sense individually to gamble. If individual rewards are sporadic and concentrated on a power law distribution then statistically most players will lose. An obvious solution is a less power-law type distribution, which tends toward some kind of socialism, but that has its own problems. Nobody's figured out yet how to provide a productive channel in a socialist system for humans' totally normal drives to compete and maximize individual outcome. Trying to suppress it doesn't work; it's like trying to totally suppress sexuality. Extreme socialist societies (e.g. Soviet Communism) often evolve into totalitarian mafia states because if there is no positive outlet for those urges they drive people into crime. Eventually you get a meritocracy concentrated in crime and the mob takes over everything.

a_bonobo 4y ago

>By then, you've probably lost most of your imagination anyway. I've recently left academia after ~10 years to work on more blue-sky-stuff in industry, and this rings so true. It took me a few months to unlearn academia's focus on the minimal project we can do to get the next paper out (still unlearning!). It's hard to get in, it's harder to get out.

anonymouskimmer 4y ago
I worked as a permatemp for almost 7 years in industry and didn't have the ability to order my own reagents. It took 3 or 4 years of working in a DOE lab to break that habit and start ordering my own reagents to innovate with. Ingrained work habits are hard to break.

teekert 4y ago

I heard established profs talk about how they ask for funding for something already done or almost successfully done. Then they do something new with the money.

asdff 4y ago

Certain fields are a lot easier to do novel work in. For a grad student in computer science/computational biology, you practically don't need any grant funding at all. You can download public data, work on the university hpc for basically free, publish whatever the hell you want, and pay your stipend with a TAship. Others, where you have to regularly buy reagents or maintain an animal model, you pretty much need grant support to buy these things.