6 ms·

I largely agree, but we do not write all these papers "so tax payers can sleep". We write these papers because the administration incentivizes us to get the hig

by timkam 6y ago

I largely agree, but we do not write all these papers "so tax payers can sleep". We write these papers because the administration incentivizes us to get the highest profile publications with the lowest effort possible. In relatively creative fields (i.e., much of computer science), this can even mean to make friends in the community and co-author papers with them without being the one who does any of the hard work. (In other sciences, it means being the one who pulls in the money, as far as I understand.) A related problem I see is that we treat all papers the same: position (opinion) papers, reviews & surveys, and "hard" results (relevant proofs, strong empirical results, software artifacts that are deployed at scale in practice etc.). As the blog post somewhat suggests, the administration should primarily ask us to summarize these results when assessing whether we are worthy of funding and faculty positions; sure, it takes more effort, but I think it is possible.

nextos 6y ago
Exactly, the big issue is that paper counts have become the dominant metric. And like with all structures dominated by a single metric, it's easy to game the system. Most professors I have worked with in experimental sciences are completely disconnected from actual research. They just optimize for hiring hordes of people to churn out tons of mediocre papers, which they don't care about. I came from theoretical CS & math, so this was really shocking. It's really depressing and it needs to change. Academia is now in a phase similar to that of an innovative startup which has been filled in with many middle managers, and has turned into a corporate monstrosity.
- retrac 6y ago
  "Paper count" is about as reliable a metric as "lines of code". Publish or die is a toxic mentality in academia and you are certainly not the first to have observed this, sadly.
  - xmprt 6y ago
    I'm not sure if those two fields are comparable. In software engineering, at least there's usually a dollar value impact for the code you write. In research, the impact isn't as tangible and probably won't be known for years.
    - throwawaygh 6y ago
      > In research, the impact isn't as tangible and probably won't be known for years. This still describes a lot of software engineering.
  - unishark 6y ago
    It's pretty outdated. In research schools you need to win funding or die nowadays. And while publications do help you, say in getting hired and promotions, your publications are dug into pretty deeply. They don't want to be tricked into hiring someone useless who can crank out fluff but can't win a grant.
    - deleted 6y ago
      [deleted]
    - derbOac 6y ago
      This is completely true -- something that many outside of academics don't understand -- but I'm not sure it really makes the situation any better. In a lot of ways it just kicks the problems down the road. By whatever metric you use, it's really about the checklist rather than substance. And the checklists are based on stereotypes, and get gamed.
      - unishark 6y ago
        > This is completely true -- something that many outside of academics don't understand -- but I'm not sure it really makes the situation any better. Indeed in some ways it's worse. When the money is insufficient, the established researchers get it first. Then everyone else has to increasingly work for established researchers as postdocs and similar positions... helping to yet further boost the advantage of the established researchers. It went from senior researchers versus junior researchers, to senior researchers with a herd of postdocs working for them, versus junior researchers. At least if it was based on publications, everyone gets co-authorship and in fact the first author counts most. So the postdoc and the PI gain the same new items on their CV's for their collaboration. For grants, credit goes entirely to the PI.
  - drran 6y ago
    Maybe, software solutions can be applied to academia. In software, we have "source code distributions", such as Debian, Fedora, Gentoo, Arch, Red Hat, SUSE, etc. They are collections of mostly a same free software, with minor differences, but with different policies and purpose. Some are conservative collections of proven software, others are bleeding edge, etc. IMHO, someone should start to collect papers, which are worth reading, and keep this collection up to date, with fixes to papers (AKA as patches in software), with one paper per topic, with ready to use math models and simulations, with in software tests for models, etc.
    - amself 6y ago
      That sounds like Wikipedia!
      - drran 6y ago
        Wikiscience, maybe?
    - JPLeRouzic 6y ago
      It's a nice idea but there are practical problems. The first is that it supposes the scientific publication is a rational and concerted process. It is not, here is an example: I am interested in Amyotrophic Lateral Disease and I blog about research done on ALS [0]. However there are 15,000 scientific papers published each year on this topic, on average only 20 to 30 articles per year are worth spending the time to read them (one every 500) and a milestone appears only every 3/4 years (TDP-43 in 2006, C9orf72 in 2011, etc). After 100 years there is still no clue about ALS etiology. Scientists have strong but widely different opinions such as for some it starts in the brain motor area, for other it starts at the peripheral nerve junction with muscles, for other (new trend) it starts in the gut. And scientists happily pursue hypothesis that their colleagues are supposed to have disproved. Sometimes the same lab (I have one in mind which is perhaps the best known in ALS research) publishes articles contradicting its previous research without discussing the issue. How to rationally blog about this domain, how to make collections that are worth reading? [0] https://padiracinnovation.org/News/tag/als https://padiracinnovation.org/News/tag/als
- cbozeman 6y ago
  Its TPS reports. That what it is. And until academia gets their own Bobs to ask academia's Lumbergh, "Yeah, Academia, lemme ask yah... real quick question here... how much time wouldja say yah spend each week dealing with these papers?"
- andi999 6y ago
  Maybe you didn't mean it, but 'gaming the system' sounds bad. While I agree the outcome is bad, I think when your boss tells you to publish 10 paper per year and you do that, that this cannot be called 'gaming the system'
- jhoechtl 6y ago
  > It's really depressing and it needs to change. It has been like that since ten years and nothing has changed since then. People find out just about the same as you did but nothing changes. It such a convenient metric it will stay for much longer.
anonymousDan 6y ago
Regarding your second paragraph, I don't think this is the case in the UK at least, where academics are judged according to the REF criteria.
unishark 6y ago
Reviews are especially bad. You can tell which people work at schools where citations count in their pay (and reviews are allowed in citation counts) by the ratio of reviews to original research papers they write. Some schools/govts really need to wise up. I think of it as a kind of predatory or scavenger behavior. They basically steal the citations from the papers they are reviewing in return for a small effort in helping us with information retrieval. We might as well just cite google and elsevier. Ideally, citations should pass through them somehow and get applied to the original sources who did the hard work.
- gradstudent 6y ago
  It's true that some review papers are shallow but it's pretty cynical to paint all contributions of this type with the same brush. In my experience, good review papers are more than summaries. They can connect separate lines of research for example, compare and contrast different ideas and they can re-contextaulise a body of research to give a broader picture and reveal new and interesting directions. These papers are hard to write well and take a long time. Even more so in cases where empirical work has been undertaken so as to make direct comparisons and provide reference implementations. Even the summary papers can be useful. On particularly active problems there can be dozens of papers per year. Sorting out wheat from chaff by highlighting notable works, and pointing out trends in published research, is again a genuine contribution, and helpful for the scientific community.
  - unishark 6y ago
    Of course they are useful, as is google scholar and the search field in various journals. Or some blog that tells you whats was new in the recent conference. That doesn't mean they deserve to be cited over the real sources. They are cited as a crutch: "go here for a more detailed list of sources". The little meta-analysis they do doesn't explain most of the citations they get.
    - bonoboTP 6y ago
      Hard disagree. We need to incentivized more curation, distillation, summarization, comparing/contrasting, systematizing, categorizing approaches along various axes, meta analyses etc., not less. These are often very valuable, more so than yet another 1% improvement paper that never gets reproduced. Review papers don't steal citations. These are cited from more distant literature so readers can familiarize themselves with the topic. Those wouldn't cite all the papers mentioned in the review individually. But as a broader point, yeah, maybe something like PageRank could be used to "pass on" the citation Ina sense.
quantified 6y ago
I’ve heard the term “minimum knowledge increment” used as a measuring unit for papers. Dice whatever research you have into papers that contribute the minimum necessary for publication. Dicing may only produce 1 paper, good enough yet meager.

by timkam 6y ago

nextos 6y ago

Exactly, the big issue is that paper counts have become the dominant metric. And like with all structures dominated by a single metric, it's easy to game the system. Most professors I have worked with in experimental sciences are completely disconnected from actual research. They just optimize for hiring hordes of people to churn out tons of mediocre papers, which they don't care about. I came from theoretical CS & math, so this was really shocking. It's really depressing and it needs to change. Academia is now in a phase similar to that of an innovative startup which has been filled in with many middle managers, and has turned into a corporate monstrosity.

retrac 6y ago
"Paper count" is about as reliable a metric as "lines of code". Publish or die is a toxic mentality in academia and you are certainly not the first to have observed this, sadly.
- xmprt 6y ago
  I'm not sure if those two fields are comparable. In software engineering, at least there's usually a dollar value impact for the code you write. In research, the impact isn't as tangible and probably won't be known for years.
  - throwawaygh 6y ago
    > In research, the impact isn't as tangible and probably won't be known for years. This still describes a lot of software engineering.
- unishark 6y ago
  It's pretty outdated. In research schools you need to win funding or die nowadays. And while publications do help you, say in getting hired and promotions, your publications are dug into pretty deeply. They don't want to be tricked into hiring someone useless who can crank out fluff but can't win a grant.
  - deleted 6y ago
    [deleted]
  - derbOac 6y ago
    This is completely true -- something that many outside of academics don't understand -- but I'm not sure it really makes the situation any better. In a lot of ways it just kicks the problems down the road. By whatever metric you use, it's really about the checklist rather than substance. And the checklists are based on stereotypes, and get gamed.
    - unishark 6y ago
      > This is completely true -- something that many outside of academics don't understand -- but I'm not sure it really makes the situation any better. Indeed in some ways it's worse. When the money is insufficient, the established researchers get it first. Then everyone else has to increasingly work for established researchers as postdocs and similar positions... helping to yet further boost the advantage of the established researchers. It went from senior researchers versus junior researchers, to senior researchers with a herd of postdocs working for them, versus junior researchers. At least if it was based on publications, everyone gets co-authorship and in fact the first author counts most. So the postdoc and the PI gain the same new items on their CV's for their collaboration. For grants, credit goes entirely to the PI.
- drran 6y ago
  Maybe, software solutions can be applied to academia. In software, we have "source code distributions", such as Debian, Fedora, Gentoo, Arch, Red Hat, SUSE, etc. They are collections of mostly a same free software, with minor differences, but with different policies and purpose. Some are conservative collections of proven software, others are bleeding edge, etc. IMHO, someone should start to collect papers, which are worth reading, and keep this collection up to date, with fixes to papers (AKA as patches in software), with one paper per topic, with ready to use math models and simulations, with in software tests for models, etc.
  - amself 6y ago
    That sounds like Wikipedia!
    - drran 6y ago
      Wikiscience, maybe?
  - JPLeRouzic 6y ago
    It's a nice idea but there are practical problems. The first is that it supposes the scientific publication is a rational and concerted process. It is not, here is an example: I am interested in Amyotrophic Lateral Disease and I blog about research done on ALS [0]. However there are 15,000 scientific papers published each year on this topic, on average only 20 to 30 articles per year are worth spending the time to read them (one every 500) and a milestone appears only every 3/4 years (TDP-43 in 2006, C9orf72 in 2011, etc). After 100 years there is still no clue about ALS etiology. Scientists have strong but widely different opinions such as for some it starts in the brain motor area, for other it starts at the peripheral nerve junction with muscles, for other (new trend) it starts in the gut. And scientists happily pursue hypothesis that their colleagues are supposed to have disproved. Sometimes the same lab (I have one in mind which is perhaps the best known in ALS research) publishes articles contradicting its previous research without discussing the issue. How to rationally blog about this domain, how to make collections that are worth reading? [0] https://padiracinnovation.org/News/tag/als https://padiracinnovation.org/News/tag/als
cbozeman 6y ago
Its TPS reports. That what it is. And until academia gets their own Bobs to ask academia's Lumbergh, "Yeah, Academia, lemme ask yah... real quick question here... how much time wouldja say yah spend each week dealing with these papers?"
andi999 6y ago
Maybe you didn't mean it, but 'gaming the system' sounds bad. While I agree the outcome is bad, I think when your boss tells you to publish 10 paper per year and you do that, that this cannot be called 'gaming the system'
jhoechtl 6y ago
> It's really depressing and it needs to change. It has been like that since ten years and nothing has changed since then. People find out just about the same as you did but nothing changes. It such a convenient metric it will stay for much longer.

anonymousDan 6y ago

Regarding your second paragraph, I don't think this is the case in the UK at least, where academics are judged according to the REF criteria.

unishark 6y ago

Reviews are especially bad. You can tell which people work at schools where citations count in their pay (and reviews are allowed in citation counts) by the ratio of reviews to original research papers they write. Some schools/govts really need to wise up. I think of it as a kind of predatory or scavenger behavior. They basically steal the citations from the papers they are reviewing in return for a small effort in helping us with information retrieval. We might as well just cite google and elsevier. Ideally, citations should pass through them somehow and get applied to the original sources who did the hard work.

gradstudent 6y ago
It's true that some review papers are shallow but it's pretty cynical to paint all contributions of this type with the same brush. In my experience, good review papers are more than summaries. They can connect separate lines of research for example, compare and contrast different ideas and they can re-contextaulise a body of research to give a broader picture and reveal new and interesting directions. These papers are hard to write well and take a long time. Even more so in cases where empirical work has been undertaken so as to make direct comparisons and provide reference implementations. Even the summary papers can be useful. On particularly active problems there can be dozens of papers per year. Sorting out wheat from chaff by highlighting notable works, and pointing out trends in published research, is again a genuine contribution, and helpful for the scientific community.
- unishark 6y ago
  Of course they are useful, as is google scholar and the search field in various journals. Or some blog that tells you whats was new in the recent conference. That doesn't mean they deserve to be cited over the real sources. They are cited as a crutch: "go here for a more detailed list of sources". The little meta-analysis they do doesn't explain most of the citations they get.
  - bonoboTP 6y ago
    Hard disagree. We need to incentivized more curation, distillation, summarization, comparing/contrasting, systematizing, categorizing approaches along various axes, meta analyses etc., not less. These are often very valuable, more so than yet another 1% improvement paper that never gets reproduced. Review papers don't steal citations. These are cited from more distant literature so readers can familiarize themselves with the topic. Those wouldn't cite all the papers mentioned in the review individually. But as a broader point, yeah, maybe something like PageRank could be used to "pass on" the citation Ina sense.

quantified 6y ago

I’ve heard the term “minimum knowledge increment” used as a measuring unit for papers. Dice whatever research you have into papers that contribute the minimum necessary for publication. Dicing may only produce 1 paper, good enough yet meager.