4 ms·

Exactly, the big issue is that paper counts have become the dominant metric. And like with all structures dominated by a single metric, it's easy to game the sy

by nextos 6y ago

Exactly, the big issue is that paper counts have become the dominant metric. And like with all structures dominated by a single metric, it's easy to game the system. Most professors I have worked with in experimental sciences are completely disconnected from actual research. They just optimize for hiring hordes of people to churn out tons of mediocre papers, which they don't care about. I came from theoretical CS & math, so this was really shocking. It's really depressing and it needs to change. Academia is now in a phase similar to that of an innovative startup which has been filled in with many middle managers, and has turned into a corporate monstrosity.

retrac 6y ago
"Paper count" is about as reliable a metric as "lines of code". Publish or die is a toxic mentality in academia and you are certainly not the first to have observed this, sadly.
- xmprt 6y ago
  I'm not sure if those two fields are comparable. In software engineering, at least there's usually a dollar value impact for the code you write. In research, the impact isn't as tangible and probably won't be known for years.
  - throwawaygh 6y ago
    > In research, the impact isn't as tangible and probably won't be known for years. This still describes a lot of software engineering.
- unishark 6y ago
  It's pretty outdated. In research schools you need to win funding or die nowadays. And while publications do help you, say in getting hired and promotions, your publications are dug into pretty deeply. They don't want to be tricked into hiring someone useless who can crank out fluff but can't win a grant.
  - deleted 6y ago
    [deleted]
  - derbOac 6y ago
    This is completely true -- something that many outside of academics don't understand -- but I'm not sure it really makes the situation any better. In a lot of ways it just kicks the problems down the road. By whatever metric you use, it's really about the checklist rather than substance. And the checklists are based on stereotypes, and get gamed.
    - unishark 6y ago
      > This is completely true -- something that many outside of academics don't understand -- but I'm not sure it really makes the situation any better. Indeed in some ways it's worse. When the money is insufficient, the established researchers get it first. Then everyone else has to increasingly work for established researchers as postdocs and similar positions... helping to yet further boost the advantage of the established researchers. It went from senior researchers versus junior researchers, to senior researchers with a herd of postdocs working for them, versus junior researchers. At least if it was based on publications, everyone gets co-authorship and in fact the first author counts most. So the postdoc and the PI gain the same new items on their CV's for their collaboration. For grants, credit goes entirely to the PI.
- drran 6y ago
  Maybe, software solutions can be applied to academia. In software, we have "source code distributions", such as Debian, Fedora, Gentoo, Arch, Red Hat, SUSE, etc. They are collections of mostly a same free software, with minor differences, but with different policies and purpose. Some are conservative collections of proven software, others are bleeding edge, etc. IMHO, someone should start to collect papers, which are worth reading, and keep this collection up to date, with fixes to papers (AKA as patches in software), with one paper per topic, with ready to use math models and simulations, with in software tests for models, etc.
  - amself 6y ago
    That sounds like Wikipedia!
    - drran 6y ago
      Wikiscience, maybe?
  - JPLeRouzic 6y ago
    It's a nice idea but there are practical problems. The first is that it supposes the scientific publication is a rational and concerted process. It is not, here is an example: I am interested in Amyotrophic Lateral Disease and I blog about research done on ALS [0]. However there are 15,000 scientific papers published each year on this topic, on average only 20 to 30 articles per year are worth spending the time to read them (one every 500) and a milestone appears only every 3/4 years (TDP-43 in 2006, C9orf72 in 2011, etc). After 100 years there is still no clue about ALS etiology. Scientists have strong but widely different opinions such as for some it starts in the brain motor area, for other it starts at the peripheral nerve junction with muscles, for other (new trend) it starts in the gut. And scientists happily pursue hypothesis that their colleagues are supposed to have disproved. Sometimes the same lab (I have one in mind which is perhaps the best known in ALS research) publishes articles contradicting its previous research without discussing the issue. How to rationally blog about this domain, how to make collections that are worth reading? [0] https://padiracinnovation.org/News/tag/als https://padiracinnovation.org/News/tag/als
cbozeman 6y ago
Its TPS reports. That what it is. And until academia gets their own Bobs to ask academia's Lumbergh, "Yeah, Academia, lemme ask yah... real quick question here... how much time wouldja say yah spend each week dealing with these papers?"
andi999 6y ago
Maybe you didn't mean it, but 'gaming the system' sounds bad. While I agree the outcome is bad, I think when your boss tells you to publish 10 paper per year and you do that, that this cannot be called 'gaming the system'
jhoechtl 6y ago
> It's really depressing and it needs to change. It has been like that since ten years and nothing has changed since then. People find out just about the same as you did but nothing changes. It such a convenient metric it will stay for much longer.

by nextos 6y ago

retrac 6y ago

"Paper count" is about as reliable a metric as "lines of code". Publish or die is a toxic mentality in academia and you are certainly not the first to have observed this, sadly.

xmprt 6y ago
I'm not sure if those two fields are comparable. In software engineering, at least there's usually a dollar value impact for the code you write. In research, the impact isn't as tangible and probably won't be known for years.
- throwawaygh 6y ago
  > In research, the impact isn't as tangible and probably won't be known for years. This still describes a lot of software engineering.
unishark 6y ago
It's pretty outdated. In research schools you need to win funding or die nowadays. And while publications do help you, say in getting hired and promotions, your publications are dug into pretty deeply. They don't want to be tricked into hiring someone useless who can crank out fluff but can't win a grant.
- deleted 6y ago
  [deleted]
- derbOac 6y ago
  This is completely true -- something that many outside of academics don't understand -- but I'm not sure it really makes the situation any better. In a lot of ways it just kicks the problems down the road. By whatever metric you use, it's really about the checklist rather than substance. And the checklists are based on stereotypes, and get gamed.
  - unishark 6y ago
    > This is completely true -- something that many outside of academics don't understand -- but I'm not sure it really makes the situation any better. Indeed in some ways it's worse. When the money is insufficient, the established researchers get it first. Then everyone else has to increasingly work for established researchers as postdocs and similar positions... helping to yet further boost the advantage of the established researchers. It went from senior researchers versus junior researchers, to senior researchers with a herd of postdocs working for them, versus junior researchers. At least if it was based on publications, everyone gets co-authorship and in fact the first author counts most. So the postdoc and the PI gain the same new items on their CV's for their collaboration. For grants, credit goes entirely to the PI.
drran 6y ago
Maybe, software solutions can be applied to academia. In software, we have "source code distributions", such as Debian, Fedora, Gentoo, Arch, Red Hat, SUSE, etc. They are collections of mostly a same free software, with minor differences, but with different policies and purpose. Some are conservative collections of proven software, others are bleeding edge, etc. IMHO, someone should start to collect papers, which are worth reading, and keep this collection up to date, with fixes to papers (AKA as patches in software), with one paper per topic, with ready to use math models and simulations, with in software tests for models, etc.
- amself 6y ago
  That sounds like Wikipedia!
  - drran 6y ago
    Wikiscience, maybe?
- JPLeRouzic 6y ago
  It's a nice idea but there are practical problems. The first is that it supposes the scientific publication is a rational and concerted process. It is not, here is an example: I am interested in Amyotrophic Lateral Disease and I blog about research done on ALS [0]. However there are 15,000 scientific papers published each year on this topic, on average only 20 to 30 articles per year are worth spending the time to read them (one every 500) and a milestone appears only every 3/4 years (TDP-43 in 2006, C9orf72 in 2011, etc). After 100 years there is still no clue about ALS etiology. Scientists have strong but widely different opinions such as for some it starts in the brain motor area, for other it starts at the peripheral nerve junction with muscles, for other (new trend) it starts in the gut. And scientists happily pursue hypothesis that their colleagues are supposed to have disproved. Sometimes the same lab (I have one in mind which is perhaps the best known in ALS research) publishes articles contradicting its previous research without discussing the issue. How to rationally blog about this domain, how to make collections that are worth reading? [0] https://padiracinnovation.org/News/tag/als https://padiracinnovation.org/News/tag/als

cbozeman 6y ago

Its TPS reports. That what it is. And until academia gets their own Bobs to ask academia's Lumbergh, "Yeah, Academia, lemme ask yah... real quick question here... how much time wouldja say yah spend each week dealing with these papers?"

andi999 6y ago

Maybe you didn't mean it, but 'gaming the system' sounds bad. While I agree the outcome is bad, I think when your boss tells you to publish 10 paper per year and you do that, that this cannot be called 'gaming the system'

jhoechtl 6y ago

> It's really depressing and it needs to change. It has been like that since ten years and nothing has changed since then. People find out just about the same as you did but nothing changes. It such a convenient metric it will stay for much longer.