4 ms·

I used to work in one of the top fMRI labs. The comments about The Science Game are spot on. The vast majority of scientists, by the time they're midway through

I used to work in one of the top fMRI labs. The comments about The Science Game are spot on. The vast majority of scientists, by the time they're midway through their postdocs, just want to do whatever research will get them a professorship and some grants. Actual knowledge generation is a distant second to keeping the ball bouncing by getting out papers and getting in grants.

ProjectArcturis 5y ago
Just want to add, it's generally not a conscious choice. Everyone wants to do something that is both a genius experiment and gets published in Nature. The Science Game gets propagated in how scientists conceive and plan their work. Everything must be done in the paradigm of publications and grants. You can't do anything if it doesn't have at least interim results within a year or two. It's an unfathomable career risk to try something absolutely brand new. There's a good chance it fails, at which point you have a 2 year blank spot in your CV and you can never get back in the top 10% of scientists where you need to be to get R01 funding and run your own lab.
- mirker 5y ago
  One problem is the incentives are biased toward “ok” papers. h-index should be replaced with a more competitive index (e.g., Max over citations). The publication hyper-competitiveness is due to publication spam, which boosts citation and publication count. However, twisting a knob and getting slightly different results should be viewed as a failed project.
  - ModernMech 5y ago
    > The publication hyper-competitiveness is due to publication spam, which boosts citation and publication count. The publication hyper-competitiveness is a direct result of grant hyper-competitiveness. Funding rates at top US agencies are around 10-20%. Grants are how researchers fund their research, pay their students, pay themselves, and earn tenure, so there's a lot riding on getting them. I see a lot of talk about the problems with publications on HN and possible fixes to the publication process (like requiring code for example, or as you suggest changing the important metrics), but publications really are just a proxy (and prerequisite) for the grant application process. You can change around the publication process all you want, but nothing will improve unless you address the issue of grants, because that's where the money is.
    - ProjectArcturis 5y ago
      But the question is, can you make grants more dependent on actual scientific breakthroughs rather than publication spam?
      - mirker 5y ago
        Right. Listing 10 papers with “ok” impact should be viewed as 10 “bad” papers and not 10 “good” papers. It’s extremely unlikely that a 50 page CV is demonstrating real value throughout. It’s the academic equivalent of putting 20 Javascript frameworks on your Resume because you used them one time.
  - Boywithhalo 5y ago
    I agree that h-index isn't very indicative, but max citations seems like a poor alternative. There is a relation between citations and certain aspects of research quality, but citation count can be easily gamed, and is affected by many factors other than the quality of the work itself. Trying to use single measures to quantiy something as complex seems flawed to begin with.
    - mirker 5y ago
      Top N papers is already a metric used for professorship slots, so Max or other functions are me simplifying the situation. I agree though that citations are proxies for goodness and are ultimately poor indications of work. For example, a paper closing the field would have no follow-up work, but a provocative paper would have plenty (regardless of quality). I think a quantitative metric is necessary because many decisions eventually hit a bean-counter (e.g., admin type), who is not able to assess research quality in the time they have. But even with experts reviewing a set of candidates on research qualifications, the panel would be unable to objectively provide a full ranking of candidates. A perhaps better way to do it would be to perform proper accounting on research output which stabilizes across a 10-year timeline or so. Each researcher is a “stock” and the stock-price reflects current and projected “revenue”. If someone publishes only papers that go nowhere, their stock tanks. At top institutions, tenure kind of does this (only one time).
    - ItsMonkk 5y ago
      Exactly. > When a measure becomes a target, it ceases to be a good measure. - Goodhart's Law > The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor. - Campbell's Law You want to use metrics as a very broad breadth first search to help cull the search space and use trust systems as a depth first search. But once you have found signal through trust, you should completely ignore that first search and even look more into things that you had previously excluded. Find researchers that you agree with, find the papers that they recommend, read those papers and check that you agree with them. If they recommend something from a no-name university, look at those first. Unfortunately if this is not your field and you aren't able to determine quality, this becomes impossible. If it's important to you, you need to learn it. This is why I don't like non-technical managers. If the people who approve the grants do not understand the result, this is inevitable. It might work early on when trust still lingers, but as metrics take over the social systems always fall apart.
failwhaleshark 5y ago
I think you're right. There's only a few people with the curiosity, tenacity, and obsessive patience to keep up their idealism and enjoyment momentum to deliver consistent intellectual quality for its own sake. You basically have to be on the spectrum or very, very diligent. Most people just want more funding. (ProTip: Full-time grant writers are key.)
- sangnoir 5y ago
  > You basically have to be on the spectrum or very, very diligent. ...and be independently wealthy (or have wealthy benefactors). Science lives and dies by funding. We may be regressing to pre-20th century science, where most of the big discoveries were made by rich people who could afford to do science all day everyday.
  - robwwilliams 5y ago
    Hmm, I do not see this at all. 40 years in neuroscience and genetics and it is far larger and more open community now that is also much more open to diversity than it was in 1978. Hundreds of collaborators and none have supported their own research.
- robwwilliams 5y ago
  I am sympathetic to many of the ideas in this thread, but view many comments as too skeptical of the process and sociology of science and of career paths in science. The high entry barrier and generally low incomes compared to alternatives means an enrichment for individuals with genuine intellectual curiosity. Yes, publishing papers and getting grants involves some gaming, but that is also true of almost all forms of communication and interaction. Science is usually a game with a good cause. Over the next 50 years we will get far beyond our still primitive methods of publishing key data and our analysis (aka papers). You can see the transition already in the form of web services, Jupyter notebooks, R/Shiny, Pluto, etc. Soon most core publications will become living, growing structures instead sad mausoleums. Data and code quality will become much more important when data and workflows and logic are highly visible and highly granular in a new post-FAIR+ science world.

ProjectArcturis 5y ago

Just want to add, it's generally not a conscious choice. Everyone wants to do something that is both a genius experiment and gets published in Nature. The Science Game gets propagated in how scientists conceive and plan their work. Everything must be done in the paradigm of publications and grants. You can't do anything if it doesn't have at least interim results within a year or two. It's an unfathomable career risk to try something absolutely brand new. There's a good chance it fails, at which point you have a 2 year blank spot in your CV and you can never get back in the top 10% of scientists where you need to be to get R01 funding and run your own lab.

mirker 5y ago
One problem is the incentives are biased toward “ok” papers. h-index should be replaced with a more competitive index (e.g., Max over citations). The publication hyper-competitiveness is due to publication spam, which boosts citation and publication count. However, twisting a knob and getting slightly different results should be viewed as a failed project.
- ModernMech 5y ago
  > The publication hyper-competitiveness is due to publication spam, which boosts citation and publication count. The publication hyper-competitiveness is a direct result of grant hyper-competitiveness. Funding rates at top US agencies are around 10-20%. Grants are how researchers fund their research, pay their students, pay themselves, and earn tenure, so there's a lot riding on getting them. I see a lot of talk about the problems with publications on HN and possible fixes to the publication process (like requiring code for example, or as you suggest changing the important metrics), but publications really are just a proxy (and prerequisite) for the grant application process. You can change around the publication process all you want, but nothing will improve unless you address the issue of grants, because that's where the money is.
  - ProjectArcturis 5y ago
    But the question is, can you make grants more dependent on actual scientific breakthroughs rather than publication spam?
    - mirker 5y ago
      Right. Listing 10 papers with “ok” impact should be viewed as 10 “bad” papers and not 10 “good” papers. It’s extremely unlikely that a 50 page CV is demonstrating real value throughout. It’s the academic equivalent of putting 20 Javascript frameworks on your Resume because you used them one time.
- Boywithhalo 5y ago
  I agree that h-index isn't very indicative, but max citations seems like a poor alternative. There is a relation between citations and certain aspects of research quality, but citation count can be easily gamed, and is affected by many factors other than the quality of the work itself. Trying to use single measures to quantiy something as complex seems flawed to begin with.
  - mirker 5y ago
    Top N papers is already a metric used for professorship slots, so Max or other functions are me simplifying the situation. I agree though that citations are proxies for goodness and are ultimately poor indications of work. For example, a paper closing the field would have no follow-up work, but a provocative paper would have plenty (regardless of quality). I think a quantitative metric is necessary because many decisions eventually hit a bean-counter (e.g., admin type), who is not able to assess research quality in the time they have. But even with experts reviewing a set of candidates on research qualifications, the panel would be unable to objectively provide a full ranking of candidates. A perhaps better way to do it would be to perform proper accounting on research output which stabilizes across a 10-year timeline or so. Each researcher is a “stock” and the stock-price reflects current and projected “revenue”. If someone publishes only papers that go nowhere, their stock tanks. At top institutions, tenure kind of does this (only one time).
  - ItsMonkk 5y ago
    Exactly. > When a measure becomes a target, it ceases to be a good measure. - Goodhart's Law > The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor. - Campbell's Law You want to use metrics as a very broad breadth first search to help cull the search space and use trust systems as a depth first search. But once you have found signal through trust, you should completely ignore that first search and even look more into things that you had previously excluded. Find researchers that you agree with, find the papers that they recommend, read those papers and check that you agree with them. If they recommend something from a no-name university, look at those first. Unfortunately if this is not your field and you aren't able to determine quality, this becomes impossible. If it's important to you, you need to learn it. This is why I don't like non-technical managers. If the people who approve the grants do not understand the result, this is inevitable. It might work early on when trust still lingers, but as metrics take over the social systems always fall apart.

failwhaleshark 5y ago

I think you're right. There's only a few people with the curiosity, tenacity, and obsessive patience to keep up their idealism and enjoyment momentum to deliver consistent intellectual quality for its own sake. You basically have to be on the spectrum or very, very diligent. Most people just want more funding. (ProTip: Full-time grant writers are key.)

sangnoir 5y ago
> You basically have to be on the spectrum or very, very diligent. ...and be independently wealthy (or have wealthy benefactors). Science lives and dies by funding. We may be regressing to pre-20th century science, where most of the big discoveries were made by rich people who could afford to do science all day everyday.
- robwwilliams 5y ago
  Hmm, I do not see this at all. 40 years in neuroscience and genetics and it is far larger and more open community now that is also much more open to diversity than it was in 1978. Hundreds of collaborators and none have supported their own research.
robwwilliams 5y ago
I am sympathetic to many of the ideas in this thread, but view many comments as too skeptical of the process and sociology of science and of career paths in science. The high entry barrier and generally low incomes compared to alternatives means an enrichment for individuals with genuine intellectual curiosity. Yes, publishing papers and getting grants involves some gaming, but that is also true of almost all forms of communication and interaction. Science is usually a game with a good cause. Over the next 50 years we will get far beyond our still primitive methods of publishing key data and our analysis (aka papers). You can see the transition already in the form of web services, Jupyter notebooks, R/Shiny, Pluto, etc. Soon most core publications will become living, growing structures instead sad mausoleums. Data and code quality will become much more important when data and workflows and logic are highly visible and highly granular in a new post-FAIR+ science world.