4 ms·

I often go back to this talk from Michael Stonebraker about, in his terms, the diarrhea of papers [1]. It's difficult to justify time towards anything that does

I often go back to this talk from Michael Stonebraker about, in his terms, the diarrhea of papers [1]. It's difficult to justify time towards anything that doesn't lead to a near-term publication or another line on your CV. [1] https://youtu.be/DJFKl_5JTnA?t=853 https://youtu.be/DJFKl_5JTnA?t=853

PeterisP 5y ago
A really awesome talk - it's nowhere close to my field of research, but it is oh so relevant anyways, thank you for linking it and I'd recommend everyone interested in this larger topic to listen to what Stonebraker had to say on this. And one of the suggested solutions (at ~22:00) seems that it would work if it can be adopted - essentially, have top university administrators evaluate top x papers for hiring/tenure decisions and ignore everything beyond that number. What you measure is what you get; if you measure count, you get a deluge of 'least publishable units', if you measure your top 3 papers, then everyone will focus on quality instead of quantity. A counterargument probably is that it's easier for the administors to measure quantity in a way that seems objective and resistant to any arguments or appeals decisions, and it's far harder to objectively compare quality especially if the candidates are from different subfields of research.
- anonymousDan 5y ago
  This is what the UK REF system does effectively. You get evaluated on at most your top 5 papers.
  - sgt101 5y ago
    The top 5 in the last 4 years - which is pretty lunatic when you think about it. A Ph.d - which is meant to extend human knowledge by 1 iota - takes 3 years to write. So what does 5 papers indicate? The REF is evaluated by humans. I take 2/3 weeks to read a paper. For good papers I might work on them for 6 weeks + to really get into the technique. How can a REF reviewer consume 5 papers to evaluate an academic? How can they consume 200 papers from 40 academics? The right thing would be to have the department sumbit it's top 5 papers - 3 reviewers could really see what is going on is a department then.
    - anonymousDan 5y ago
      REF is 7 years I believe. 2/3 weeks per paper - I'm not sure what stage of your career you are at but I can't see how this would be necessary to evaluate the quality of a paper ( which presumably has already passed peer review and been published in a top tier venue ). Also, a PhD is in effect a trainee researcher. I don't think it's unreasonable to expect a more experienced academic to have a higher output rate.
      - sgt101 5y ago
        >which presumably has already passed peer review and been published in a top tier venue So - how many IEEE conferences are there!? Also things change, it was much easier to get into Neurips 7 years ago... but it's not necessarily the case that the papers from this year will have as much impact or be as good as the papers from then. And as Neurips itself showed, the peer review process is somewhat random - papers are rejected by different panels meaning that getting in is likely an achievement but also possibly a bit lucky. I don't think that evaluating the papers based on where they were published is a good way to allocate public money. As a strong concrete example, paper 13 from Ferguson's group at Imperial is probably one of the most important documents for the last 20 years (if you live in the UK or France), it was "published" on a website...
        anonymousDan 5y ago
        No comment on the quality of many IEEE conferences :) I think most people active in a field know what the best venues are. I certainly don't disagree that there is a lot of randomness to the review process. And frankly picking which papers are likely to be most influential in ten years time is super difficult (e.g. see how often best papers have minimal subsequent impact). Regarding the Ferguson document, I haven't served on a REF review panel, but I think if you can justify it as being influential/high quality through other means that should be acceptable?
        sgt101 5y ago
        OFC you are right - it would be fairly career limiting to try to talk it down I would think! But I am concerned that the REF should work by the reviewers actually looking at the material, rather than looking at where the material was published...
indeedmug 5y ago
This is a great lecture. While he is specifically talking about database research, a lot of what he is relevant to academia.
- ItsMonkk 5y ago
  "When a measure becomes a target, it ceases to be a good measure." - Goodhart's Law It's not even just about academia in general, it's a universal problem caused by lack of personal accountability through the globalization of talent. The exact same dynamic was shown multiple times in HBO's The Wire. Gaming the metrics. LPU's are no different than 10 minute YouTube videos.

PeterisP 5y ago

A really awesome talk - it's nowhere close to my field of research, but it is oh so relevant anyways, thank you for linking it and I'd recommend everyone interested in this larger topic to listen to what Stonebraker had to say on this. And one of the suggested solutions (at ~22:00) seems that it would work if it can be adopted - essentially, have top university administrators evaluate top x papers for hiring/tenure decisions and ignore everything beyond that number. What you measure is what you get; if you measure count, you get a deluge of 'least publishable units', if you measure your top 3 papers, then everyone will focus on quality instead of quantity. A counterargument probably is that it's easier for the administors to measure quantity in a way that seems objective and resistant to any arguments or appeals decisions, and it's far harder to objectively compare quality especially if the candidates are from different subfields of research.

anonymousDan 5y ago
This is what the UK REF system does effectively. You get evaluated on at most your top 5 papers.
- sgt101 5y ago
  The top 5 in the last 4 years - which is pretty lunatic when you think about it. A Ph.d - which is meant to extend human knowledge by 1 iota - takes 3 years to write. So what does 5 papers indicate? The REF is evaluated by humans. I take 2/3 weeks to read a paper. For good papers I might work on them for 6 weeks + to really get into the technique. How can a REF reviewer consume 5 papers to evaluate an academic? How can they consume 200 papers from 40 academics? The right thing would be to have the department sumbit it's top 5 papers - 3 reviewers could really see what is going on is a department then.
  - anonymousDan 5y ago
    REF is 7 years I believe. 2/3 weeks per paper - I'm not sure what stage of your career you are at but I can't see how this would be necessary to evaluate the quality of a paper ( which presumably has already passed peer review and been published in a top tier venue ). Also, a PhD is in effect a trainee researcher. I don't think it's unreasonable to expect a more experienced academic to have a higher output rate.
    - sgt101 5y ago
      >which presumably has already passed peer review and been published in a top tier venue So - how many IEEE conferences are there!? Also things change, it was much easier to get into Neurips 7 years ago... but it's not necessarily the case that the papers from this year will have as much impact or be as good as the papers from then. And as Neurips itself showed, the peer review process is somewhat random - papers are rejected by different panels meaning that getting in is likely an achievement but also possibly a bit lucky. I don't think that evaluating the papers based on where they were published is a good way to allocate public money. As a strong concrete example, paper 13 from Ferguson's group at Imperial is probably one of the most important documents for the last 20 years (if you live in the UK or France), it was "published" on a website...
      - anonymousDan 5y ago
        No comment on the quality of many IEEE conferences :) I think most people active in a field know what the best venues are. I certainly don't disagree that there is a lot of randomness to the review process. And frankly picking which papers are likely to be most influential in ten years time is super difficult (e.g. see how often best papers have minimal subsequent impact). Regarding the Ferguson document, I haven't served on a REF review panel, but I think if you can justify it as being influential/high quality through other means that should be acceptable?
        sgt101 5y ago
        OFC you are right - it would be fairly career limiting to try to talk it down I would think! But I am concerned that the REF should work by the reviewers actually looking at the material, rather than looking at where the material was published...

indeedmug 5y ago

This is a great lecture. While he is specifically talking about database research, a lot of what he is relevant to academia.

ItsMonkk 5y ago
"When a measure becomes a target, it ceases to be a good measure." - Goodhart's Law It's not even just about academia in general, it's a universal problem caused by lack of personal accountability through the globalization of talent. The exact same dynamic was shown multiple times in HBO's The Wire. Gaming the metrics. LPU's are no different than 10 minute YouTube videos.