5 ms·

The bigger issue here, and what is threatening the integrity of the research, is blind reliance on conference publication count as a proxy for research quality.

The bigger issue here, and what is threatening the integrity of the research, is blind reliance on conference publication count as a proxy for research quality. Maybe it's time to move on from some of these conferences, and focus on interactions that maximize sharing research findings. I know that is unrealistic, but like every other metric, conference acceptance ceases to have value once the metric itself is what people care about.

psychomugs 5y ago
I often go back to this talk from Michael Stonebraker about, in his terms, the diarrhea of papers [1]. It's difficult to justify time towards anything that doesn't lead to a near-term publication or another line on your CV. [1] https://youtu.be/DJFKl_5JTnA?t=853 https://youtu.be/DJFKl_5JTnA?t=853
- PeterisP 5y ago
  A really awesome talk - it's nowhere close to my field of research, but it is oh so relevant anyways, thank you for linking it and I'd recommend everyone interested in this larger topic to listen to what Stonebraker had to say on this. And one of the suggested solutions (at ~22:00) seems that it would work if it can be adopted - essentially, have top university administrators evaluate top x papers for hiring/tenure decisions and ignore everything beyond that number. What you measure is what you get; if you measure count, you get a deluge of 'least publishable units', if you measure your top 3 papers, then everyone will focus on quality instead of quantity. A counterargument probably is that it's easier for the administors to measure quantity in a way that seems objective and resistant to any arguments or appeals decisions, and it's far harder to objectively compare quality especially if the candidates are from different subfields of research.
  - anonymousDan 5y ago
    This is what the UK REF system does effectively. You get evaluated on at most your top 5 papers.
    - sgt101 5y ago
      The top 5 in the last 4 years - which is pretty lunatic when you think about it. A Ph.d - which is meant to extend human knowledge by 1 iota - takes 3 years to write. So what does 5 papers indicate? The REF is evaluated by humans. I take 2/3 weeks to read a paper. For good papers I might work on them for 6 weeks + to really get into the technique. How can a REF reviewer consume 5 papers to evaluate an academic? How can they consume 200 papers from 40 academics? The right thing would be to have the department sumbit it's top 5 papers - 3 reviewers could really see what is going on is a department then.
      - anonymousDan 5y ago
        REF is 7 years I believe. 2/3 weeks per paper - I'm not sure what stage of your career you are at but I can't see how this would be necessary to evaluate the quality of a paper ( which presumably has already passed peer review and been published in a top tier venue ). Also, a PhD is in effect a trainee researcher. I don't think it's unreasonable to expect a more experienced academic to have a higher output rate.
        sgt101 5y ago
        >which presumably has already passed peer review and been published in a top tier venue So - how many IEEE conferences are there!? Also things change, it was much easier to get into Neurips 7 years ago... but it's not necessarily the case that the papers from this year will have as much impact or be as good as the papers from then. And as Neurips itself showed, the peer review process is somewhat random - papers are rejected by different panels meaning that getting in is likely an achievement but also possibly a bit lucky. I don't think that evaluating the papers based on where they were published is a good way to allocate public money. As a strong concrete example, paper 13 from Ferguson's group at Imperial is probably one of the most important documents for the last 20 years (if you live in the UK or France), it was "published" on a website...
        anonymousDan 5y ago
- indeedmug 5y ago
  This is a great lecture. While he is specifically talking about database research, a lot of what he is relevant to academia.
  - ItsMonkk 5y ago
    "When a measure becomes a target, it ceases to be a good measure." - Goodhart's Law It's not even just about academia in general, it's a universal problem caused by lack of personal accountability through the globalization of talent. The exact same dynamic was shown multiple times in HBO's The Wire. Gaming the metrics. LPU's are no different than 10 minute YouTube videos.
geraltofrivia 5y ago
The playing field where one can share research findings is far more uneven, more tilted in favor of the prestigious labs and individuals more than the current field of conferences and the blind review process. Findings and papers from well known individuals (read: twitter accounts) do get far more attention , more citations. Of course, one can argue that, broadly, well known labs and individuals are wel known because of their tendency to do great work, write better papers. And that’s true. However, the above still holds, in my experience as PhD student in ML. Anecdotally, I have seen instances where a less interesting paper from a renowned lab got more attention and eventually more citations than a better paper accepted at the same venue by a less renowned lab on the same topic.
- cycomanic 5y ago
  I completely agree with you and this is a significant issue, because it makes it hard for someone not from one of the established places to get research attention. I would also argue that with the increased importance of the (mostly) commercial "high-impact" journals this has become worse. I know that some of the professional (non-expert) editors of these journals specifically look at the citation counts of authors before accepting to send them out to review, because their main aim is to get people reading the articles, not necessarily good science.
  - geraltofrivia 5y ago
    This is objectively worse. Commercial journals need to be bygone. I feel that the publishing venues in the field of machine learning are a great example of non commercial, (relatively) transparent, open access publications. Look at ICLR, ICML, ACL, NeurIPS, for instance. All of their proceedings are open access. Most papers their are available on arXiv as well (which has its pros and cons both that I'd rather not get into here). Many of them also have an open review process wherein the reviewers post their reviews for all to see in a forum like interface, allowing frequent back and forth with the authors. In the best case, its more akin to assist the authors refine their potentially sound findings, a dynamic process where the paper may get revised a couple times before acceptance. I believe journals need to adapt the openreview process. There are obvious parallels in the lifecycle of a journal submission (from submission to acceptance) and the openreview process, except that the latter is accelerated (for better or for worse).
mycologos 5y ago
I made a similar comment in another thread here, but the problem is that there will always be non-experts who want to assess the quality of research being done in different fields. These might be deans, department heads, VPs of R+D labs, and so on. Ideally, all research would have clear measurable impact, but a lot of good research doesn't, at least not immediately, so there needs to be some way for non-experts to measure quality. Conference/journal publications and citation counts are highly imperfect solutions, but I'm not sure what the better candidates are.
- nradov 5y ago
  The department head at least should be something of an expert, and be able to consult with more specialized experts. Then that department head can pass on an evaluation to management.
  - sgt101 5y ago
    Most experts (who understand what a department head is) don't want to be department heads. In the UK there used to be a tradition of making someone be head of department for 2 years or so and then letting them become a professor again so that the next junior could do the head of stint! This has faded because of much nastier politics (everyone hates you after you sacked their friends) and a rise in admin due to the industrialization of the bigger departments (its not a job for an amateur any more)
    - BlueTemplar 5y ago
      Looks like it could work if you had to do it in a different university ? (Also, less cronyism.)
- jonathanstrange 5y ago
  Assessment can never be done by non-experts. They do not understand the subject matter and cannot even judge which journals are good. It just makes no sense. When someone is hired (whether for tenure or time-limited), their research as a whole has to be evaluated by external, independent committees who take into account the content of the research and do not base their judgment on indicators only. There is no shortcut around that. The biggest annoyance nowadays is the decision-makers' insistence on "excellence", though. You cannot have only excellent people everywhere, as per the definition of "excellent", yet this demand is in every fucking guideline for postdocs and tenure-track position. It's absolutely ridiculous.

psychomugs 5y ago

I often go back to this talk from Michael Stonebraker about, in his terms, the diarrhea of papers [1]. It's difficult to justify time towards anything that doesn't lead to a near-term publication or another line on your CV. [1] https://youtu.be/DJFKl_5JTnA?t=853 https://youtu.be/DJFKl_5JTnA?t=853

PeterisP 5y ago
A really awesome talk - it's nowhere close to my field of research, but it is oh so relevant anyways, thank you for linking it and I'd recommend everyone interested in this larger topic to listen to what Stonebraker had to say on this. And one of the suggested solutions (at ~22:00) seems that it would work if it can be adopted - essentially, have top university administrators evaluate top x papers for hiring/tenure decisions and ignore everything beyond that number. What you measure is what you get; if you measure count, you get a deluge of 'least publishable units', if you measure your top 3 papers, then everyone will focus on quality instead of quantity. A counterargument probably is that it's easier for the administors to measure quantity in a way that seems objective and resistant to any arguments or appeals decisions, and it's far harder to objectively compare quality especially if the candidates are from different subfields of research.
- anonymousDan 5y ago
  This is what the UK REF system does effectively. You get evaluated on at most your top 5 papers.
  - sgt101 5y ago
    The top 5 in the last 4 years - which is pretty lunatic when you think about it. A Ph.d - which is meant to extend human knowledge by 1 iota - takes 3 years to write. So what does 5 papers indicate? The REF is evaluated by humans. I take 2/3 weeks to read a paper. For good papers I might work on them for 6 weeks + to really get into the technique. How can a REF reviewer consume 5 papers to evaluate an academic? How can they consume 200 papers from 40 academics? The right thing would be to have the department sumbit it's top 5 papers - 3 reviewers could really see what is going on is a department then.
    - anonymousDan 5y ago
      REF is 7 years I believe. 2/3 weeks per paper - I'm not sure what stage of your career you are at but I can't see how this would be necessary to evaluate the quality of a paper ( which presumably has already passed peer review and been published in a top tier venue ). Also, a PhD is in effect a trainee researcher. I don't think it's unreasonable to expect a more experienced academic to have a higher output rate.
      - sgt101 5y ago
        >which presumably has already passed peer review and been published in a top tier venue So - how many IEEE conferences are there!? Also things change, it was much easier to get into Neurips 7 years ago... but it's not necessarily the case that the papers from this year will have as much impact or be as good as the papers from then. And as Neurips itself showed, the peer review process is somewhat random - papers are rejected by different panels meaning that getting in is likely an achievement but also possibly a bit lucky. I don't think that evaluating the papers based on where they were published is a good way to allocate public money. As a strong concrete example, paper 13 from Ferguson's group at Imperial is probably one of the most important documents for the last 20 years (if you live in the UK or France), it was "published" on a website...
        anonymousDan 5y ago
indeedmug 5y ago
This is a great lecture. While he is specifically talking about database research, a lot of what he is relevant to academia.
- ItsMonkk 5y ago
  "When a measure becomes a target, it ceases to be a good measure." - Goodhart's Law It's not even just about academia in general, it's a universal problem caused by lack of personal accountability through the globalization of talent. The exact same dynamic was shown multiple times in HBO's The Wire. Gaming the metrics. LPU's are no different than 10 minute YouTube videos.

geraltofrivia 5y ago

The playing field where one can share research findings is far more uneven, more tilted in favor of the prestigious labs and individuals more than the current field of conferences and the blind review process. Findings and papers from well known individuals (read: twitter accounts) do get far more attention , more citations. Of course, one can argue that, broadly, well known labs and individuals are wel known because of their tendency to do great work, write better papers. And that’s true. However, the above still holds, in my experience as PhD student in ML. Anecdotally, I have seen instances where a less interesting paper from a renowned lab got more attention and eventually more citations than a better paper accepted at the same venue by a less renowned lab on the same topic.

cycomanic 5y ago
I completely agree with you and this is a significant issue, because it makes it hard for someone not from one of the established places to get research attention. I would also argue that with the increased importance of the (mostly) commercial "high-impact" journals this has become worse. I know that some of the professional (non-expert) editors of these journals specifically look at the citation counts of authors before accepting to send them out to review, because their main aim is to get people reading the articles, not necessarily good science.
- geraltofrivia 5y ago
  This is objectively worse. Commercial journals need to be bygone. I feel that the publishing venues in the field of machine learning are a great example of non commercial, (relatively) transparent, open access publications. Look at ICLR, ICML, ACL, NeurIPS, for instance. All of their proceedings are open access. Most papers their are available on arXiv as well (which has its pros and cons both that I'd rather not get into here). Many of them also have an open review process wherein the reviewers post their reviews for all to see in a forum like interface, allowing frequent back and forth with the authors. In the best case, its more akin to assist the authors refine their potentially sound findings, a dynamic process where the paper may get revised a couple times before acceptance. I believe journals need to adapt the openreview process. There are obvious parallels in the lifecycle of a journal submission (from submission to acceptance) and the openreview process, except that the latter is accelerated (for better or for worse).

mycologos 5y ago

I made a similar comment in another thread here, but the problem is that there will always be non-experts who want to assess the quality of research being done in different fields. These might be deans, department heads, VPs of R+D labs, and so on. Ideally, all research would have clear measurable impact, but a lot of good research doesn't, at least not immediately, so there needs to be some way for non-experts to measure quality. Conference/journal publications and citation counts are highly imperfect solutions, but I'm not sure what the better candidates are.

nradov 5y ago
The department head at least should be something of an expert, and be able to consult with more specialized experts. Then that department head can pass on an evaluation to management.
- sgt101 5y ago
  Most experts (who understand what a department head is) don't want to be department heads. In the UK there used to be a tradition of making someone be head of department for 2 years or so and then letting them become a professor again so that the next junior could do the head of stint! This has faded because of much nastier politics (everyone hates you after you sacked their friends) and a rise in admin due to the industrialization of the bigger departments (its not a job for an amateur any more)
  - BlueTemplar 5y ago
    Looks like it could work if you had to do it in a different university ? (Also, less cronyism.)
jonathanstrange 5y ago
Assessment can never be done by non-experts. They do not understand the subject matter and cannot even judge which journals are good. It just makes no sense. When someone is hired (whether for tenure or time-limited), their research as a whole has to be evaluated by external, independent committees who take into account the content of the research and do not base their judgment on indicators only. There is no shortcut around that. The biggest annoyance nowadays is the decision-makers' insistence on "excellence", though. You cannot have only excellent people everywhere, as per the definition of "excellent", yet this demand is in every fucking guideline for postdocs and tenure-track position. It's absolutely ridiculous.