6 ms·

I know nothing of his work, and he may well be an excellent researcher, but the use of “number of papers published” as a proxy for the quality of someone’s rese

I know nothing of his work, and he may well be an excellent researcher, but the use of “number of papers published” as a proxy for the quality of someone’s research is one of the biggest problems in academia. What matters is the precise nature of a researcher’s work, which is a qualitative measure that cannot be summarised with a simplistic metric. It’s so odd that we even discuss such things; it’s an overused analogy but “that Einstein publishes 5 PLA papers a year” vs “that dude developed the theory of general relativity” is so revealing. Don’t mean to pick on you personally, but as the top voted comment it feels like we’re rewarding this harmful perspective on academia. I’d like to say “academia needs to mature” but the problem is actually that it’s gone backwards, and undoing that invasion of management consultant culture and subsequent game playing into a somewhat altruistic and intellectual vocation seems like a very hard problem.

boibombeiro 5y ago
The parent post used the number of papers at the top conferences in their respectives fields. It means they are unquestionably pushing the state of art fowards. Or the system is completely broken. I see your point though. This is a recurrent discussion. I think the change of paradigm was positive. Nowadays, papers are published in interactive and incremental way, instead of, doing a parallel with software engineering, with the waterfall methodology. Publising the research incrementally allows more people to get involved, create more branches, and spot errors earlier. In sofware developement, the change of the process lead to huge advanvements. I believe this is also true for academia.
- randomsearch 5y ago
  > It means they are unquestionably pushing the state of art fowards. Or the system is completely broken. I agree with this statement: the system is completely broken. I've peered inside top level conferences and I can tell you that publishing there does not mean you are pushing the state of the art. Instead you may be: popular, good at playing the political game, lucky (when acceptance rates are low, luck plays a huge part - and why they are low is a story in itself), good at writing marketing copy rather than doing science, engaging in corruption to get your paper accepted, or playing to the gallery in a way that might even harm science. More widely, if your field is 99% politicians and 1% scientists, then the _scientific_ barrier to entry at an elite conference is not going to be a good endorsement of your work.
  - deleted 5y ago
    [deleted]
  - boibombeiro 5y ago
    I don't think you are completly wrong. For instance, last year there was a whole debacle at one of those top conferences because someone from nvidia, that was in the board of this conference, made (and posted on twitter!) a black list for people who said something that was not alligned with hers ideology. (I wonder what happened after the whole thing went quiet). Yet, I do still think those conference, and their sponsors, have interest in only selecting the papers based on their content. I don't think you can get them to publish your papers so many time only with influence. Also, things like blind peer review exists to mitigate that (although, it favors writing quality over content). Also. Politics isn't intrinsically evil. One of the great success of Einstein was spreading his idea. Argbly, others made more important achievement, yet didn't amass the same level of fame as him. In my experience, people who are good at politics, usually persue others path. Doing so only to get publication is not the most rewarding thing for those kinda of people.
    - krageon 5y ago
      What exactly was the sort of stuff people are blacklisted for? It could be claiming that racial segregation is a great thing (which should be shunned and kept away) or it could be supporting multivitamin consumption (for which it seems like an overreach).
      - boibombeiro 5y ago
        https://news.ycombinator.com/item?id=25597008 https://news.ycombinator.com/item?id=25597008 The twitter thread https://web.archive.org/web/20201215001433/https://twitter.com/AnimaAnandkumar/status/1338282250614411264 https://web.archive.org/web/20201215001433/https://twitter.c...
        krageon 5y ago
        I cannot find anything on what was done except shunning neonazis. Is that the extent?
- derbOac 5y ago
  The problem isn't the incremental nature of the papers or the collaboration, it's the equating of scientist quality with numbers of publications (or something like it). By your own analogy, it would be like saying "someone is a great computer scientist" based on the number of commits they or their colleagues make. I might even argue that regardless of how the product is or is not improved by the process, the way we attribute credit is worse. If you move to a collaborative incremental process but still talk in terms of specific individuals as the source, rather than a group, there's a problem.
  - op00to 5y ago
    The comment wasn’t that they had a lot of papers, but they had a lot of papers in influential conferences.
guerrilla 5y ago
Those weren't just papers published but papers published at elite conferences. Each one must be a serious contribution.
- randomsearch 5y ago
  > Each one must be a serious contribution I'm sorry to inform you that this is untrue. Elite conferences are full of work that is _scientifically_ awful and does not contribute anything to _science_.
  - op00to 5y ago
    Hello, this is the wild comment police. Please provide evidence that Elite conferences are “full of work that is scientifically awful”. Full meaning significantly most of the work is of no value? License, registration, and proof of your statement please.
  - choeger 5y ago
    That's not true in general. Case in point: https://icfp21.sigplan.org/track/icfp-2021-papers#event-overview https://icfp21.sigplan.org/track/icfp-2021-papers#event-over... the key is that "elite" in science means "small", amongst other things. Any conference that accepts so many papers that not very attendant can visit every presentation is already not "elite" anymore.
    - randomsearch 5y ago
      You've chosen a good conference for a counter-example, I agree. But even within that conference, there are examples of exactly what I'm talking about. I'm not going to personally criticise individuals as it's a widespread and often systemic problem. Peter Thiel has summarised this better than I ever could.
shrimpx 5y ago
I agree partially, but SIGGRAPH and CVPR are highly regarded and aren't known to accept crappy work. There are some "soft" sub-fields of CS that can be a wash, like HCI and Software Engineering. Those fields have relied more and more on "marketability" of ideas instead of depth of technical contribution. But top conferences in fields like graphics and programming languages (POPL, PLDI) are no joke, you will not be able to get a paper that isn't top-notch into those confs.
- randomsearch 5y ago
  Don’t want to talk about specific conferences or individuals but I agree that some conferences (really, a handful) have far less of this problem than others.

boibombeiro 5y ago

The parent post used the number of papers at the top conferences in their respectives fields. It means they are unquestionably pushing the state of art fowards. Or the system is completely broken. I see your point though. This is a recurrent discussion. I think the change of paradigm was positive. Nowadays, papers are published in interactive and incremental way, instead of, doing a parallel with software engineering, with the waterfall methodology. Publising the research incrementally allows more people to get involved, create more branches, and spot errors earlier. In sofware developement, the change of the process lead to huge advanvements. I believe this is also true for academia.

randomsearch 5y ago
> It means they are unquestionably pushing the state of art fowards. Or the system is completely broken. I agree with this statement: the system is completely broken. I've peered inside top level conferences and I can tell you that publishing there does not mean you are pushing the state of the art. Instead you may be: popular, good at playing the political game, lucky (when acceptance rates are low, luck plays a huge part - and why they are low is a story in itself), good at writing marketing copy rather than doing science, engaging in corruption to get your paper accepted, or playing to the gallery in a way that might even harm science. More widely, if your field is 99% politicians and 1% scientists, then the _scientific_ barrier to entry at an elite conference is not going to be a good endorsement of your work.
- deleted 5y ago
  [deleted]
- boibombeiro 5y ago
  I don't think you are completly wrong. For instance, last year there was a whole debacle at one of those top conferences because someone from nvidia, that was in the board of this conference, made (and posted on twitter!) a black list for people who said something that was not alligned with hers ideology. (I wonder what happened after the whole thing went quiet). Yet, I do still think those conference, and their sponsors, have interest in only selecting the papers based on their content. I don't think you can get them to publish your papers so many time only with influence. Also, things like blind peer review exists to mitigate that (although, it favors writing quality over content). Also. Politics isn't intrinsically evil. One of the great success of Einstein was spreading his idea. Argbly, others made more important achievement, yet didn't amass the same level of fame as him. In my experience, people who are good at politics, usually persue others path. Doing so only to get publication is not the most rewarding thing for those kinda of people.
  - krageon 5y ago
    What exactly was the sort of stuff people are blacklisted for? It could be claiming that racial segregation is a great thing (which should be shunned and kept away) or it could be supporting multivitamin consumption (for which it seems like an overreach).
    - boibombeiro 5y ago
      https://news.ycombinator.com/item?id=25597008 https://news.ycombinator.com/item?id=25597008 The twitter thread https://web.archive.org/web/20201215001433/https://twitter.com/AnimaAnandkumar/status/1338282250614411264 https://web.archive.org/web/20201215001433/https://twitter.c...
      - krageon 5y ago
        I cannot find anything on what was done except shunning neonazis. Is that the extent?
derbOac 5y ago
The problem isn't the incremental nature of the papers or the collaboration, it's the equating of scientist quality with numbers of publications (or something like it). By your own analogy, it would be like saying "someone is a great computer scientist" based on the number of commits they or their colleagues make. I might even argue that regardless of how the product is or is not improved by the process, the way we attribute credit is worse. If you move to a collaborative incremental process but still talk in terms of specific individuals as the source, rather than a group, there's a problem.
- op00to 5y ago
  The comment wasn’t that they had a lot of papers, but they had a lot of papers in influential conferences.

guerrilla 5y ago

Those weren't just papers published but papers published at elite conferences. Each one must be a serious contribution.

randomsearch 5y ago
> Each one must be a serious contribution I'm sorry to inform you that this is untrue. Elite conferences are full of work that is _scientifically_ awful and does not contribute anything to _science_.
- op00to 5y ago
  Hello, this is the wild comment police. Please provide evidence that Elite conferences are “full of work that is scientifically awful”. Full meaning significantly most of the work is of no value? License, registration, and proof of your statement please.
- choeger 5y ago
  That's not true in general. Case in point: https://icfp21.sigplan.org/track/icfp-2021-papers#event-overview https://icfp21.sigplan.org/track/icfp-2021-papers#event-over... the key is that "elite" in science means "small", amongst other things. Any conference that accepts so many papers that not very attendant can visit every presentation is already not "elite" anymore.
  - randomsearch 5y ago
    You've chosen a good conference for a counter-example, I agree. But even within that conference, there are examples of exactly what I'm talking about. I'm not going to personally criticise individuals as it's a widespread and often systemic problem. Peter Thiel has summarised this better than I ever could.

shrimpx 5y ago

I agree partially, but SIGGRAPH and CVPR are highly regarded and aren't known to accept crappy work. There are some "soft" sub-fields of CS that can be a wash, like HCI and Software Engineering. Those fields have relied more and more on "marketability" of ideas instead of depth of technical contribution. But top conferences in fields like graphics and programming languages (POPL, PLDI) are no joke, you will not be able to get a paper that isn't top-notch into those confs.

randomsearch 5y ago
Don’t want to talk about specific conferences or individuals but I agree that some conferences (really, a handful) have far less of this problem than others.