3 ms·

> The peer-reviewed research papers allows you to “measure” productivity. How many papers in top-tier venues did research X produce? And that is why it grew so

> The peer-reviewed research papers allows you to “measure” productivity. How many papers in top-tier venues did research X produce? And that is why it grew so strong. If I were king, my rule would be to look at the number of citations per paper and years of citation, rather than total publication count. I think that would solve a number of issues, although I'm sure that it would eventually be gamed as well. I should mention though: It turns out that people have been bitching about this for decades. For example: I was at Bell Labs in the 90s, and there was regular lunchtime discussion about the Least Publishable Increment (LPI). Everyone had some example about how the LPI had decreased to near zero, and then a few wags went on to show how the LPI had in fact gone negative in certain subfields, not least because reviewers were overloaded and couldn't keep up. Hopefully, some of this will be self-correcting: Publications are no longer the money-making activity they used to be, so resources will begin to dry up. Eventually, pubs are going to start rejecting valid papers for being too incremental.

colincooke 6y ago
Most people do actually judge by citations rather than raw pubs. It's an open secret that you can shove out papers (acceptance to any venue, regardless of the quality of the work is probabilistic) with enough effort. Getting cites is harder (other than self-cites which is a seperate issue). When I look up a researcher first thing I look at is their number of citations and H-10 index, then I look through their top papers (the ones that have been cited a lot). As far as I know hiring committees also care about these things. Of course it isn't the only thing that matters, or the most important, but its much more useful than raw number of papers published.
- mattkrause 6y ago
  Even those are garbage measures though. They can be gamed (albeit with a bit more effort), they vary a ton between fields and subfields, and citation rates often reflect the "prestige" of the lab/authors and the trendiness of the topic, rather than the intrinsic "merit" of a paper. I don't think there's any good automatic proxy for research quality.
  - t_serpico 6y ago
    Garbage is a strong word. I think there's definitely a moderate to strong correlation between h-index and quality of researcher. The downside with the h-index is that it also measures your ability to play the 'game' (i.e network, cite your peers, etc.), but I really see no way around that. You can also argue that this is an important skill to have. If you produce the most amazing research in the world and people don't cite you, that could point toward a number of red flags (e.g you're an asshole, your work isn't clear to understand, you don't appropriately cite your peers, etc.)
garden_hermit 6y ago
This is whats done in science evaluation—the H-index for instance is built from a combination of publication and citation counts. Similarly, journal prestige is quantified using the Journal Impact Factor, which aggregates the average number of citation a paper in the journal receives. As you speculate, these can also be gamed. Authors will cite themselves more to inflate their citations, or form citation carters with other authors to cite each other's work. Similarly, journals have been known to coerce authors to cite other works in the venue, in order to inflate their impact factor. Beyond these, there are issues with comparing citations across disciplines, article type, and other contexts. > Eventually, pubs are going to start rejecting valid papers for being too incremental. Many of hte most prestigious journals, such as Science, Nature, and Cell, already do this. However newer Mega-journals, like PLoS, have had explicitly the opposite policy, and state that they accept anything that is "sound science", no matter the size of its contribution; they have however become more selective over time.

colincooke 6y ago

Most people do actually judge by citations rather than raw pubs. It's an open secret that you can shove out papers (acceptance to any venue, regardless of the quality of the work is probabilistic) with enough effort. Getting cites is harder (other than self-cites which is a seperate issue). When I look up a researcher first thing I look at is their number of citations and H-10 index, then I look through their top papers (the ones that have been cited a lot). As far as I know hiring committees also care about these things. Of course it isn't the only thing that matters, or the most important, but its much more useful than raw number of papers published.

mattkrause 6y ago
Even those are garbage measures though. They can be gamed (albeit with a bit more effort), they vary a ton between fields and subfields, and citation rates often reflect the "prestige" of the lab/authors and the trendiness of the topic, rather than the intrinsic "merit" of a paper. I don't think there's any good automatic proxy for research quality.
- t_serpico 6y ago
  Garbage is a strong word. I think there's definitely a moderate to strong correlation between h-index and quality of researcher. The downside with the h-index is that it also measures your ability to play the 'game' (i.e network, cite your peers, etc.), but I really see no way around that. You can also argue that this is an important skill to have. If you produce the most amazing research in the world and people don't cite you, that could point toward a number of red flags (e.g you're an asshole, your work isn't clear to understand, you don't appropriately cite your peers, etc.)

garden_hermit 6y ago

This is whats done in science evaluation—the H-index for instance is built from a combination of publication and citation counts. Similarly, journal prestige is quantified using the Journal Impact Factor, which aggregates the average number of citation a paper in the journal receives. As you speculate, these can also be gamed. Authors will cite themselves more to inflate their citations, or form citation carters with other authors to cite each other's work. Similarly, journals have been known to coerce authors to cite other works in the venue, in order to inflate their impact factor. Beyond these, there are issues with comparing citations across disciplines, article type, and other contexts. > Eventually, pubs are going to start rejecting valid papers for being too incremental. Many of hte most prestigious journals, such as Science, Nature, and Cell, already do this. However newer Mega-journals, like PLoS, have had explicitly the opposite policy, and state that they accept anything that is "sound science", no matter the size of its contribution; they have however become more selective over time.