5 ms·

> and the primary reputation metric in science is being published in prestigious journals, such as Nature, Science, and The Lancet. Absolutely not. The primar

> and the primary reputation metric in science is being published in prestigious journals, such as Nature, Science, and The Lancet. Absolutely not. The primary metric is citations. Where do people get these ideas?

mjn 14y ago
Speaking of metrics, using a very rough one (ye olde Google Book Ngrams), we can guess that citation-counting became prominent in the late 1970s: http://books.google.com/ngrams/graph?content=citation+count%2Chighly+cited&year_start=1800&year_end=2008&corpus=15&smoothing=3&share= http://books.google.com/ngrams/graph?content=citation+count%...
- jpdoctor 14y ago
  I remember tenure committees would have "vertical inches" measured from the Science Citation Index. In the world of big data, not all citations are created equal, and I look forward to new metrics based on quality of citations.
  - mjn 14y ago
    I've long wanted some kind of citation-sorting for research, though I'm more skeptical of the whole business for assessment. For research, I think it could improve discoverability and ability to trace chains of influence. If I find a paper that was cited 190 times and I want to see whether anyone's really followed up on it (critiqued it, extended it, revised it, refuted it, etc.), the reverse-citation list is too noisy. It'd be great if I could filter the citations, for example to exclude "related work exists [2,5,12,44]" type throwaway citations.
nmcfarl 14y ago
From my time working with academic researchers it was clear that they absolutely coveted publication in the above journals as they are so widely read - unlike the niche journals in their fields. To paraphrase one of them "They're crap, but everyone reads them and everyone cites them"
- mjn 14y ago
  That's true, but then a focus on more metrics like citation counts and download counts seems like it'll just increase the strength of those journals, not undermine it. People will try to publish in Nature or Science to get attention for their paper, because those journals are widely read, and have good connections with journalists who'll further publicize your paper, making it easier to improve your metrics.
jessriedel 14y ago
For young researchers--say, within 5 years of your PhD--you haven't had time for citations to accumulate, and journal name really is the important thing. And even for older researchers, they go after big-name journals because of the citations generated.
- jpdoctor 14y ago
  > For young researchers--say, within 5 years of your PhD--you haven't had time for citations to accumulate, and journal name really is the important thing. Maybe for people who don't know better. For example: 5 years out is when a "real" tenure review takes place. (Tenure occurs at 7 years: Thumbs up or down usually occurs at 6 years, you get most of the readout at 5.) Mere publication may suffice at non-research institutions, which are primarily centers-for-profiting-off-education, but at any place where research is worth a damn, they have already measured impact. One of those measures is citations. The same was essentially true at places like Bell Labs, but the process was not as formal. Edit: I see from your profile that you are at TJ Watson? Last I knew, they definitely paid attention to citations for 5 year reviews, much like Bell. I find it hard to believe that they've watered down.
  - jessriedel 14y ago
    Maybe we're talking about different fields, or maybe you misunderstood when I said "say, within 5 years of your PhD"? By this I mean that 5 years is the rough time when citations become useful. Before that, when you're on your first or second post-doc, there just hasn't been enough time for a meaningful number to accumulate. Would you agree with my statement if I had said 3 or 4 years? EDIT: Let me put it this way: suppose you're 4.5 years out of your PhD and you're applying for a tenure-track assistant professor position after two post-docs. Your competitor has 30% more citations than you, but in average journals, and last year you published in Nature; that article hasn't had time to rack up cites (which it will). Without knowing anything else about situation, who do you think is more likely to get the position?
    - jpdoctor 14y ago
pseut 14y ago
> The primary metric is citations. What field are you in? This is absolutely wrong for tenure decisions in economics. It is pretty wrong for promotion from associate to full professor.

mjn 14y ago

Speaking of metrics, using a very rough one (ye olde Google Book Ngrams), we can guess that citation-counting became prominent in the late 1970s: http://books.google.com/ngrams/graph?content=citation+count%2Chighly+cited&year_start=1800&year_end=2008&corpus=15&smoothing=3&share= http://books.google.com/ngrams/graph?content=citation+count%...

jpdoctor 14y ago
I remember tenure committees would have "vertical inches" measured from the Science Citation Index. In the world of big data, not all citations are created equal, and I look forward to new metrics based on quality of citations.
- mjn 14y ago
  I've long wanted some kind of citation-sorting for research, though I'm more skeptical of the whole business for assessment. For research, I think it could improve discoverability and ability to trace chains of influence. If I find a paper that was cited 190 times and I want to see whether anyone's really followed up on it (critiqued it, extended it, revised it, refuted it, etc.), the reverse-citation list is too noisy. It'd be great if I could filter the citations, for example to exclude "related work exists [2,5,12,44]" type throwaway citations.

nmcfarl 14y ago

From my time working with academic researchers it was clear that they absolutely coveted publication in the above journals as they are so widely read - unlike the niche journals in their fields. To paraphrase one of them "They're crap, but everyone reads them and everyone cites them"

mjn 14y ago
That's true, but then a focus on more metrics like citation counts and download counts seems like it'll just increase the strength of those journals, not undermine it. People will try to publish in Nature or Science to get attention for their paper, because those journals are widely read, and have good connections with journalists who'll further publicize your paper, making it easier to improve your metrics.

jessriedel 14y ago

For young researchers--say, within 5 years of your PhD--you haven't had time for citations to accumulate, and journal name really is the important thing. And even for older researchers, they go after big-name journals because of the citations generated.

jpdoctor 14y ago
> For young researchers--say, within 5 years of your PhD--you haven't had time for citations to accumulate, and journal name really is the important thing. Maybe for people who don't know better. For example: 5 years out is when a "real" tenure review takes place. (Tenure occurs at 7 years: Thumbs up or down usually occurs at 6 years, you get most of the readout at 5.) Mere publication may suffice at non-research institutions, which are primarily centers-for-profiting-off-education, but at any place where research is worth a damn, they have already measured impact. One of those measures is citations. The same was essentially true at places like Bell Labs, but the process was not as formal. Edit: I see from your profile that you are at TJ Watson? Last I knew, they definitely paid attention to citations for 5 year reviews, much like Bell. I find it hard to believe that they've watered down.
- jessriedel 14y ago
  Maybe we're talking about different fields, or maybe you misunderstood when I said "say, within 5 years of your PhD"? By this I mean that 5 years is the rough time when citations become useful. Before that, when you're on your first or second post-doc, there just hasn't been enough time for a meaningful number to accumulate. Would you agree with my statement if I had said 3 or 4 years? EDIT: Let me put it this way: suppose you're 4.5 years out of your PhD and you're applying for a tenure-track assistant professor position after two post-docs. Your competitor has 30% more citations than you, but in average journals, and last year you published in Nature; that article hasn't had time to rack up cites (which it will). Without knowing anything else about situation, who do you think is more likely to get the position?
  - jpdoctor 14y ago

pseut 14y ago

> The primary metric is citations. What field are you in? This is absolutely wrong for tenure decisions in economics. It is pretty wrong for promotion from associate to full professor.