8 ms·
The core problem here is that universities think that citation statistics are a useful metric to evaluate the quality of the work of a scientist. There's plenty
by hannob 7y ago
The core problem here is that universities think that citation statistics are a useful metric to evaluate the quality of the work of a scientist. There's plenty of evidence that this is not the case or that even the reverse may be the case [1], but this idea refuses to die.
[1]
- godelski 7y agoDid you forget to include the source? (I'm using an app so maybe it's not displaying it?)
- esotericn 7y agoWhoosh.
- hdrujvw-4579 7y agonot everyone is in the same mindset, no need to be a jerk about it.
- esotericn 7y agotickle
- godelski 7y agoLol that's definitely what happened. Looked back after this and had a laugh at myself. It's been a long day. I appreciate the defense but I don't find the "whoosh" comment offensive. I'll own up to my lapse.
- Raphmedia 7y agoIs "Woosh" really being that much of a jerk? It's a reference to https://www.xkcd.com/1627/ https://www.xkcd.com/1627/
- Raphmedia 7y ago[1] https://news.ycombinator.com/item?id=20752035 https://news.ycombinator.com/item?id=20752035
- marmaduke 7y agoNot just individual scientists but entire labs.
- esotericn 7y agoBravo.
- lonelappde 7y agoYou forgot to edit post to link to your own comment.
- ghostbrainalpha 7y agoDid he forget or is the Hacker News smart enough to remove recursive links automatically? [Link Test] [/Link Test]
- deleted 7y ago[deleted]
- PeterisP 7y agoIt sucks as a metric but it does have some rough correlation in most cases, and I'm not aware of any better easily measurable metric - if you have one in mind, it'd be great to hear. The alternative of having a bureaucrat "simply judge quality" IMHO is even worse, even less objective, and even more prone to being gamed. The main problem is that there is an objective need (or desire?) by various stakeholders to have some kind of metric that they can use to roughly evaluate the quality or quantity of scientist's work, with the caveat people outside your field need to be able to use it. I.e. let's assume that we have a university or government official that for some valid reason (there are many of them) needs to be able to compare two mathematicians without spending excessive time on it. Let's assume that the official is honest, competent and in fact is a scientist him/herself and so can do the evaluation "in the way that scientists want" - but that official happens to be, say, a biologist or a linguist. What process should be used? How should that person distinguish insigtful, groundbreaking novel and important research from pseudoscience or salami-sliced paper that's not bringing anything new to the field? I can evaluate papers and people in my research subfield, but not far outside of it. Peer review for papers exists because we consider that people outside of the field are not qualified to directly tell whether that paper is good or bad. The other problem, of course, is how do you compare between fields - what data allows you to see that (for example) your history department is doing top-notch research but your economics department is not respected in their field? I'm not sure that a good measurement can exist, and despite all their deep flaws it seems that we actually can't do much better than the currently used bibliographic metrics and judgement by proxy of journal ratings. Saying "metric X is bad" doesn't mean "metric X shouldn't get used" unless a better solution is available.
- IggleSniggle 7y agoYes, bizarrely, reputation / trust is still the primary foundation of academia from a pragmatic perspective, even though it is the antithesis of science. At least some disciplines can have replication studies cross-culturally. It’s a hard problem to solve; knowledge is inherently a social/shared experience.
- pbhjpbhj 7y ago
- 19ylram49 7y agoHaha. Well done.
- asimjalis 7y agoPageRank might be better way to evaluate quality. It too can be gamed. Maybe not as easily, though.
- apathy 7y agoAka eigenfactor
- ravar 7y agoMy understanding is that eigenfactor rates journals, not individual papers, so if somehow you get low-quality (whatever you want that to mean) papers into nature it has no independent way to realize that your specific paper is low quality. Also eigenfactor is biased towards favoring larger journals, which is not obviously a good thing. It would honestly be really cool if someone did page rank for individual papers. It seems like a much saner metric than anything that is currently used.
- apathy 7y agoOh good grief you’re right. This is doubly sad because using an ensemble metric for per-author eigenfactor seems like it would be tractable. Carl Bergstrom is a smart guy so I suppose the practical implementation of the above must have some wrinkles, but with enough brute force it seems tractable. What I despise more than anything is the gaming that takes place for “impact factor”. I do OK by standard metrics but would very much like to know where I stand by less easily gamed metrics of influence.
- brownbat 7y agoThis is such a pointed reference that I can't tell if you're just being ironic. In a very loose sense, PR is the same algorithm universities use, evaluate quality of some content based on the number of references to that content. It is definitely gamed in similar ways. I'm surprised we haven't seen professors hire SEO firms to help increase citation counts of their research.
- coldtea 7y ago
- justicezyx 7y agoI am pretty sure self citation is required to be listed separately. I mean, even the eb1 visa application required that.
- jasonzemos 7y agoIt seems as if universities live in the dystopia that the software industry avoided when it stopped counting lines of code.
- cassowary37 7y agoBefore everyone goes bananas citing Goodhart's law: many universities and academic medical centers in the US don't care at all about impact factor - they care about grant $$, period full stop. (They appreciate the occasional high-impact paper that they can use in marketing materials, but it's really all about the $$.) And for what it's worth, I've almost never heard impact factors discussed at NIH study sections, where investigator quality is explicitly on the agenda. Reviewers talk about relevant prior publications in the field, esp in marquee journals. [this latter feature is the reason we don't just put everything on biorxiv or equivalent and move on.]
- dwd 7y agoSo when querying a count of citations, did anyone consider adding a GROUP BY contributors which will at least give you distinct groups of contributors (assuming they always get listed alphabetically) Even better split it into individual contributors to give a count of researchers who have cited the paper?
- abdullahkhalids 7y agoWhat about the metric: count citations from all papers on which you were not a co-author.
- ramblerman 7y agoThe flip side of that is in the soft sciences. There are so many PHDs walking around in gender studies, sociology that have published papers that were never cited once! We do need a metric imo, but I agree we don't have a perfect one yet.