10 ms·
It sucks as a metric but it does have some rough correlation in most cases, and I'm not aware of any better easily measurable metric - if you have one in mind,
by PeterisP 7y ago
It sucks as a metric but it does have some rough correlation in most cases, and I'm not aware of any better easily measurable metric - if you have one in mind, it'd be great to hear. The alternative of having a bureaucrat "simply judge quality" IMHO is even worse, even less objective, and even more prone to being gamed.
The main problem is that there is an objective need (or desire?) by various stakeholders to have some kind of metric that they can use to roughly evaluate the quality or quantity of scientist's work, with the caveat people outside your field need to be able to use it. I.e. let's assume that we have a university or government official that for some valid reason (there are many of them) needs to be able to compare two mathematicians without spending excessive time on it. Let's assume that the official is honest, competent and in fact is a scientist him/herself and so can do the evaluation "in the way that scientists want" - but that official happens to be, say, a biologist or a linguist. What process should be used? How should that person distinguish insigtful, groundbreaking novel and important research from pseudoscience or salami-sliced paper that's not bringing anything new to the field? I can evaluate papers and people in my research subfield, but not far outside of it. Peer review for papers exists because we consider that people outside of the field are not qualified to directly tell whether that paper is good or bad.
The other problem, of course, is how do you compare between fields - what data allows you to see that (for example) your history department is doing top-notch research but your economics department is not respected in their field?
I'm not sure that a good measurement can exist, and despite all their deep flaws it seems that we actually can't do much better than the currently used bibliographic metrics and judgement by proxy of journal ratings.
Saying "metric X is bad" doesn't mean "metric X shouldn't get used" unless a better solution is available.
- IggleSniggle 7y agoYes, bizarrely, reputation / trust is still the primary foundation of academia from a pragmatic perspective, even though it is the antithesis of science. At least some disciplines can have replication studies cross-culturally. It’s a hard problem to solve; knowledge is inherently a social/shared experience.
- pbhjpbhj 7y agoTrust is a curious one, it's not the antithesis of science, indeed it's required for science to actually work rather than simply be an idea. You trust in people, in consistency of physical laws, in coherency of your own mind, in constancy of temporal flow, in so many things because - as the Pyrrhonist refrains - nothing is certain, not even this.
- gdhbcc 7y agoWhy not simply use replication as a measure? Have your studies been replicated? How many other studies have you replicated? Would both help solve the replication crisis, and resolve this problem. Of course then you might have 10 000 studies replicating the same easy to do study... which is why the "score" should be reduced based on how many other times that study has been replicated.
- PeterisP 7y agoThis solution seems to assume that all studies are equal, and they're not. An insightful study that's replicable (but has not yet been) is valuable. A lousy study that's been replicated five times (not because it's interesting, but because it was easy to do, and the replicators knew that they'd be rewarded for replicating anything) is not valuable. A metric that says "number of studies" is IMHO even more arbitrary, more gameable, and more detached from actual value than citation count - which does have some notion that your study actually matters to other poeple; that it was worth writing that paper because someone read it.
- dmix 7y agoExtreme self replication!
- newacctjhro 7y agoThis might work for hard sciences, but not for mathematics. Or, I dunno, paleontology or sociology or other stuff.
- CrazyStat 7y agoIndeed. My research (in statistics) is primarily methodological: I invent and describe methods that might be useful, and on a good day prove some theoretical results demonstrating that they might be useful. There's nothing to replicate there. Citations can be a useful metric here, particularly if you can identify citations of people actually using the method (as opposed to people just mentioning it in passing, or other methodological researchers comparing their own methods to it).
- wpietri 7y agoI think a problem here is Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." [1] And it seems like there's an element of the streetlight effect [2], too; sometimes a bad metric really is worse than no metric. Also, I really question your notion that people outside a field should be able to evaluate the quality of someone's work, especially in academia, where the whole point is to be well ahead of what most people can understand. That theory seems like part of managerialism [3], which I'll grant is the dominant paradigm in the western corporate world. I understand why a managerialist class would like to set themselves up as the well-paid judges of everybody else. But I'm not seeing why anybody would willingly submit themselves to that. It's a commonplace here on HN that we avoid letting managers make technical decisions, however fancy their MBA, because they're fundamentally not competent to do it. That seems much more important for people doing cutting-edge research. [1] https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law [2] https://en.wikipedia.org/wiki/Streetlight_effect https://en.wikipedia.org/wiki/Streetlight_effect [3] https://en.wikipedia.org/wiki/Managerialism https://en.wikipedia.org/wiki/Managerialism
- dpark 7y ago> the whole point is to be well ahead of what most people can understand That’s not the case at all. Being at the leading edge of research should mean that you are creating new knowledge. That doesn’t imply that people cannot understand it. This expectation that laypeople cannot possibly understand science is one of the reasons so many papers are written so densely and obtusely. “They” can’t understand it anyway, right? Feynman said if he couldn’t explain it to freshmen he didn’t understand it himself.
- wpietri 7y agoI think a lot of cutting-edge work is also done at the edge of understanding, and that's fine. It can be hard enough to explain groundbreaking work to experts with deep context; it's reasonable to me that it takes more time and work to find the explanations that make sense to the average person. I do agree that researchers should be able to give decent "here's what I do" explanations to the general public. But that's very different than a member of the general public understanding the context well enough that they can judge the value of the work to the field.
- whatshisface 7y ago>The alternative of having a bureaucrat "simply judge quality" The bureaucrat in this case would be another university professor working in the same field.
- euske 7y agoI think that we could add some random (i.e. pure luck) factor for evaluation. Although this may sound unfair, almost all optimization problems do this, be it natural or man-made. Evolution does this by adding random gene mutation, and machine learning does this by randomizing certain parameters to avoid being stuck at local minima. In theory, the right mixture of rigid metrics and randomization can make a better result.
- stubish 7y agoBlaming the authors for demonstrating the deep flaws in this metric is certainly wrong however. This article is almost accusing some rigorous scientists, who choose to publish often and diligently reference their prior work to encourage fact checking and peer review, as frauds. I disagree with the assertion that bad metrics should be used if there are no alternatives. Bad metrics give wrong answers, and only the illusion of meaningful information. The most common use of bad metrics is to lie to people, and it isn't the scientists using the metrics but the organizations that employ them.
- clatan 7y agoHow about page rank instead of only using number of citations
- otakucode 7y ago>I'm not aware of any better easily measurable metric Why should an easily measurable metric which has meaningful value exist? It doesn't seem obvious to me that it should at all. Determining the capability of a researcher is inherently a very complex intellectual task. The desire is to reduce that task to something which removes the need for the person doing the evaluation to read and understand the produced research, or to even understand the field of study in many cases. Perhaps, instead, those who are put in charge of things like awarding grant funding, granting tenure at universities, and deciding who to hire to teach ought to be expected and required to evaluate the research on its merits. This would greatly increase the intellectual sophistication and capability needed for people in those positions, but the alternative will always be fairly easily exploitable because it is easier to goose a metric than to do solid research. We see the shortcomings of trying to reduce complex intellectual challenges to checklists or metrics all the time. And we simply ignore the alternative of relying upon intellectually capable people meeting the challenge. Personally, I don't understand why.