3 ms·

I see a bigger problem: the fact that things like citation counts are being used as a metric. We should be judging researchers by whether or not they are solv

I see a bigger problem: the fact that things like citation counts are being used as a metric. We should be judging researchers by whether or not they are solving problems (or if they are helping other researchers solve problems -- like developing new research tools and techniques, or publishing negative results that tell other researchers what to avoid). It makes no difference whether a paper is cited one time or a hundred times. It makes no difference if a research publishes one big paper or ten smaller papers. What matters is whether or not progress is being made.

czr80 13y ago
But this avoids the central issue - how can we know if a paper helps people make progress on solving problems? Citation counts are just a proxy for exactly that - to move away from them you have to come up with a better proxy or evaluation method.
- betterunix 13y ago
  Conferences are a good starting point: researchers meet each other, share work, judge others' work, and discuss research. I am not saying it is a complete solution, just that citation counts are a terrible solution that encourage the wrong things (and that we need something better).
- testbro 13y ago
  In the UK universities' research output are assessed by HEFCE using the "Research Excellence Framework" (which replaced the Research Assessment Exercise) [1]. Sadly, it suffers the same problem of being another metric [2]. It's a time sink for academics and it fosters a cutthroat atmosphere in the run-up. I heard some unconfirmed rumours that academics not submitting case studies to the REF will have their contracts re-assessed. I think this exposes the bigger problem of metrics in general. It's hard to determine how much impact research has, particularly in the short term. I'd be interested to see what the outcome of the REF is, but I remain sceptical and apathetic towards it, as does everyone else I've spoken to. [1] : http://www.ref.ac.uk/ http://www.ref.ac.uk/ [2] : I get the impression it does anyway - I'm not involved in it this round.
agilord 13y ago
I completely agree with this. Only if there was a way to translate this into politics (e.g. not just the popularity should decide where we are going).
- deleted 13y ago
  [deleted]
ekm2 13y ago
I agree with you that using citation counts as a metric is flawed.But I was primarily suggesting that we need "HN for science research papers ",not a particular way that it should be implemented.

czr80 13y ago

But this avoids the central issue - how can we know if a paper helps people make progress on solving problems? Citation counts are just a proxy for exactly that - to move away from them you have to come up with a better proxy or evaluation method.

betterunix 13y ago
Conferences are a good starting point: researchers meet each other, share work, judge others' work, and discuss research. I am not saying it is a complete solution, just that citation counts are a terrible solution that encourage the wrong things (and that we need something better).
testbro 13y ago
In the UK universities' research output are assessed by HEFCE using the "Research Excellence Framework" (which replaced the Research Assessment Exercise) [1]. Sadly, it suffers the same problem of being another metric [2]. It's a time sink for academics and it fosters a cutthroat atmosphere in the run-up. I heard some unconfirmed rumours that academics not submitting case studies to the REF will have their contracts re-assessed. I think this exposes the bigger problem of metrics in general. It's hard to determine how much impact research has, particularly in the short term. I'd be interested to see what the outcome of the REF is, but I remain sceptical and apathetic towards it, as does everyone else I've spoken to. [1] : http://www.ref.ac.uk/ http://www.ref.ac.uk/ [2] : I get the impression it does anyway - I'm not involved in it this round.

agilord 13y ago

I completely agree with this. Only if there was a way to translate this into politics (e.g. not just the popularity should decide where we are going).

deleted 13y ago
[deleted]

ekm2 13y ago

I agree with you that using citation counts as a metric is flawed.But I was primarily suggesting that we need "HN for science research papers ",not a particular way that it should be implemented.