5 ms·
Why not simply use replication as a measure? Have your studies been replicated? How many other studies have you replicated? Would both help solve the replicati
by gdhbcc 7y ago
Why not simply use replication as a measure? Have your studies been replicated? How many other studies have you replicated?
Would both help solve the replication crisis, and resolve this problem.
Of course then you might have 10 000 studies replicating the same easy to do study... which is why the "score" should be reduced based on how many other times that study has been replicated.
- PeterisP 7y agoThis solution seems to assume that all studies are equal, and they're not. An insightful study that's replicable (but has not yet been) is valuable. A lousy study that's been replicated five times (not because it's interesting, but because it was easy to do, and the replicators knew that they'd be rewarded for replicating anything) is not valuable. A metric that says "number of studies" is IMHO even more arbitrary, more gameable, and more detached from actual value than citation count - which does have some notion that your study actually matters to other poeple; that it was worth writing that paper because someone read it.
- dmix 7y agoExtreme self replication!
- newacctjhro 7y agoThis might work for hard sciences, but not for mathematics. Or, I dunno, paleontology or sociology or other stuff.
- CrazyStat 7y agoIndeed. My research (in statistics) is primarily methodological: I invent and describe methods that might be useful, and on a good day prove some theoretical results demonstrating that they might be useful. There's nothing to replicate there. Citations can be a useful metric here, particularly if you can identify citations of people actually using the method (as opposed to people just mentioning it in passing, or other methodological researchers comparing their own methods to it).
- didibus 7y agoWouldn't replication here just be peer reviews ?
- CrazyStat 7y agoIf you judge contributions by just getting papers through peer review then that's even worse than using citations.
- didibus 7y agoWell, I'm talking about a math paper. So to me, it's the same as a code review. Someone has to go over the logic and proofs, and double check no mistakes are made. The number of people who did and gave their approval would be a good indicator I can trust the paper. What does a citation do that's better then this? For experiments, or non math papers, you might need something more robust. I think mostly because reviewing the paper isn't really reviewing the full study, but only what the researcher put in the paper. So it is very hard to review methodology and details to be sure they followed proper protocols, etc. You'd need someone to have been reviewing the study as it is happening, and not just the output paper from it.
- CrazyStat 7y agoA citation indicates that other people actually care about the content of the paper. Consider Researcher A, who has one paper with a hundred citations, and Researcher B, who has ten papers with two citations each. Probably Researcher A has made a larger contribution. Whether you're in math or any other field, the fact that a paper is correct or reasonable enough to make it past peer review doesn't mean anybody gives a shit about it.
- jbsimpson 7y agoThe issue is how many researchers want to spend their time replicating the research of other people rather than doing their own original work. Getting funding is already incredibly hard, plus no-one is going to give you tenure or promote you for replicating the work of others.
- ohyes 7y agoIn academic situations yeah, but in industry? You can make money replicating other people’s work.