5 ms·

1. He addresses this repeatedly throughout the piece. Journal impact factor is (largely) uncorrelated to replication probability. 2. Yes, but this hardly seems

• matthewdgreen 4y ago
  > He is not suggesting the use of citation count as a metric for quality. I have no idea how you could have possibly gotten that from reading this article. TFA is extremely clear that the presence of citations (in the aggregate, as a count) on “weak” papers is something the author considers a problem and a perhaps a moral failure on the part of citing authors. The author also believes that citations should be “allocated” to true claims. * “Yes, you're reading that right: studies that replicate are cited at the same rate as studies that do not. Publishing your own weak papers is one thing, but citing other people's weak papers?” Here citations are clearly treated as a bulk metric, and “weak” is a quality metric. * “ As in all affairs of man, it once again comes down to Hanlon's Razor. Either: Malice: [the citing authors] know which results are likely false but cite them anyway. or, Stupidity: they can't tell which papers will replicate even though it's quite easy.” Aside from being gross and insulting — here the author claims that the decision to cite a result can have only two explanations, malice and stupidity. Not, for example, the much more straightforward explanation that I mention above (and that the author even admits is likely.) * “ Whatever the explanation might be, the fact is that the academic system does not allocate citations to true claims.” The use of “allocate citations” clearly recognizes that citation counts are treated as a metric, and indicate that that the author wishes this allocation to be done differently. * “This is bad not only for the direct effect of basing further research on false results, but also because it distorts the incentives scientists face. If nobody cited weak studies, we wouldn't have so many of them.” Here the author makes it clear that they see citation count (correctly) as a metric that encourages researchers, and believes the optimal solution is to remove all (but perhaps explicitly negative?) citations to those papers. > A bullet point in his "what to do" section is literally "ignore citation counts". Yes, after extensively complaining about the fact that citations aren’t used by authors in a manner that reflects the way they’re used as a metric, then complaining further about the fact that authors do not use them this way and repeatedly urging them to change the way citations are used — the author then admits that their use of a metric is problematic and should be ended. We agree! The only problem here was that the author took a detour to a totally absurd place to get there.
  • darawk 4y ago
    > TFA is extremely clear that the presence of citations (in the aggregate, as a count) on “weak” papers is something the author considers a problem and a perhaps a moral failure on the part of citing authors. The author also believes that citations should be “allocated” to true claims. As I see it, there are two independent properties that the author is saying ought to be dependent. And I think you (and I) actually think the same. If citations are going to be treated as a metric, then the way they are written (without regard for quality or accuracy) is bad. If citations are not going to be written without regard for quality and accuracy, then they shouldn't be used as a metric. Either one of these models would be fine. What is not fine is the present reality: Citations are written without regard for quality and accuracy, and then still used as a metric ubiquitously! Impact factors, the most common method of ranking journals, are literally measures of citations. > Yes, after extensively complaining about the fact that citations aren’t used by authors in a manner that reflects the way they’re used as a metric, then complaining further about the fact that authors do not use them this way and repeatedly urging them to change the way citations are used — the author then admits that their use of a metric is problematic and should be ended. The crux of your point though seems to be that nobody uses them as a metric, and I'm just going to have to fundamentally disagree with that. It's true that authors, when writing papers, appear not to give them the care that a metric would deserve. What is not true is that citations aren't used as prima facie evidence of quality/importance throughout academia.
    • matthewdgreen 4y ago
      >The crux of your point though seems to be that nobody uses them as a metric I didn't say that at all: what I said is right there in my post. What I did say is that citation counts are a bad and noisy metric, one that is a side-effect of measuring a tool that has a very different purpose, and which researchers can use (usefully) for a variety of reasons that don't require them to validate the technical correctness of all cited works. Nor would it even make sense for citations to be used that way. The author's criticism in TFA (which he delivers in the strongest and most explicitly moralistic terms) is that based on using this citation count metric, which he selected, the field is broken because bad work gets cited. That's his criticism and his choice of measures to make it. But since it is quite normal for people to cite work that they haven't carefully reviewed for technical correctness, this criticism is essentially bunk. At a deeper level, the criticism fails to appreciate how people use citations as a measure for academic promotion. In most cases tenure committees care about aggregate statistics like total citations, h-index or i10-index. If a researcher publishes a work that receives hundreds citations for ten years and then fails to replicate, then it basically doesn't matter if the work stops receiving future citations. A retraction might matter. Reports of the failed replication might matter. But nobody is going to lose out on a promotion specifically because some random paper receives 8,000 citations in the first ten years and then zero citations after the failed replication.