11 ms·

Reading through this quickly I see several problems that make me wonder if the author has worked in academia. 1. Publication venues aren’t created equal. Peopl

Reading through this quickly I see several problems that make me wonder if the author has worked in academia. 1. Publication venues aren’t created equal. People outside of academia don’t understand that anyone can and will start a journal/conference. If I want to launch the Proceedings of the of Confabulatory Results in Computer Science Conference, all I need is a cheap hosting account and I’m ready to go. In countries like the US this is explicitly protected speech, and the big for-profit sites run by Elsevier et al. will often go ahead and add my publications to their database as long as I make enough of an effort to make things look legitimate. In any given field there are probably a handful of “top” venues that everyone in the field knows, and a vast long tail ranging from mid-tier to absolute fantasist for-profit crap. If you focus on the known conferences you might see bad results, but if you focus on the long tail you’re doing the equivalent of searching the 50th+ page of Google results for some term: that is, you’re deliberately selecting for SEO crap. And given the relatively high cost of peer-reviewing the good vs not peer-reviewing the bad, unfiltered searches will always be dominated by crap (surely this is a named “law”?) Within TFA I cannot tell if the author is filtering properly for decent venues (as any self-respecting expert would do) or if they’re just complaining about spam. Some mention is made of a DARPA project, so I’m hopeful it’s not too bad. However even small filtering errors will instantly make your results meaningless. 2. Citations aren’t intended as a goddamn endorsement (or metric). In science we cite papers liberally. We do not do this because we want people to get a cookie. We don’t do it because we’re endorsing every result. We do it because we’ve learned (or been told) about a possibly relevant result and we want the reader to know about it too. When it comes to citations, more is usually better. Just as it is much better to let many innocent people go free rather than imprison one guilty one, it is vastly better to cite a hundred mediocre or incorrect papers than to miss one important citation. Readers should not see a citation and infer correctness or importance unless the author specifically states this in the text at which point, sure, that’s an error. But most citation-counting metrics don’t actually read the citing papers, they just do text matching. Since most bulk citations are just reading lists, this also filters for citations that don’t mean much about the quality of a work. The idea of using citation counts as a metric for research quality is a bad one that administrators (and the type of researchers who solve administrative problems) came up with. It is one of those “worst ideas except for any other” solutions: probably better than throwing darts at a board and certainly more scalable than reading every paper for quality. But the idea is artificial at best, and complaints like “why are people citing incorrect results” fundamentally ask citations to be something they're not. Overall there are many legitimate complaints about academia and replicability to be had out there. But salting them with potential nonsense does nobody any favors, and just makes the process of fixing these issues much less likely to succeed.

WalterBright 4y ago
> for-profit crap And of course we all know that making something for profit means it's bad.
- matthewdgreen 4y ago
  For-profit in this case means "I pay you $2500 and you let me give you a list of the peer-reviewers I'd like to review my paper, all of whom happen to be my friends and business associates."
darawk 4y ago
1. He addresses this repeatedly throughout the piece. Journal impact factor is (largely) uncorrelated to replication probability. 2. Yes, but this hardly seems like a defense of citing something false (without comment), or something that has literally been retracted years ago, which is a large part of his complaint. He is not suggesting the use of citation count as a metric for quality. I have no idea how you could have possibly gotten that from reading this article. A bullet point in his "what to do" section is literally "ignore citation counts".
- matthewdgreen 4y ago
  > He is not suggesting the use of citation count as a metric for quality. I have no idea how you could have possibly gotten that from reading this article. TFA is extremely clear that the presence of citations (in the aggregate, as a count) on “weak” papers is something the author considers a problem and a perhaps a moral failure on the part of citing authors. The author also believes that citations should be “allocated” to true claims. * “Yes, you're reading that right: studies that replicate are cited at the same rate as studies that do not. Publishing your own weak papers is one thing, but citing other people's weak papers?” Here citations are clearly treated as a bulk metric, and “weak” is a quality metric. * “ As in all affairs of man, it once again comes down to Hanlon's Razor. Either: Malice: [the citing authors] know which results are likely false but cite them anyway. or, Stupidity: they can't tell which papers will replicate even though it's quite easy.” Aside from being gross and insulting — here the author claims that the decision to cite a result can have only two explanations, malice and stupidity. Not, for example, the much more straightforward explanation that I mention above (and that the author even admits is likely.) * “ Whatever the explanation might be, the fact is that the academic system does not allocate citations to true claims.” The use of “allocate citations” clearly recognizes that citation counts are treated as a metric, and indicate that that the author wishes this allocation to be done differently. * “This is bad not only for the direct effect of basing further research on false results, but also because it distorts the incentives scientists face. If nobody cited weak studies, we wouldn't have so many of them.” Here the author makes it clear that they see citation count (correctly) as a metric that encourages researchers, and believes the optimal solution is to remove all (but perhaps explicitly negative?) citations to those papers. > A bullet point in his "what to do" section is literally "ignore citation counts". Yes, after extensively complaining about the fact that citations aren’t used by authors in a manner that reflects the way they’re used as a metric, then complaining further about the fact that authors do not use them this way and repeatedly urging them to change the way citations are used — the author then admits that their use of a metric is problematic and should be ended. We agree! The only problem here was that the author took a detour to a totally absurd place to get there.

WalterBright 4y ago

> for-profit crap And of course we all know that making something for profit means it's bad.

matthewdgreen 4y ago
For-profit in this case means "I pay you $2500 and you let me give you a list of the peer-reviewers I'd like to review my paper, all of whom happen to be my friends and business associates."

darawk 4y ago

1. He addresses this repeatedly throughout the piece. Journal impact factor is (largely) uncorrelated to replication probability. 2. Yes, but this hardly seems like a defense of citing something false (without comment), or something that has literally been retracted years ago, which is a large part of his complaint. He is not suggesting the use of citation count as a metric for quality. I have no idea how you could have possibly gotten that from reading this article. A bullet point in his "what to do" section is literally "ignore citation counts".

matthewdgreen 4y ago
> He is not suggesting the use of citation count as a metric for quality. I have no idea how you could have possibly gotten that from reading this article. TFA is extremely clear that the presence of citations (in the aggregate, as a count) on “weak” papers is something the author considers a problem and a perhaps a moral failure on the part of citing authors. The author also believes that citations should be “allocated” to true claims. * “Yes, you're reading that right: studies that replicate are cited at the same rate as studies that do not. Publishing your own weak papers is one thing, but citing other people's weak papers?” Here citations are clearly treated as a bulk metric, and “weak” is a quality metric. * “ As in all affairs of man, it once again comes down to Hanlon's Razor. Either: Malice: [the citing authors] know which results are likely false but cite them anyway. or, Stupidity: they can't tell which papers will replicate even though it's quite easy.” Aside from being gross and insulting — here the author claims that the decision to cite a result can have only two explanations, malice and stupidity. Not, for example, the much more straightforward explanation that I mention above (and that the author even admits is likely.) * “ Whatever the explanation might be, the fact is that the academic system does not allocate citations to true claims.” The use of “allocate citations” clearly recognizes that citation counts are treated as a metric, and indicate that that the author wishes this allocation to be done differently. * “This is bad not only for the direct effect of basing further research on false results, but also because it distorts the incentives scientists face. If nobody cited weak studies, we wouldn't have so many of them.” Here the author makes it clear that they see citation count (correctly) as a metric that encourages researchers, and believes the optimal solution is to remove all (but perhaps explicitly negative?) citations to those papers. > A bullet point in his "what to do" section is literally "ignore citation counts". Yes, after extensively complaining about the fact that citations aren’t used by authors in a manner that reflects the way they’re used as a metric, then complaining further about the fact that authors do not use them this way and repeatedly urging them to change the way citations are used — the author then admits that their use of a metric is problematic and should be ended. We agree! The only problem here was that the author took a detour to a totally absurd place to get there.