4 ms·

As an academic, I find that positive or negative sentiment isn't expressed in citations — it's expressed in the review process before the paper is published. O

by toomim 4y ago

As an academic, I find that positive or negative sentiment isn't expressed in citations — it's expressed in the review process before the paper is published. Once a paper is published, academics assume that it's already been vetted and is good, and the citations rarely bother to express agreement or disagreement. When I cite a paper, I don't include a sentence saying whether the paper is good; I say why it is good, because the author of that paper is likely to be reviewing my paper, and I need to acknowledge his strengths before differentiating my work, or he'll kill my paper in the review process. My advisor taught me that the point of a citation is not to give your opinion on which other papers are good or bad; it's to tell the editors which other scientists should be on your paper's review committee. The way editors choose reviewers is to look through the citations, find people cited who they know, and then ask those people to review the paper. Thus, the author's goal when citing other work is to (1) highlight potential reviewers who would see your work favorably, and (2) say something positive about them to put them into a mood to see your work favorably. There is nary an incentive to express negative sentiment in citations.

toomim 4y ago
That said, I have hope we can solve this problem outside the academic incentive system. Here's a relevant prototype developed by my research lab: https://braid.news/about https://braid.news/about
- robocat 4y ago
  Thanks. I’ve wanted something similar to rank HN comments - I really want to read some users and I really don’t care to read the same tripe repeated by some other users. I presume it is a hobby project but calling it PeeryView and then using braid.news was confusing to me. I didn’t even realise the “home” link would take me to the site (or maybe I didn’t see the home link, because I look to top left for that? e.g. Y logo on HN)
jll29 4y ago
While the parent is right in stressing the importance of the review phase, which is a filter to decide whether we will ever see a candidate paper or not, it is indeed the case, as the OP posits, that there is merit in extracting the context in which a citation is used, which has been termed "citation polarity". For example, 'Chomsky (1969) was entirely wrong when he said that “it must be recognized that the notion of 'probability of a sentence' is an entirely useless one, under any known interpretation of this term".' There are Natural Language Processing methods and tools to extract the citation polarity. Researchers like S. Teufel have used citation polarity and other methods to analyze publications ("rhetorical zoning") and to create maps of the published literature: https://www.cl.cam.ac.uk/~sht25/az.html https://www.cl.cam.ac.uk/~sht25/az.html A scientific literature search engine (Semantic Scholar, CiteSeerX etc. - sadly Microsoft Academic has recently been discontinued) can benefit from such knowledge; for example, PageRank can be modified so as to incorporate citation polarity in the random walker model that underlies it. Think of it as adding a "dislike" button to a system that already offers a graph with "like" button relations.
- zozbot234 4y ago
  The Microsoft Academic dataset has been picked up by OpenAlex.org
anonymousDan 4y ago
I disagree that that is the sole (or even most important) point of a citation. Most of the time you can view a citation as part of the (extended, implicit) problem statement for the paper, e.g. here is what X did in this space, and we are different/better/build on that work by doing Y. If an important citation is missing, it shows you aren't familiar with what has been done before in relation to the problem you are trying to solve (bad). Of course reviewers and readers are free to disagree with your characterization of a cited paper in relation to your own. Having said that, maybe if you publish in journals a lot this is more of an issue. In conferences the Program Committee is usually set in advance so there is less scope to target specific reviewers (other than those on the PC).
sampo 4y ago
This is a very cynical and opportunistic way to do science. It may be the winning strategy to play the academic game and advance your career, though.
concinds 4y ago
Citation rings are a huge problem, but couldn't the sentiment-analysis side be readily addressed? There's a difference between papers that just cite a paper without judgment (+0), papers whose results explicitly support another paper (+1) and papers that explicitly reject another paper (-1). Such an AI could also bypass "citation analysis" and instead ingest the entire scientific literature, and find refutations in papers that don't even cite each other. In practice, 2 problems: 1) this requires sophisticated AI; and 2) many people will keep assuming that high citations = credible (i.e. assume that +0 and +1 are equivalent). But this method will help researchers go beyond "it's highly-cited therefore I trust it"; it'll help students and rigorous scholars find how well-supported a paper is, despite its popularity. And it'll surface refutations that may be hard to otherwise find; for example when a scholar that's famous in a field, and unknown in another, decides to publish a paper in the latter field that may not be widely seen. Ultimately (though this is far off: theoretically completely possible but highly technically difficult) we can also imagine AIs that ingest the whole literature, gains deep cross-domain knowledge, and are trained to detect poor methodologies, or automatically highlight new insights in one discipline that can enrich another discipline, or (ideally) make its own judgments about the merits of any finding based on the knowledge it acquired. After human review, it could systemically help "clean up" the scientific literature of bad methodologies and ideas, and free scientists to spend less time working on scientific dead ends. That last paragraph won't be doable for a while, but the first 3 seem readily within reach of today's AI technologies (and seems to be what scite is doing).
retcon 4y ago
>My advisor taught me that the point of a citation is not to give your opinion on which other papers are good or bad; it's to tell the editors which other scientists should be on your paper's review committee As a professional consumer of academic papers paying commercial rates for decades I appreciate your candor profoundly. Nevertheless what you say is a statement at least of disingenuity and in my opinion prima facie admission of the existence of a defrauded system.

by toomim 4y ago

toomim 4y ago

That said, I have hope we can solve this problem outside the academic incentive system. Here's a relevant prototype developed by my research lab: https://braid.news/about https://braid.news/about

robocat 4y ago
Thanks. I’ve wanted something similar to rank HN comments - I really want to read some users and I really don’t care to read the same tripe repeated by some other users. I presume it is a hobby project but calling it PeeryView and then using braid.news was confusing to me. I didn’t even realise the “home” link would take me to the site (or maybe I didn’t see the home link, because I look to top left for that? e.g. Y logo on HN)

jll29 4y ago

While the parent is right in stressing the importance of the review phase, which is a filter to decide whether we will ever see a candidate paper or not, it is indeed the case, as the OP posits, that there is merit in extracting the context in which a citation is used, which has been termed "citation polarity". For example, 'Chomsky (1969) was entirely wrong when he said that “it must be recognized that the notion of 'probability of a sentence' is an entirely useless one, under any known interpretation of this term".' There are Natural Language Processing methods and tools to extract the citation polarity. Researchers like S. Teufel have used citation polarity and other methods to analyze publications ("rhetorical zoning") and to create maps of the published literature: https://www.cl.cam.ac.uk/~sht25/az.html https://www.cl.cam.ac.uk/~sht25/az.html A scientific literature search engine (Semantic Scholar, CiteSeerX etc. - sadly Microsoft Academic has recently been discontinued) can benefit from such knowledge; for example, PageRank can be modified so as to incorporate citation polarity in the random walker model that underlies it. Think of it as adding a "dislike" button to a system that already offers a graph with "like" button relations.

zozbot234 4y ago
The Microsoft Academic dataset has been picked up by OpenAlex.org

anonymousDan 4y ago

I disagree that that is the sole (or even most important) point of a citation. Most of the time you can view a citation as part of the (extended, implicit) problem statement for the paper, e.g. here is what X did in this space, and we are different/better/build on that work by doing Y. If an important citation is missing, it shows you aren't familiar with what has been done before in relation to the problem you are trying to solve (bad). Of course reviewers and readers are free to disagree with your characterization of a cited paper in relation to your own. Having said that, maybe if you publish in journals a lot this is more of an issue. In conferences the Program Committee is usually set in advance so there is less scope to target specific reviewers (other than those on the PC).

sampo 4y ago

This is a very cynical and opportunistic way to do science. It may be the winning strategy to play the academic game and advance your career, though.

concinds 4y ago

Citation rings are a huge problem, but couldn't the sentiment-analysis side be readily addressed? There's a difference between papers that just cite a paper without judgment (+0), papers whose results explicitly support another paper (+1) and papers that explicitly reject another paper (-1). Such an AI could also bypass "citation analysis" and instead ingest the entire scientific literature, and find refutations in papers that don't even cite each other. In practice, 2 problems: 1) this requires sophisticated AI; and 2) many people will keep assuming that high citations = credible (i.e. assume that +0 and +1 are equivalent). But this method will help researchers go beyond "it's highly-cited therefore I trust it"; it'll help students and rigorous scholars find how well-supported a paper is, despite its popularity. And it'll surface refutations that may be hard to otherwise find; for example when a scholar that's famous in a field, and unknown in another, decides to publish a paper in the latter field that may not be widely seen. Ultimately (though this is far off: theoretically completely possible but highly technically difficult) we can also imagine AIs that ingest the whole literature, gains deep cross-domain knowledge, and are trained to detect poor methodologies, or automatically highlight new insights in one discipline that can enrich another discipline, or (ideally) make its own judgments about the merits of any finding based on the knowledge it acquired. After human review, it could systemically help "clean up" the scientific literature of bad methodologies and ideas, and free scientists to spend less time working on scientific dead ends. That last paragraph won't be doable for a while, but the first 3 seem readily within reach of today's AI technologies (and seems to be what scite is doing).

retcon 4y ago

>My advisor taught me that the point of a citation is not to give your opinion on which other papers are good or bad; it's to tell the editors which other scientists should be on your paper's review committee As a professional consumer of academic papers paying commercial rates for decades I appreciate your candor profoundly. Nevertheless what you say is a statement at least of disingenuity and in my opinion prima facie admission of the existence of a defrauded system.