6 ms·
Citation cartels help mathematicians-and their universities-climb the rankings
- jampekka 3y agoWhen all you have is a counter, everything looks like a bean.
- _kb 3y agoSo this essentially appears to be the SEO blogspam playbook oozing into academia? I wonder if any (likely well considered) techniques developed to combat this context may be able to be backported to help reclaim the usefulness of general internet search.
- amadeuspagel 3y agoGoogle is already way more advanced then this. If anything the question is more the reverse: Can Google Scholar publish a university ranking?
- eimrine 3y agoGoogle is advancing only in the field of showing ads.
- amadeuspagel 3y agoThe techniques described in the article would not work on Google. That makes Google more advanced then Clarivate.
- eimrine 3y ago> I wonder if any (likely well considered) techniques developed to combat this context may be able to be backported to help reclaim the usefulness of general internet search. For a level of an individual researcher this is easy, you just learn what is true and what is false. For a level of the whole society the battle is already lost, your (highly valuable and competent) solution to make a gigachad search just will be either lost or gamed or sued/prohibited.
- bsdpufferfish 3y agoSEO spam is worse than ever. The only thing Google figured out is to raise the cost of website above 0 such as by requiring ssl and well written English.
- lucioperca 3y ago"All metrics of scientific evaluation are bound to be abused. Goodhart's law [...] states that when a feature of the economy is picked as an indicator of the economy, then it inexorably ceases to function as that indicator because people start to game it." https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- User3456335 3y ago[flagged]
- throwaway12314 3y agoI worked as a research assistant for a bit over a year after the graduation and I had an experience that put me off. We submitted a paper and got some review feedback and one of them contained something like this ... this and that paper also worked on this research area and contain this and that stuff you can citate ... The prof I was working with told me that this anonymous reviewer was probably the writer of those papers and asking for citation.
- fock 3y agowell, if it's that transparent... I still wonder what to do with a handful of slightly off, but relatively good papers suggested to us. 2/5 would augment existing citations, 3/5 are related but not really relevant... No author is shared between them.
- enriquto 3y agoWhy did that put you off? This is a very common thing for reviewers to do. It is useful for you, because it makes you aware of other people who work in the same problem as you, leading to future job prospects, etc. But more importantly, it is useful for the readers of the article! The alternative is several independent communities of researchers that work separately on the same problem, but do not acknowledge the existence of each other. Now, that would put you off!
- mycologos 3y ago> The prof I was working with told me that this anonymous reviewer was probably the writer of those papers and asking for citation. I think this is mostly just a kind of fun way of complaining about reviewers. Having been a reviewer for many computer science conference, I've often had the names of other reviewers visible to me. Most requests for citations are not for the reviewer's own work, and truly irrelevant citation requests seem to be pretty rare. I do perceive journal/lab-centric fields to be worse about this, though.
- chriskanan 3y agoIt depends on context. Often reviewers are chosen because they work specifically in that area, and of course they know their work best. I've often done this as a reviewer in appropriate situations, where I point out closely related work that I've or others have done. Of course, I've also seen EDITORS email me saying they would like me to cite some papers from their journal after I submitted my paper. That was definitely a turn off.
- McDyver 3y agoThis seems like the early days of search engines and seo poisoning, where random sites were referencing others to make them climb up the search results.
- tokai 3y agoNah its older than SEO shenanigans. Interestingly quantitative research evaluation was an inspiration for page rank, so it make sense that they have similar issues.
- maranas 3y agoThis has been an issue with university rankings for a while. A lot of top US universities engage in this practice too - anecdotal, but I've heard a lot of professors force students to add/remove some citations, or even add their names to the list of people who worked on a paper to help the numbers for their university. It would be good to see what the criteria is for deciding if a journal is "to be taken seriously". I imagine for example that Chinese or Arabic language journals wouls be published and citsd in journals of those languages. That doesn't necessarily mean that they arent to be raken seriously in the field, it's just that they aren't Western publications.
- xqcgrek2 3y agoI've seen it go in the other direction too. Groups deliberately not citing other competing groups because it might help them. It's like the other groups don't exist.
- chriskanan 3y agoI've observed this in multiple AI niches. In some cases I've emailed people saying they ignored very similar work and failed to cite it, and in at least some cases, they were apologetic and said they would update the arXiv version. Although of those times, they do that 50% of the time. Kind of tells you that the reviewers at top AI conferences themselves aren't that familiar with the breadth of the literature.
- michaelmior 3y agoTo be fair, there are so many publications in fields such as AI that it's really hard to stay on top of things, sometimes even within your own specific subarea. I'm not saying that to give reviewers a total pass, but I think it's reasonable that sometimes a group of reviewers might miss a relevant paper.
- generic92034 3y agoYes, there are just too many publications, even with a very narrow focus. I am reminded of this article: https://slatestarcodex.com/2017/11/09/ars-longa-vita-brevis/ https://slatestarcodex.com/2017/11/09/ars-longa-vita-brevis/
- oytis 3y agoIs it just about mathematicians? Is it something new? I though that was pretty much how academia is done.
- zensayyy 3y agosame. I read through publications of my institute for my MS (EU, pretty big tech-focused university) and they pretty much also cite in circles (same institute and other related institutes of our university).
- bsdpufferfish 3y agoThere are also ethnic/cultural circles as well (Chinese students come to us to study with Chinese professors who only cite other Chinese papers).
- Sirizarry 3y agoI’ve never been a part of academia but I know it’s been fraught with all sorts of fraud for the last few decades (maybe forever idk) but this is a new type of weirdness to me. Is anything trustworthy is that space?
- User3456335 3y agoIt's not really fraud though. If we get paid to upvote each other, the problem is not that we upvote each other but that there's someone paying us to do so.
- brodo 3y agoI've been told who and what to cite in articles (even if it was only very tangentially relevant) as a Ph.D. student in HCI as far back as 2014. I also had to put my advisor as a co-author on papers that he didn't even bother to read...
- butler14 3y agoBacklink farms for universities
- ajsnigrutin 3y agoAcademia wrote the playbook and now they're abusing the rules. Nothing new here.
- questinthrow 3y agoI wonder what percentage of academic work is based on cargo culting and practices like these. There are more "scientists" than ever and yet we dont really have a quantitative increase of people that are as productive as Euler, Einstein, Newton, Neumann or whoever you want to pick as a luminary
- mjburgess 3y agoWould any of them work in academia today? It seems unlikely. There are research problems to solve in industry, with more prestige, more pay, and so on.
- bsdpufferfish 3y agoCompanies hire few of these people too (bad culture fit).
- tutfbhuf 3y agoI think that also has to do with the amount of complexity. Henri Poincaré is often said to have been the last man who knew all of math. Today, you need research teams across the world working on a very narrow topic to make tiny progress iteratively in the best case. I think the time when one man could bring us a century forward like Newton or Einstein is gone.
- questinthrow 3y agoYeah the quantity and complexity of what you need to know now is daunting. But even so I dont think Einstein for example had to know the entirety of physics and math in order to produce a groundbreaking revolution in physics.
- mandmandam 3y ago> I think the time when one man could bring us a century forward like Newton or Einstein is gone. I'm certain his contemporaries said the same about Pythagoras. You're not wrong about how much more complex things have gotten, certainly. But you don't know what you don't know. Another Ramanujan or von Neumann could be right around the corner. Just look at all the wasted potential in our education system - it's impossible to quantify just how much effort is profoundly wasted. This article points directly to that wastage. We've tolerated an academic system that's hated by just about everyone except publishers, in an era where publishing is as close to free as it could be.
- ronald_raygun 3y agoIn mathematics, this is just called a clique!
- _kb 3y agoI thought it was a semigroup.
- User3456335 3y agoNow I realize why it's called a field. I always thought of a physical field but this makes way more sense. If there's enough division in a group, it is considered a field.
- ants_everywhere 3y agoThese citation games are at least decades old. The PageRank algorithm came out of the literature on ranking academic papers by citation, and many of the SEO manipulation techniques work in both cases. I'd guess it's being done on an increasing scale given the article. As a semi-related phenomenon, I also feel like mathematicians are culturally prone to under-citation. I don't just mean that nobody cites older mathematicians like, say, Leibniz or Euler, for techniques they invented. But pretty frequently I'd read papers that really seemed like they were heavily indebted to a paper that came only a few years before but didn't appear in the 5-10 citations. Maybe there it wasn't mandatory to cite those papers, but it made it hard to chain backwards into the literature to get more context on methods and ideas. And sometimes it came off as the author trying to make a tool appear ex nihilo. I'm sure this happens in other fields too.
- _the_inflator 3y agoSEO and CO (citation optimization) maybe rhyme for a reason.
- fluoridation 3y agoMy understanding is that citations are used to avoid having to argue for the truth value of a proposition, not to give credit, nor to establish some kind of chain of trust. A sort of "I assume this is true, if you want to argue about it take it up with this guy". If two papers are very similar in structure, but neither uses the conclusions of the other to support the truth of a dependent claim, then I don't see why the older paper should be cited by the newer paper.
- michaelmior 3y agoIt's somewhat field dependent. I would generally expect citations to be used for both purposes. That is, if someone is proposing work substantially similar to existing work, they should be expected to cite that other work and identify differences with the new work since novelty is a key component generally (although not always) required for publication in my area.
- canadiantim 3y ago
- bArray 3y agoThe situation is out of control. I've also seen that during peer review a reviewer may give a list of publications they 'suggest' for the literature review, sometimes completely unrelated fields. If you choose not to, you will likely see your paper rejected. The truth is that almost all papers published across almost all fields cannot be replicated, full stop. For those that can, the results rarely match up, indicating that the tests are chosen to boost the performance of the method. If you want to boost citations legitimately, purposefully leak your own paper to get past the publisher pay wall, make it colourful and accessible (if it's too dense they are unlikely to understand it), give examples on how to replicate your method, the abstract is likely the only part that will be read so make it count.
- eimrine 3y ago> The truth is that almost all papers published across almost all fields cannot be replicated, full stop. This is not true for Mathematics.
- sukruh 3y agoSure, but most Maths papers are read by like 3 people in total globally, so there's that.
- psychoslave 3y agoCitation needed. XD
- lucioperca 3y agoThe pareto effect!?
- User3456335 3y agoIf that's true it's only because most people don't understand these pure maths fields. But their results are generally very useful to the applied mathematicians (whose results are and should be used in most other fields) and in this way there is still an impact although it's not always reflected in the citations. The fact is that these pure mathematicians could almost surely write the more applied papers but don't because it would feel to repetitive. You don't need to read Euler's work in order to indirectly benefit from it. It's the same for a lot of pure math.
- mycologos 3y agoNote that the article specifies that these cartels mostly feature participants from bad universities in countries that have really whacked-out academic reward mechanisms that arose because some powerful central government force that doesn't understand research productivity came up with a dumb way to incentivize it. Prestige is not a failsafe guarantor of quality, but it kind of is, in the sense that somebody with a reputation to protect is going to be more careful about coming across as "unserious", and doing stuff like this is a good way to get a bad reputation quickly.
- lupire 3y ago"Prestigious" people just do it in a more refined way, managing perception instead of raw numbers.
- alephnerd 3y agoWe can safely assume MIT has a much higher quality output than King Abdulaziz University back in Jeddah. In general, State Flagships and Ivy Leagues+Ivy Tier Privates are largely comparable research output wise. The GP is right about prestige minimizing bad practices. Look at how diluted Northeastern's brand has become by trying to artificially inflating their prestige [1] [1] - https://www.bostonmagazine.com/news/2014/08/26/how-northeastern-gamed-the-college-rankings/ https://www.bostonmagazine.com/news/2014/08/26/how-northeast...
- jampekka 3y agoIt happens everywhere to some extent.
- ineedasername 3y ago>powerful central government force that doesn't understand research productivity [1]* I'm sure they understand it. But their goal in doing this isn't really research productivity (though it would be a happy byproduct) it's to obtain prestige, or its illusion. And/or be able to tell the people of your country how great you're doing. "Our country is a rising powerhouse in field X, in the top 3 in the world and still rising!" It's internal propaganda. And externally targeted towards populations of other countries unfamiliar with the academic publishing system, which is most people. So for example the vast majority of people in the US that hear these claims by China, which are then echoed in click-bait headlines or simply taken at face value after a journalist checks something like the various HCR lists. It can even be propaganda, not from China, but from other countries. One of the studies was funded by the US State Department to an Australian group, thus obscuring a primary influence on the results of the study. Then even a single newspaper article can spark a dozen others citing it, propagating the message. All of these directly or indirectly rely on citations to support their headlines [2] https://www.science.org/content/article/china-rises-first-place-most-cited-papers https://www.science.org/content/article/china-rises-first-pl... https://www.wsj.com/articles/american-universities-continue-to-falter-in-world-rankings-china-rising-11665535646 https://www.wsj.com/articles/american-universities-continue-... https://money.usnews.com/investing/news/articles/2023-03-02/china-leads-us-in-global-competition-for-key-emerging-technology-study-says https://money.usnews.com/investing/news/articles/2023-03-02/... https://www.reuters.com/article/idUSTRE72R6FQ/ https://www.reuters.com/article/idUSTRE72R6FQ/ [1] The quality of this comment will be measured by metrics that I made up myself. [2] https://www.reuters.com/technology/china-leads-us-global-competition-key-emerging-technology-study-says-2023-03-02/ https://www.reuters.com/technology/china-leads-us-global-com... *This comment was funded by a grant from its author.
- blantonl 3y agoA classic textbook example of perverse incentives and unintended negative consequences. Someone could write a paper on this.
- _v7gu 3y agoIt’s not like the “principal-agent problem” already exists in the literature
- achenet 3y agoGoodhart's law: When a measure becomes a target, it ceases to be a good measure.
- xqcgrek2 3y agoCitations are simply a flawed metric for anything useful. A prominent paper with a false hypothesis can get many citations because people reference it when they refute it.
- bachmeier 3y agoThat's the example I often use. There are papers in prominent journals that draw negative citations. A related but much larger problem is that citations are a measure of the number of researchers working on a particular problem and nothing else. There was one topic in particular that had many researchers making small contributions, but all of those papers cited all the others, simply because they were written on that topic. Maybe the methods weren't the best or the conclusions weren't robust, but the only reason to cite a paper is because it is prior literature. It's not like you can refuse to cite papers because you believe they are of low quality.
- lkdfjlkdfjlg 3y agoOnce a measurement becomes a target, it ceases to be a good measurement.
- Tycho 3y agoCan some billionaire not fund some sort of 'Academic Integrity Institute' non-profit - bust citation rings - catch plagiarism - flag statistical malpractice/mistakes - attempt replicate published studies/experiments Could give jobs to loads of young scientists and be a huge net positive for society.
- deleted 3y ago[deleted]
- alephnerd 3y agoWhat's the point? Most R&D is driven by Government Procurement or Commercialization anyhow, which can validate if shit works or not. A lot of the subfields with a replication crisis are those that aren't exactly "tangible", and as such don't have a quantitative way or vested interest in validating that stuff works as expected. If I'm researching a compound as a MatSE on a grant from the DoD or Applied Materials, it's on the assumption that tangible results will happen, and any attempt at fraud will be detected once commercialization or productionizing kicks in. At the end of the day, a field like Pure Math or Anthropology just doesn't have a commercial incentive that forces reproducibility because there isn't a tangible impact.
- lupire 3y agoWhy are taxpayers paying these people to tell us lies about things that don't matter? That's doubly wrong, even worse than truths that don't matter.
- alephnerd 3y ago> Why are taxpayers paying these people to tell us lies about things that don't matter Research that gets federal grantmaking attached to it essentially gets validated at the production stage. Fields dealing with a severe replication crisis are also those which aren't validating or productionizing research.
- j7ake 3y agoThe solution is to abandon quantitative university rankings. They're obviously not useful.
- vladms 3y agoWhat about taking them with a grain of salt? Just because something is approximate that does not mean it is not useful. I never got this obsession with detailed rankings. Sure "above average" and "under average" makes sense, but position x or x + 1 out of 10k does not make much sense anymore (and by similarity if you are 1 or in the first 1k out of 10k could be similar)
- dev1ycan 3y agoI can't wait until AI is used to take away thousands if not dozens of thousands of masters and PHDs
- WhyOhWhyQ 3y agoWhy's that? Just because you're vindictive or something?
- kzz102 3y agoCitation numbers doesn't provide any useful information for a serious researcher. It's all a game for administrators.
- lupire 3y agoWith computer and AI search, the obvious fix is to cite everything related to the new work, up to thousands of citations.
- attractivechaos 3y agoThis is also why I think paid review won't work although many in academia advocate that. If paper mill remains a problem now, review mill will be much worse when journals start to pay reviewers.
- jampekka 3y agoDo many in academia advocate that? I agree that paying for reviewing would make it even worse. Money in general is a bad incentive if honesty and rigor are wanted.
- attractivechaos 3y ago> Do many in academia advocate that? That was my impression whenever paid review was mentioned. Nonetheless, it is also possible that more who dislike the idea were not voicing themselves.
- amadeuspagel 3y agoGoogle has a better university ranking[1], with more experience in withstanding this kind of manipulation and no need to conform to expectations that Ivy League universities must be at the top. [1]: https://www.google.com/search?q=university https://www.google.com/search?q=university
- russfink 3y agoIs the reward of doing this worth the risk of being blacklisted by the metrics keepers?
- nsagent 3y agoIt's not just cartels, but also self-citations. For example, the original preprint of this paper [1] has 5 self-citations, with the version accepted to EMNLP increasing that to 10. This kind of behavior is sadly pretty common now and when you mention it to conference organizers they just shrug. [1]: https://arxiv.org/abs/2304.02210 https://arxiv.org/abs/2304.02210
- kzz102 3y agoSpeaking as a researcher, I think the whole citation business was invented by and consumed by administrators. Researchers don't need citation numbers to discover new works and evaluate people. They are perfectly fine asking their circle of friends, using journal quality and author fame as a crude proxy, and (most importantly) just read the damn papers and reach their own conclusions. Administrators needed some objective metric that allow them to make decisions themselves without relying on the personal opinions of researchers, whom they don't trust. This is magnified by rankings that like these metrics for the same reason, publishers trying to promote journals, and database access sellers. Unfortunately, this has now circled back to the researchers, because the administrators are making hiring / promoting decisions based on these metrics, and people started to game the system. However, I think in many circles, there is a healthy culture of rejecting this nonsense.
- zelphirkalt 3y agoI find it kind of ridiculous and depressing, that we are playing these kind of games in academics and risk careers of bright mind and scientific progress. What a waste of potential and time.
- rcpt 3y agoI left academia a long time ago and now I buy ads for one of my old papers for fun. I still get citations from time to time which is nice.
- cherryteastain 3y agoI have seen all sorts of crooked behaviour in academia, such as reviewers demanding authors to cite their unrelated papers. However, this article's main premise seems to be predicated on another assumption that is also terrible for academia - elitism. Specifically, this quote > There were people that published in journals that no serious mathematician reads, whose work was cited by articles that no serious mathematicians would read, coming from institutions that nobody knows in mathematics is really bad. It suggests one can be a 'serious' mathematician only if you went to XYZ universities and published in ABC journals. This elitism is the main reason why people try to game the system in the first place, because they know that they can't advance in their careers unless they build that perception of prestige. The single minded focus on prestige is then what smothers the real science.