15 ms·
The fight against fake-paper factories that churn out sham science
- ceejayoz 6y ago> Physicians in China are a particular target market because they typically need to publish research articles to gain promotions, but are so busy at hospitals that they might not have time to do the science, says Chen. Last August, the Beijing municipal health authority published a policy stipulating that an attending physician wanting to be promoted to deputy chief physician must have at least two first-author papers published in professional journals; three first-author papers are required to become a chief physician. These titles affect a physician’s salary and authority, as well as the surgeries they are allowed to perform, says Changqing Li, a former senior physician and gastroenterology researcher at a Chinese hospital who now lives in the United States. "When a measure becomes a target, it ceases to be a good measure." - https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- tomlue 6y agoIt's not just china. Enormous effort is put into achieving high H-Index scores everywhere. I'd love to see a review of all researchers with high H-indexes. I bet you would see a disproportionately high incidence of self citing, citing rings, journal bias, outright corruption and much more. But the bigger problem is that many of these people are highly intelligent and capable. When they are told that their careers depend on a gameable metric, they can figure out clever ways to game it. Mountains of paper don't improve the human condition, we need better success metrics.
- nextos 6y agoThere are professors at top institutions that publish more than a paper per week. Obviously they are not contributing with much more than just stamping their name as last authors. Least to say, this is completely ridiculous.
- Calamity 6y agoHaving the citation's impact on one's H-index decay with author position would cause an interesting stir to the "academic game". I wonder if we would start seeing supervisor's bully for first authorship.
- lumost 6y agoThis would be the inevitable outcome. Currently the first author did the work, and the last author supported in some way (such as by supervising). This is purely by convention.
- sn41 6y agoH-index should go away, it cannot be fixed. Traditionally, academics have been encouraged to publish in prestigious venues. Metrics like H-index do not take this into account.
- ramraj07 6y agoComing full circle much? The h-index was touted as a better metric than judging someone’s paper by the merit of the journal since high profile researchers would get their papers in NSC easily even if it never served a purpose. The h-index is definitely less gameable than you think - the only effective way to game the h-index is to become completely illegitimate in your publishing, using farms like above. That would be easily identified if anyone even vaguely related to the field tries to read the titles and journal names.
- sn41 6y agoThere are moderate tools like multi-round double blind reviews to resolve such issues. A tool which was used to resolve issues among chemists and biologists is now running rampant over fields which do not have a high volume of citations. Mathematics is suffering, for example. An Annals of Math paper might have fewer citations than a paper in Journal of Applied Statistics. But the prestige is incomparable.
- ramraj07 6y agoI am working on a website that will precisely do this! I want to create factors that will show citation rings and self citation ratios! Trying to figure out what to call the metric, my current favourite is CJ factor!
- asciident 6y agoCool of you to do this, though I wonder if this will be more of a measure of community size. Smaller niche topic communities probably behave more like citation rings (you'd cite everyone else in the community frequently). It also seems like it'd penalize people working in a new area (like if you're the first person to use a method, you'd probably cite yourself a bunch).
- ramraj07 6y agoI will try my best to control for field specific practices for sure, the RCR metric published by NIH is already a good starting point in this regard. Valid point about not penalising people in new fields, will keep that in mind!
- deleted 6y ago[deleted]
- eigenket 6y agoTo some extent this already exists http://www.vanityindex.com/ http://www.vanityindex.com/
- atleta 6y agoOne of these guys is (well, probably) Didier Raoult. The person who started the chloroquine craze for the treatment of COVID. He has an h-index of 148 and over 2300 papers.
- alacombe 6y agoWhat does 148 over 2300 papers means exactly ?
- s0rce 6y ago148 is the h-index, which is defined as the maximum value of h such that the author has published h papers that have each been cited at least h times. (from Wikipedia). My h-index is 13.
- alacombe 6y agois 148 high ? low ? What does in intrinsically measure (beside being a competition of who piss further) ?
- andi999 6y agoYes it is high. It means you published at least 148 paper each of it cited by more than 147 times. It is supposed to measure how much interesting/useful work you do.
- sva_ 6y agoSo if you publish a single paper that only gets cited a few times, it'd completely drag you down. That doesn't seem so great honestly.
- saithound 6y agoNo, that wouldn't affect your h-index at all. If you had sixty papers that have each been cited more than sixty times, you'll still have those sixty papers even if you publish a new paper that gets cites only once.
- dguest 6y agoIf you work on a bigger LHC experiment (ATLAS or CMS) you publish about 100 papers a year. I'm on the "author list" for many hundreds of papers that I contributed nothing to and never read. It's actually very difficult to get yourself taken off the author list. I'm sure my H-index is great, but it's completely bogus. Some organizations have stopped counting papers with over some number of authors (i.e. 1000) which is progress, but by that metric I'm the author on zero papers a year.
- unixhero 6y agoGordon? You're late.
- sammorrowdrums 6y agoBeen playing Black Mesa recently, what an awesome remake!
- unixhero 6y agoYeah it is stunning, isn't it.
- contravariant 6y agoWait, so all the papers you were part of had more than 1000 authors?
- dguest 6y agoAlmost every paper I've been a part of. Some of us will do a few independently, but if we want to use LHC data it has to include everyone on the collaboration on the author list.
- helsinkiandrew 6y agoDid a quick search and found the LHC papers, all the ones I looked at have around 1K authors: https://lpcc.web.cern.ch/lhc-data-publications https://lpcc.web.cern.ch/lhc-data-publications
- Fragoel2 6y agoWhile I agree that we need better metrics (and, more in general, a better academic system) I think that publishing fake research (committing fraud) is still way more severe than committing H-index hacking. The latter mostly translates into publishing lots of papers that present small advancements and does encourage scientists to work on topics where there is very little risk of failing (rather than chase challenging topics). These are have long-term negative implications for the systems and for our society as a whole, but, at the end of the day, it's still real research.
- joelthelion 6y ago> Mountains of paper don't improve the human condition, we need better success metrics. I'd argue the problem is the concept of "metric" in itself. You can't measure complex human endeavors with a simple objective number. Whenever the concept pops up (lines of code, anyone?), it's always a disaster.
- emteycz 6y agoPerhaps we should make their actual science the metric...
- sgt101 6y agoAlso luck (says someone with a pathetic H index!) But - I don't care about it, so I can make the observation that of my peers in my 20s the ones that went on to get a massive H-index lucked out with an early paper that became widely cited for a random reason. They introduced a definition, repurposed something that they had done into a fashionable domain, or "got adopted" into a paper by a famous professor and got lots of profile for it. Once they got profile subsequent papers attracted many citations and a virtuous cycle kicked in. Of course all these people are mega smart and do fab research - but they got lucky, and there were lots of others who were as smart and got pushed out.
- cycomanic 6y agoActually the problem is that funding agencies and universities started to implement success metrics. I mean you are trying to measure something which is inherently difficult to impossible to measure (what is good science). Moreover, you essentially make the career of some very intelligent people depend on this arbitrary metric that you have created, what do you expect to happen? Obviously they will work to that metric, which then people complain about they are gaming the system. No, they actually work toward the metric that you (not you personally obviously) have created.
- randomsearch 6y agoWhy do we need metrics at all? If I’m sat in Einstein’s office doing a performance review with him, I’d probably say “Yes, Alf, you’ve done pretty well these last few years. Keep it up.” Don’t think I’d need a metric. And if he came to me for funding for a postdoc I’d say - yes, sure. I don’t know when we decided that _management_ had to be replaced with _metrics_ but it doesn’t seem like a good idea. Management involves doing things that don’t scale, that’s why we have lots of managers.
- Tenoke 6y agoProbably because it's much easier judging whether to fund Einstein than whether to fund someone much further down the chain.
- randomsearch 6y agoI don't see how that is relevant, but let's consider someone further down the chain. "Ah yes, Jimmy Postdoc, I see you have published 3 papers with an average impact factor of 3.1. Have a promotion." vs "Ah yes, Jimmy Postdoc, I see that you're making progress in improving quantum error correction, as evidenced by the fact that we can now use 80% of the previously required qubits to complete Shor's factoring assuming a surface code - great work, you should get that paper out at some point but keep focused on the work for now. I'm also really pleased with your contribution to the academic community in the dept, particularly helping out Polly PhD student with her SAT formulation of decoding. The constructive questions in her talk really opened up a new line of enquiry. Great job. Given all the above, have a promotion." Metrics are stupid, people are smart, stop using stupid.
- watwut 6y agoThe subjective valuation lead to quite a lot of nepotism and unfairness. There is techno-babble you can use to pick pretty much anyone if you are good enough with words.
- randomsearch 6y ago
- rscho 6y agoYes, and we can generalize this statement to physicians worldwide. This is absolutely not a chinese-specific problem.
- christophilus 6y agoIncentives are broken just about anywhere in the world that you look. You’d think that of all fields, the scientific ones would understand the power of incentives and solve for that. But... I guess the incentives aren’t set up for it!
- Engineering-MD 6y agoRequiring first author papers for physicians is not unusual. In the UK surgeons are required to have published several as a requirement to finish training [0]. Often in the uk this makes a phd or other higher degree necessary to completing training. [0]https://publishing.rcseng.ac.uk/doi/pdfplus/10.1308/rcsbull.TB2020.11?download=true https://publishing.rcseng.ac.uk/doi/pdfplus/10.1308/rcsbull....
- iujjkfjdkkdkf 6y agoTo some extent, if, as a journal, you have to start policing submissions in the ways described, you have already lost. In that so much of the journal industry is not about sharing knowledge, but about generating a metric (publication count) for profit. Rather than create more hoops that are only going to push good science away to other venues, they should reconsider what they are trying to accomplish overall, and where appropriate, go out of business. I feel like this is less a problem of serious science (where it's generally known what authors and journals are reliable, and where people using the results of others know enough to subject them to scruitiny). It is a problem of the publishing industry and of evaluating academics for tenure and promotion.
- caddemon 6y agoI think even serious science can be problematic, at least to the extent that "generally known" means wildly different things across different subfields. During graduate rotations I encountered a few instances where something was known to be very hard to replicate, but it was basically impossible to find that information publicly. So if you didn't happen to "network" with labs that were aware of this (which could occur for a variety of reasons besides just being early career), you could end up wasting a lot of time trying to build on work that is at best very finicky, if not outright wrong. Trusting only the work of specific authors may not be a bad strategy, albeit quite conservative. But trusting work because of the journal or the university it came from is way more likely to yield misses IMO. Not saying there isn't a correlation, just that there's still plenty of bad science going on at supposedly top institutions. And I've lost count of how many times a supposedly big result published in Nature or Science just quietly drops off the face of the earth. No rebuttal or retraction, but 10 years go by and nobody in another lab does any real followup on a seemingly cool result? I find that quite sketchy. Biology papers are especially bad at not releasing raw data or detailed methodology, so in some cases it's not possible for even a very educated person to evaluate the quality of the work from reading the paper. There's definitely some trust that goes into it, probably too much given how oversubscribed/hypercompetitive that field is.
- erostrate 6y agoIs there a good analysis somewhere of pragmatism in Chinese culture? It seems that there is a particular brand of ultra-pragmatism and rule-gaming that is overrepresented in China. Researchers are judged on published papers? Publish papers that appear just good enough to be published, ignore everything else. Products are bought based on looks on price? Make a cheap product that appears just good enough to be sold, ignore everything else. I don't mean to stereotype all of China negatively. I've worked with enough amazing Chinese colleagues to understand this is not all of China. I'm just curious if this pattern has some truth to it or if it's just a random racist stereotype, and how it can be explained.
- chrischen 6y agoThere are a lot of people in China and everyone's place in society was recently (relatively) reset. Rapidly developing and everyone scrambling to get on top of one another. There's bound to be rough edges in the developing of a society basically from scratch. While China is an ancient country, the rules, regulations, institutions, people in power, basically all reset last century. Also what you described are characteristics of any poorer country. They'll try to get some money from wealthier places in the easiest way possible (Indian IT scams, Nigerian prince scams, etc). For most of the last century and even until today China is still a lot poorer than the US. It's also a massive country. It's easy to discount the 1.4 billion Chinese population because a lot of them are dirt poor still.
- rrmm 6y agoI've seen some good summaries in the past, but unfortunately can't find them at the moment. Some things that play into it are the damage from the Cultural Revolution/Great Leap(s) Foward, government corruption with many rules designed to move money up to party leaders, extreme hardship and poverty, as well as recent affluence. As with anything, it's complicated, and I don't understand things well enough to really synthesize it any better.
- geraneum 6y agoIt’s possible that such a phenomenon (existence of paper-mills) could be caused by systemic flaws in policy making (in health care system in this case). As it’s mentioned in the article, such strategies are observed to be adopted by some in Iran and Russia. IMO attributing it to the “culture” of the people in those counties based on the facts presented in the article is far-fetched.
- llsf 6y agoLove how Dr. Bik exposed @IHU_Marseille and Dr. Raoult (aka Dr. Photoshop) especially. Those doctored images in the Dr. Raoult papers are comically bad. https://twitter.com/MicrobiomDigest/status/1374046924551032835 https://twitter.com/MicrobiomDigest/status/13740469245510328... We hear about some Chinese examples, but this happens also in Europe.
- jkingsman 6y agoDr. Bik is tireless, and her knack for spotting duplicated/reused images and image segments is incredible -- I often have to stare even at her marked/highlighted images to see it, but it's indeed there.
- the_af 6y ago> We hear about some Chinese examples, but this happens also in Europe. Indeed, fake paper factories are a thing in Western democracies as well. In Latin America, for example.
- woeirua 6y agoThe current academic publishing system is so broken that I can’t see it surviving for much longer. In addition to outright fraud, journals are chock full of low quality papers and papers that cannot be reproduced. Citations are a useless metric because of how circular the citation networks are. The best way forward is probably some metric of reproducibility. Can your paper/experiment be reproduced? Has anyone done so? Did they succeed or fail? Did they publish their results? That would quickly separate the wheat from the chaff.
- Calamity 6y agoThis would indeed be great, but the impact on one's career of reproducing another's work would have to be equivalent to that of the original publication on the original author's career. Otherwise, nobody will want to spend the time certifying another's work. Could have this so that it's a multi-stage process where all involved parties become authors on the paper, but this would need a substantial shift in what academia is today.
- woeirua 6y agoThat’s exactly what it should be. After author there should be multiple lines for individuals/groups that reproduced that research. Prizes should be divvied amongst original researchers and the first N groups to reproduce it. Also, I’d love to see the Masters degree become focused on reproducing others work for two years. You do original research in a PhD only.
- fma 6y agoOr...a scientist's reputation can also be bolstered by how many papers they are able to reproduce, or debunk. Debunking science also brings value to science. The incentivizes of the system needs to change. As an analogy, performing code review brings value to software development.
- asciident 6y agoPeople have been saying this for decades. Both the idea that the academic publishing model is about to die (due to too many papers, low quality, broken peer review, etc.), and that we need to focus on reproducibility. These ideas itself are valid but not nuanced. I could also argue that academic publishing as at its most successful ever with science advancing at an incredible pace even as complexity is so high, and I could argue that everyone is now thinking about reproducibility and it's baked into the academic process these days (for decades, having another lab reproduce your paper is critical to it being accepted by the community even after being accepted for publication). Maybe an analogy I would use it if I were to say "software is so broken, the quality is getting worse over time, new programmers can't fizzbizz, security bugs everywhere, no one can even reliably make a heathcare website or computer game on time." Same argument that you're making.
- LetThereBeLight 6y agoThe current publication system is very odd. Mostly government money is being used to fund research, then professors are asked to review a manuscript (who may just hand it off to a grad student or postdoc) without any compensation for their time, and the final result then locked behind a paywall of some private journal. It seems that the incentive of a journal is to maintain an appearance of legitimacy, rather than actually enforcing it. This is why, as the article mentions, journals tend to be fairly quiet about retractions and issues of misconduct. There is a better incentive by those that are funding the research (the taxpayers/government) that their investments are resulting in legitimate works. This also goes in line with the idea that these final manuscripts should be freely available to the public. Now with that said, I also acknowledge that the idea of having the NIH, NSF. etc. operate the editorial and review process would be nightmarish.
- IG_Semmelweiss 6y agoI think you are on point. I also wonder what the system would look like if you suddenly took out govt money out of the university system. I suspect there would be a lot less research, but perhaps whatever papers made it to the finish line, they would be much more meaningful ?
- Calamity 6y agoI think you would just get a lot less research overall. It might lead to a better fraction of research being of higher quality, but in absolute terms, it would be a drastic cut. R&D is always going to be a sunk cost, especially when exploring "new frontiers". Funding people to "waste their time" going down different uncharted paths - with most of them leading to dead-ends - is still the only way to have a few return with new insight from those paths that led to new and wondrous areas. You can already see this at the scale of "what" government chooses to fund. For years they have funded cancer research which has made many advances, but they did not fund "ageing" research anywhere close to as much. Recent advances in the latter have shown this to be a promising field, but imagine the advances we could have had had this been funded to the same extent that cancer research has!
- qchris 6y agoWhenever something like this comes up, I think that a closely-related topic is replication of scientific studies. One of the best articles I've read on this was by Stuart Buck here [1], which I think offers a great starting point. I also think it's really relevant to this particular discussion; the added emphasis on attempted replication probably wouldn't prevent paper mills from working period, but they could drastically increase the cost of good forgeries and create an environment of accountability that could potentially make their use by researchers much more risky, which might drop their use even further. Of course, publishers could help as well; maybe only the most innovative papers that have been replicated by unaffiliated third parties that have been provided with the pre-print or low-impact initial publication get re-published in the higher-impact journals, and those that can't get watermarked with with "unable to be reproduced x of y times." [1] https://worksinprogress.co/issue/escaping-sciences-paradox/ https://worksinprogress.co/issue/escaping-sciences-paradox/
- throwawaysea 6y agoThis is an article focused on Chinese paper mills, but let’s not forget bad research happens in American too and doesn’t require paper mills. That was revealed in the grievance studies scandal: https://areomagazine.com/2018/10/02/academic-grievance-studies-and-the-corruption-of-scholarship/ https://areomagazine.com/2018/10/02/academic-grievance-studi... Joe Rogan interviewed two of the people involved in this project: https://youtu.be/OlqU_JMTzd4 https://youtu.be/OlqU_JMTzd4
- chrisjharris 6y agoBit different for medical studies, though, isn't it?
- nabla9 6y agoNature's own publication: Scientific Reports is also suspect. https://www.nature.com/srep/ https://www.nature.com/srep/ More about Scientific Reports controversies: https://en.wikipedia.org/wiki/Scientific_Reports https://en.wikipedia.org/wiki/Scientific_Reports
- caddemon 6y agoInteresting that the journal claims to judge only on scientific process and not impact, which sounds like a good idea for publishing negative results and replications and such, but then they wind up publishing some really outrageous claims like a paper supporting homeopathy. Seems like the opposite of the stated goal, but I guess they can make more money by publishing straight up bad science instead of staying true to their mission statement. That controversies page was surprisingly entertaining overall though, I have to say. Clearly the authors don't even take the journal seriously: "The face of Donald Trump was hidden in an image of baboon feces in a paper published in 2018. The journal later removed the image."
- nabla9 6y agoScientific Reports is very common source of HN posts. People see only the domain name nature.com and think it's quality source.
- hajuna 6y agoEvery student knows that some papers are a scam. But life is about getting to a position of certain security.
- euske 6y agoI think that a part of the reason is China's hyper-competitiveness, because the country has simply too many people. But then I guess that the more or less same thing would happen in India too? That country has also too many people, and from what I've seen their job market is pretty competitive. I'd be curious to see if these two country appear similarly in the academic landscape.
- PragmaticPulp 6y agoMany of these fake papers are a major boon to the supplement industry. Obscure supplements or herbal medicines are a common target for these papers. The more obscure the topic, the less likely the authors are to encounter conflicting results. So you end up with streams of papers showing that various herbal remedies or supplements show efficacy against COVID or cancer or other hot topic issues. Supplement producers love this because it creates demand for their products. In some cases, I've been shocked to discover that the same researchers who published these papers are now marketing their own proprietary blend of those supplements. They create both the supply and the demand.
- CGamesPlay 6y ago> In some cases, I've been shocked to discover that the same researchers who published these papers are now marketing their own proprietary blend of those supplements. They create both the supply and the demand. Well, fakes aside, legitimate pharmaceutical researchers should also be expected to do this, right?
- dodobirdlord 6y agoNot really, no. Once you have a compound that looks promising you then need to spend huge sums of your own money running a battery of human trials compelling enough to convince local regulators that your compound actually works and isn't incidentally harmful. Doing shoddy initial studies is just shooting yourself in the foot. Supplements are their own special area because they generally aren't regulated as pharmaceuticals.
- dukeofdoom 6y agoThere's a lot of fakery in the world. Perhaps its because we want to pretend we are better than we actually are. You can get fake teeth, fake hair, fake boobs, fake tan, fake muscles, support fake charities that take 98% for themselves. Read fake news and vote for fake politicians that will tell you sweet lies to make your fake self feel better. Fakery is only growing more popular.
- injidup 6y agoThere are companies in Europe that produce suspicious looking papers backing products in the alternative medicine industry. "Darsch Scientific" comes to mind. They have done papers supporting Menon and Powerinsole. The papers published include graphs and pictures of cell cultures and conclude amazing results but methodology described in the papers seem to be less than robust for products claiming miraculous properties. https://www.dartsch-scientific.com/ https://www.dartsch-scientific.com/ https://www.powerinsole.com/science-en/?lang=en https://www.powerinsole.com/science-en/?lang=en https://www.memon.eu/blog/schutz-hochfrequente-strahlung-elektrosmog-zellbiologisch-wirksam/ https://www.memon.eu/blog/schutz-hochfrequente-strahlung-ele...
- Fomite 6y agoThe author emphasizes China, but the massive incentive to publish papers based on volume with little/no consideration for reputation of the journal in question, is prevalent in a number of different national systems. China just has a disproportionate impact because of the size. It's not unique, and the comments talking about this being some aspect of China or the character of the Chinese are, I think, missing the mark. This is just humans responding to clear incentives.
- TeeMassive 6y ago> China just has a disproportionate impact because of the size. Hugh no, fake papers and total disregard to ethics in general has always been a huge problem in China. Remember, China is still a poor contrary to what they claim, and its academic population is therefore small in comparison the rest of the developed world.
- Fomite 6y agoI'm not necessarily talking about developed world systems. I encounter these same incentives when working with researchers in Africa and other parts of Asia. But many commenters are making this inherently Chinese: "Is there a good analysis somewhere of pragmatism in Chinese culture?" "I lived in China for 6 months. This fits my experience, and is remarkably similar to a story I was told by a native Chinese: 'We were taught in school about Mao, who rose to power on idealism. At first he showed us many great things that we could do, but eventually idealism led us to the great famine, and many people starved. Since then, Mao lost some power, and now in school they teach us that Mao was 2/3 right, but 1/3 wrong. And that 1/3 that was wrong was idealism. So now we practice pragmatism.'" "In China, the concept of shame [in contrast to guilt] is widely accepted due to Confucian teachings" "ADV China talk about this a lot. China is basically a low trust society where amoral opportunism is not only tolerated but also expected, as long its 'chabuduo' (just good enough)." etc. etc. It's just as silly to talk about that as it would be to talk about the inherent pragmatism of the Tanzanians. It is, again, just people responding to very clear incentives, and the market rising to meet them.
- argvargc 6y agoI wonder if the different issues we'd face would be less obfuscating, if science was authored anonymously. In principle, why should we care who makes a particular claim? The claim should be decided on its own merits.
- gspr 6y agoPapers aren't published under real names for the sake of the papers or to increase the acceptance of the claims in them. They're published under real names for the sake of the authors writing them. "See, I did that!" It would be hard to evaluate a researcher's abilities if one couldn't read her papers.
- argvargc 6y agoThis is exactly my point. When assessing research, the evaluation of a researcher should always be far, far secondary to the evaluation of the research itself. In the contrary case, which is a separate issue - that of hiring a scientist, in every other arena, employees are hired based on interviews and testing, with some consideration given to past experience - but this is more and more, especially on the cutting edge, the least pertinent concern. In science however, employees are hired sometimes entirely based upon their "experience", with little to no consideration given to their presently-assessable ability. If I were to hire a research scientist, I would far prefer to test potential suitors by having them walk through the conducting of research in front of me. Many, however, would balk - indignant, and stand proud on their legacy of papers instead. This gives me no idea of their actual ability, and taken on face-value - ignoring tradition and dogma - is actually a red-flag.
- rswskg 6y agoand they didn't even get started on social sciences..