3 ms·
Unfortunately, it's an open secret that fake or low-effort almost useless papers are very common in every area of scientific research. Typically, it doesn't af
by alsodumb 3y ago
Unfortunately, it's an open secret that fake or low-effort almost useless papers are very common in every area of scientific research.
Typically, it doesn't affect people working in that specific area - they develop/have a sixth sense to detect bullshit papers - it comes with experience but depends on several factors including the authors reputation, their institution (for the first screening), what journal/conference the paper was published in, authors other work, and sometimes things as simple as how much effort was put into the figures, polishing the text, etc. Some of these things are LLM proof, some of them are not - e.g. a senior professor I was talking to, who's been getting like 50-100 emails a week from non-english speaking countries (primarily India, China, Pakistan, and Bangladesh) mentioned that the quality of the text in the emails went up significantly almost overnight after ChatGPT was made open to public. It'll be interesting to see how things change in the next few months/years.
- 2devnull 3y agoRight. If one can develop a sixth sense about bullshit papers then so too can an LLM. If you have a bullshit paper you will need to pipe it through an LLM to debullshittify it so that reviewers cannot tell. The reviewers themselves may need LLM to fight the rising tide of passable bullshit papers. None of that seems productive to me, just throwing gasoline on the dumpster fire of phacking, credential inflation, publish or perish, etc..
- version_five 3y agoRight- academic papers are written for academicians who don't have any issues separating good papers and journals from bad. The fact that many journals have set themselves up as or allowed themselves to become part of the tenure and promotion metrics game, is more of an issue with tenure and promotion. If the requirement for simple metrics dissapeared, the fake papers would go away on their own. In any event, it's not really a problem for researchers.
- alsodumb 3y agoYup, that’s sums up the incentive for publishing so many papers and get citations. Some professor put it in a nice way - the current system motivates us to think of research in terms of LPUs - least publishing units. No matter how established your lab is, you’d try to publish as soon as possible, leading to a lot of papers with not a lot of contribution. If tenure committees and all other systems that gauge academicians require people to say present their only top 3 or 5 seminal papers, then people would try to put their best work out there without the constant pressure of always publishing - win win for everyone. Unfortunately, the ones with the power to make these changes are the ones gaining the most in the current system so it’s unlikely to happen.
- caddemon 3y agoI mean it is a problem for researchers though. The blatantly fake paper mill ones (which seem to be the topic of this article anyway) aren't, but scientific fraud or even just minor misconduct from people that know how to mask it can waste a great deal of grant funding and scientist time to figure out. Like look how many times that 2006 Nature paper on amyloid beta in Alzheimer's was cited, turns out some of the images were completely fabricated.
- Gareth321 3y agoMy favourite example is the grievance studies (https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair). The authors published, among others, portions of directly copied Mein Kampf. In one "study" submitted, they claimed to have observed thousands of hours of dogs having sex in parks, to observe the "patriarchal" linked to "rape culture". The entire thing is a horrible indictment on the level of scrutiny undertaken in the various activist and social "science" oriented journals.
- cauch 3y agoI have no sympathy for social science journals, but when you look at the details of this study, it's way less obvious that the rumor says it is. On the tens of articles they have proposed, the majority was rejected. The article "copied from Mein Kampf" was taking sentence from the book but changing words to create sentences that were scientifically correct (for example: "this social class is bad and we should avoid it" into "stress is bad and we should avoid it"), which means that the article content in itself had no reason to be refused. It's very ironical that this study that was all about "bad science" since then created a totally whimsical rumor on the real situation.
- Fomite 3y agoSimilarly, I read the one about dog parks, and if you approach it with the good faith notion that it wasn't just fabricated whole cloth, it reads like an okay-ish Master's level paper being submitted to an okay-ish journal. And features a decent sample size, which is a rare thing in the veterinary literature.
- deleted 3y ago[deleted]