8 ms·
Scientific sleuths spot dishonest ChatGPT use in papers
- naillo 3y agoScientific sleuths spot dishonest Spellcheck use in papers
- fhd2 3y agoApples and oranges: The authors in question obviously missed ChatGPT generated nonsense ("Regenerate"), who's to say they didn't miss anything else? A spell checker helps you _not_ miss inconsequential mistakes. ChatGPT makes it harder to miss subtle yet consequential mistakes.
- Zambyte 3y agoSpellcheck does not assert information as factual unless the author of the text being checked made that assertion already. The same does not hold true for GPT.
- ghaff 3y agoThe examples here seem to go pretty far beyond spellchecking though. I guess I understand the reasons for the disclosure requirement but if you're really just using LLMs to help you more quickly write some background text that you could otherwise do yourself, 1.) It's hard to see the harm beyond the tools that are in every word processor and 2.) I probably just wouldn't use the LLM at that point.
- pseudo_meta 3y ago> It’s not the only case of a ChatGPT-assisted manuscript slipping into a peer-reviewed journal undeclared. Since April, Cabanac has flagged more than a dozen journal articles that contain the telltale ChatGPT phrases ‘Regenerate response’ or ‘As an AI language model, I …’ Not sure what is more astonishing here: The fact that copypaste makes it into a peer reviewed journal, or that the use of ctrl+f is being called "detective work"
- aio2 3y agoIt's both of em, you can't top that off.
- slfnflctd 3y agoCtrl+F has been an important part of detective work since shortly after Ctrl+F was invented by Larry Tesler.
- vorpalhex 3y agoIf the technique works... Excel has caught lots of financial crimes. No reason to hate on old text search.
- flangola7 3y agoWhy is regenerate response in the text? Clicking that button runs it again, it doesn't add to the text.
- nvm0n2 3y agoThey tried to copy/paste by dragging, and in browsers, everything is selectable by default.
- SonOfLilit 3y agoThe big news here are that there are journals in the Springer and Elsevier network that do not read the papers they publish whatsoever. For this service of copy-pasting text into a digital PDF they charge their crazy premium. This is amazing. In a sane world this would finally destroy their reputations so we could replace them with a reasonable solution for evaluating researchers.
- mathisfun123 3y ago> The big news here are that there are journals in the Springer and Elsevier network that do not read the papers they publish whatsoever. This is only news to people outside of academia. Anyone who has ever submitted a paper has gotten comments that can be attributed to the reviewer either being braindead or the reviewer not actually having read the paper. Since it's safe to assume said reviewer actually tied their shoes and put on their underwear the morning they reviewed the paper, it's also safe to assume it's the latter and not the former.
- sigmoid10 3y agoExactly. Journals have sucked for a long time and practically all of academia just went along because publishing lots of stuff is the only way to have a career. The big news is that the rest of the public is starting to catch on how their tax dollars (that were supposed to go to research) end up in publisher's pockets even though they provide hardly any real value.
- mathisfun123 3y ago> Exactly. Journals have sucked for a long time It's funny people are raising their pitch-forks about the journals. I guess it's not widely known/understood that the journals aren't reviewing anything ever. So when I say reviewers I'm talking about PhDs in academia that do this work as "community service". So if peer review is fundamental to the scientific method then your tax dollars are being wasted but the bulk of the waste is on worthless, lazy, BS spouting academics not journal admins. > and practically all of academia just went along because there were no other options. Arxiv has been around for a long time now. Do you really think "the academy" couldn't have come to some consensus on how to allocate review duties for papers that are uploaded/stored/submitted there? The fact that they haven't strongly indicates that lazy ass peer review as orchestrated by the journals is to the benefit of the academy. Journal (and conference) reviewers are completely anonymous and there's absolutely no appeal process (rebuttal phase is BS). Does that sound like a democratic/fair/auditable/meritocracatic setup to you? Sounds like review by inscrutable tribunal to me. So if the process were open/community driven (ie democratic) then people wouldn't be able to hide from the shitty job they do.
- deleted 3y ago[deleted]
- fhd2 3y agoI write a lot of documents for my consulting work - technical due diligence, analyses, that sort of stuff. The kind of stuff where some hallucination I forgot to double check could cause bad business decisions. Until now it didn't occur to me that I should maybe proactively declare that I did _not_ use LLMs whatsoever in any stage of working on these. Does anybody here do that kind of thing? Declaring usage is one thing, but I think declaring non-usage might be the more valuable thing in the end.
- ghaff 3y agoI use LLMs some for background information, parenthetical detail, etc. It's all stuff I know, double-check, and edit as I see fit. It does sometimes save some time but you raise an interesting point. If using LLMs becomes sort of a scarlet letter, I guess I'd just forgo the tool? I'm not going to lie and the benefits aren't really that great.
- fhd2 3y agoI don´t want to come off wrong here: I totally don't mind if people use GenAI to produce text responsibly and carefully. I do find it fair to communicate it however: As a reviewer, I'll look at something entirely written by a human and (even partially) generated by a stochastic model with different eyes. If I trust someone for example, I know they wouldn't knowingly lie to me. If they used an LLM, they might do just that without even noticing. Personally, I love writing - gets me in the flow. I'm weird that way. But I realise lots of people struggle in the early stages of drafting something. If they have a tool that helps them, that's cool. It's just easier to spot mistakes in something someone (or something) else has written than to avoid typing them manually, is all.
- ghaff 3y agoI think if you're going to require disclosure though, most people are either not going to use it or lie because readers will expect the worst even if it was only used to smooth out come wording or stick in a boilerplate explanation of what a particular technology is for.
- pakitan 3y ago> But if researchers delete the boilerplate ChatGPT phrases, the more sophisticated chatbot’s fluent text is “almost impossible” to spot, says Hodgkinson. Well here is an idea - check the actual paper contents, rather than text patterns. If you still can't tell if it's made up by a bot, it's either not a paper worth of publishing or you're not much of a peer reviewer.
- nojs 3y agoHot take: The best thing about LLMs is that they’re going to expose all the areas of written work that were bullshit to begin with. Entire occupations and scientific disciplines will fall, one by one, initially claiming gen AI fraud, and eventually being forced to admit that there’s not any actual difference.
- tomlue 3y agoso strongly agree. Grant writing is another good example of this. So much nonsense writing that just wastes everyone's time.
- I_Am_Nous 3y agoWhat we need is an oracle machine LLM that spits out barely intelligible high-philosophy dialectic so we can have a go at proving that people should just say what they mean without inventing pseudo-languages that only make sense to them.
- jhbadger 3y agoYou can't win that way. Humans like Alan Sokal deliberately wrote nonsense in the 1990s that was mistaken for deep postmodernism, but it didn't really change anything.
- datadrivenangel 3y agoAutomated Humanities!
- jacquesm 3y ago
- KMnO4 3y agoAs if papers aren’t filled with filler BS anyway. This is kinda the perfect use case for LLMs. It’s not like the authors are using them for the actual results. It just helps when you need to turn: “We can probably go deeper” Into “It is postulated that further inquiry may elucidate abstruse intricacies on the topic”
- nvm0n2 3y agoIt's not like the authors are using them for the actual results indeed. After all, that would be dishonest! From the article: Cabanac noticed that some of the equations in the paper didn’t make sense, but the giveaway was above a table: ‘Please note that as an AI language model, I am unable to generate specific tables or conduct tests …’
- Miraltar 3y agoI think there's a distinction to be made between case A where someone uses AI to speed up and enhance the redaction process and case B where someone uses AI to publish complete BS
- bakuninsbart 3y agoIt is also a godsend for non-native speakers. To sound eloquent in scientific papers in English isn't super hard but does require a level of English higher than what is required for most good science. Being able to translate jumbled, clunky, overburdened author-written sentences into a couple of clear and concise sentences serves both the writer and the reader.
- ChrisMarshallNY 3y agoThis is one of those "it depends" areas. For example, if there is a specific advantage to using fluent English text, and the submitters are not fluent in English, then using ChatGPT seems like a fairly natural way to handle that. I'll bet that Google Translate is already being used for it. Not sure that AI is quite up to actually falsifying the research (although I'll bet it will allow fakers to do much more realistic fake data).
- ahmadmijot 3y agoYup. I'm not a native English speaker, thus I use chatgpt (and before this, quillbot or grammarly) to rephrase my sentences to become clearer, concise and have that 'professional' tone, apart from make them grammatically correct.
- ChrisMarshallNY 3y agoEven native English speakers could use it. For example, I am constantly reading prose by extremely intelligent and well-educated people, that is almost painful to read. One of the things that ChatGPT is great for, is generating eminently readable prose. In fact, I'm recommending to my team, that we consider using it to generate some of our help dialogs. In many cases, I wish more folks would use it.
- jprete 3y agoWhatever definition of “falsifying research” you have, AI is up to it. The best illustration is the lawyer earlier this year that assumed ChatGPt was truthful and used nonexistent references with complete text.
- ChrisMarshallNY 3y agoFair point, but we'll see. A lot of folks will get away with it, but some folks will hose it up (like the lawyer, who may end up getting disbarred, I think).
- fardinahsan 3y agoI hope the incoming overabundance of GPT generated papers at least as a side effect gives the scientific community a wakeup call. If publishing becomes a non prestigious enough thing because anyone can GPT can do it, then at least those who do it for reasons other than "publish or peril" will be the ones remaining. Given good enough filtering exists.
- pelasaco 3y agoAs a person using ChatGPT to write university assignments and essays myself, I wonder what is the problem with that? When I get an assignment "write 3000 words about a subject" which actually can be resumed in 500 words, the whole introduction, and sections that are just there as fill-in, can really be speed-up with ChatGPT. Yes, you have to read it, check for plagiarism, add references, but i cannot see it as bad, if the meat, is written by me.
- thaw13579 3y agoCounterpoint, in the context of education, the goal of the assignment isn’t to produce the end product of a 3000 word essay, but to have you go through the associated process of formulating, organizing, and presenting your thoughts. LLMs give you the product, but not that process. Outside of school the product matters more though, so learning to use LLMs as a tool seems like a worthwhile part of education as well, as long as you learn enough about writing to prompt and judge the quality of its outputs.
- ghaff 3y agoYeah. A university paper assignment and a 1000 word article for the company blog seem like two totally different things even barring explicit policies. (Though I'd be surprised if a lot of universities didn't have policies by now.)
- pelasaco 3y agoI understand that, and I go through the process. Even though we know that " the goal of the assignment isn’t to produce the end product of a 3000 word essay", it doesn't change the fact that I have to produce a document with 3000 words that could be written and summarized in 1000. IMO, with such tools, we should maybe question if such "Write an essay with X words" still make sense.. I'm sure the (instructors, master and doctors) that do the review of 3000 word texts would welcome such change.
- skepticATX 3y agoI have been highly critical of using LLMs in places where they shouldn’t be used. With that being said, isn’t writing assistance the one thing we can all agree that they should be used for? I do think that declaring AI assistance is a good thing, but even if an author does not do so I don’t see a huge issue here. Sure, you can have your LLM generate bogus data, but you are willing to do this you probably were unethical before LLMs anyways.
- YearZeroWriter 3y agoBe careful. As a generalist (non-scientific) writer, I've had a few AI flagged text things (sentences and paragraphs). It was only when I showed the editor that my text had not changed since writing the flagged parts in going back to 2014 and 2016 that they said, "hmm, a false positive". This normally happened when I context switched from dialog driven storytelling to long blocks of narrative storytelling. So, people should take each problem as a unique problem and not a systemic problem rampant through the system. Also AI check tools are different with differnt thresholds. To me it feels like AI is part boogeyman too, "What are we going to do with this?! It's clearly our Y2K-borg-Napster-assimilation-with-VHS-and-CDs moment." I will close by saying, if you are not augmenting your life with AI, you're only hurting yourself. I can guarantee you that your competition is doing this, (both as individuals and for their teams using Midjourney, Grammarly, ChatGPT, etc). I should also qualify that I don't take the hardline ethical stance on AI things (clearly). Some people do. I don't know what do with hardliners, it's damn hard to have a conversation these days.
- rawgabbit 3y agoThat is something I have been wondering about LLMs. Do they keep track of large chunks of text, they were trained on? e.g., Does ChatGPT know that "Four scores and seven years" was a speech given by Abraham Lincoln to commemorate the battle of Gettysburg? Or does some articles say ChatGPT just write "plausible" sentences. In other words, the reverse ChatGPT that tries to identify papers written by AI. Is it really sure that the text is plagiarized or is it simply "plausibly" written by AI. That is if something like this goes go to court and the judge asks serious questions... will these AI detectors be laughed at?
- thfuran 3y agoBased on the long history of taking complete nonsense very seriously in forensics, I'd say the odds are pretty good that it'll be taken way too seriously, at least for other AI systems, but perhaps not specifically plagiarism detectors if a large chunk of the jury has personal experience fighting with overzealous anticheat crapware at school.
- 3y ago
- AbrahamParangi 3y agoI can only speak for the fields with which I am familiar but in those fields, peer review is probably a negative indicator. Peer review says that the ideas of the paper are not interesting, meaningful, or true enough to stand on their own. That they require the imprimatur of legitimacy in order to be taken seriously.
- speak_plainly 3y agoIt's way past due that the university system reevaluates their relationship with journals and the pressures to pump out papers. Tying professors and researchers careers to the volume of papers they churn out is silly. The result is the antithesis of quality. Public money is often driving most research but then the results are thrown into an antiquated pay-walled for-profit journal system. The entire peer review system is rife with issues, from like low-level corruption (like the 'blind' reviewer asking for references in exchange for publishing) to no one even reading the paper. When some major technology like ChatGPT comes along it's usually a good time to get back to basics while strengthening the foundation through modernization.
- musha68k 3y agoAlso there can be so much elitism, "old boys networks" and general arbitrariness under veneers of "scientific objectivity".
- derbOac 3y agoI think maybe the problem with academics is that the fundamental paradigm we use to think about scientific contributions is broken. That is, the genius myth, the idea of the lone contributor with the eureka moment that hasn't occurred to anyone else, and so forth and so on. So to stand out, you have to start grasping at all these secondary indicators of skill or contribution or whatever, which leads to everyone doing it, which leads to this runaway process. If you have a bunch of people who don't differ that much, but are incentivized to look like they do, what do you think is going to happen? It's like Goodheart's law applied to an entire swath of indicators for an entire sector of society. It's interesting but I think Goodheart's law sort of implies that the metric being gamed is being gamed because there isn't an alternative? Sometimes that might be because what's trying to be measured is difficult to measure, but one possibility is that it's difficult to measure because it doesn't really exist as it's assumed to be. I'm not trying to say academics are idiots or something like that, if anything the opposite. People in general are just a lot smarter than — or at least don't differ as much — our current dominant schema for scientific contributions recognize.
- mschuster91 3y agoThe fix for that would be to invest serious amounts of money into replication - a problem known at least since the early 2010s [1]. The problem is, replication is expensive, thankless and most importantly rewardless busywork, so it's mostly limited to cases where a successful replication itself promises value like in the LK-99 "superconductor" craze. [1] https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- MagaMuffin 3y ago[dead]
- ta988 3y agoI've seen reviews made with ChatGPT as well, it was dramatically bad, no way it was even proofread.
- RecycledEle 3y agoThe article confuses several things by talking about authors using ChatGPT to help drafted their papers. I can use my friend to help draft a paper in several ways: I can plagiarize from my friend's work, which is unethical. Or, I can quote my friend with a citation. This is ethical. Or, I can ask my friend to look over my paper and - without rewriting any part of it - make suggestions on how to improve it. This does not require a citation and is ethical. Nature seems clueless about the different ways my friend (or ChatGPT) can help me draft a paper.