10 ms·

Signs of undeclared ChatGPT use in papers mounting

103 points by williamsmj 3y ago

johngossman 3y ago
This is going to be like Photoshop usage. Very quickly it will be expected that papers use AI to write parts, just as fixing white balance in Photoshop doesn’t require alerting the viewer that the photo has been enhanced. More extensive use may need to be acknowledged. But I’m not sure…design tools are rarely credited except in cases like a Fx heavy movie
- RugnirViking 3y ago
  Absolubtely. Im involved in writing right now and i've caught people using its output. It's hard to make an outright ban, heck ive used it myself for things like tricky LaTeX formatting (make this stuff into a nice table with color boxes corresponding to the rgb values). But as with all papers the issue is just a lack of dilligence, be it people straight up using hallucinated sources or the paper bizzarely fawning over every aspect of the project, even things that are objectively rubbish and author knows that when questioned (such as a paragraph praising the high quality of servo motors used when they are blamed for test failure in a later section). Someone taking the time to use it well will use it well, someone lazy will still be lazy.
  - siftrics 3y ago
    > Someone taking the time to use it well will use it well, someone lazy will still be lazy. Exactly. It's a self-solving problem as long as draconian luddites don't outright ban LLMs.
- improv 3y ago
  Reminds me of when teachers told students not to use Wikipedia for research purposes.
  - siftrics 3y ago
    I clearly remember getting in trouble in class each year for using Wikipedia. My podunk teachers couldn't fathom the difference between directly citing Wikipedia (just cite the source it references instead) and merely reading it. Pain.
    - thomasahle 3y ago
      > just cite the source it references instead You also shouldn't cite sources you haven't read yourself.
      - siftrics 3y ago
        I agree?
    - hotnfresh 3y ago
      Some teachers notice if you only cite sources on the Wikipedia page, too. More likely if the whole class is writing on the same topic. The work-around is to write it from Wikipedia, but find sources by searching on Google Books. Being able to read a couple pages around each part you’re citing is enough, so their page allowance for restricted works is usually plenty. Hit your citation count and real-book-source requirement (if they still do that second one) without leaving your chair. Barely more work than writing from Wikipedia and not covering your tracks.
  - ProllyInfamous 3y ago
    My wikipedia account is over two decades old, and I often bring this up when talking bee -keeping stuff... them both being "bottom up organization that results in a beautiful exchange of information." This experience "went full circle" for me, January 2023... when ChatGPT cited SOMETHING I HAD WRITTEN into wikipedia (re: transistor density).
- jprete 3y ago
  This is a more serious problem than that, because AI does in fact make stuff up, and will bullshit logical connections that make superficial sense but don't match the system under discussion. I think a better analogy is airbrushing photos to remove people; it changes the facts of the photo in a way people care about. Likewise, it's a problem if people sloppily use GenAI to fix up text or graphs or whatever, and it creates things that match the paper really well but don't match reality. Those are _exactly_ the kinds of problems that will slip through peer review very easily. Fixing white balance, or a thousand other old-school photo changes, is very different; the underlying structure of the photo is there and nobody cares about white balance or most of those old corrections, because our eyes and brains are constantly mucking with them anyway. People care about airbrushing or GenAI-"airbrushing" parts of the picture, because the information content we care about is changed.
  - hn_throwaway_99 3y ago
    > I think a better analogy is airbrushing photos to remove people; it changes the facts of the photo in a way people care about. I was thinking about this a lot recently with the announcement of the Pixel 8. One of the big announced features was "Magic Editor", where you could edit all your photos to automatically put smiles on people who were frowning, remove people altogether like that inconvenient homeless person or garbage truck: https://blog.google/products/pixel/google-pixel-8-pro/ https://blog.google/products/pixel/google-pixel-8-pro/ This may be a bit "get off my lawn", but honestly I was disgusted. Why do people feel their everyday family photos need to match some Insta-bullshit ideal? Some of my favorite old family pics are where something "wrong" or unexpected happens. I don't fault Google for this, and I think it's natural to want to touch up photos when you're taking an important picture. But I feel like we're careening towards a future where we're starting to forget what reality actually is. When TikTok came out with their extremely realistic "makeup filters", I remember seeing a video where a photographer commented that she would take pictures of women who were shocked at how "ugly" they were because they weren't used to seeing "un-Facetuned" photos of themselves. Sorry for the semi-rant, just feels like a situation where any individual, small technological improvement makes sense, but in totality we end up at a place that is overall worse than where we started.
- sxg 3y ago
  I don't think authors need to credit AI in their written works, but I do think this is the double-edged sword of claiming partly AI-generated content as your own (for reference, the WGA recently won the right to claim AI-inspired and partially AI-created work as a human writer's own work without having to credit the AI). I think it's fair for a scientific human author to take full credit for papers that are partially written by AI, but any logical errors and hallucinations inserted by AI should also be attributed to the human author. It's the human author's responsibility to ensure the final written work is accurate. Edit: for further context, I have published a few scientific papers in recent years, and using AI is not all that different from a senior author relying on junior authors. Early on in my career, I created the first draft of most papers, and my more senior co-authors would verify my work and create direct edits or suggestions for me to revise the draft. The final paper was created after a few rounds of this cycle.
- registeredcorn 3y ago
  I don't mean to be super nitpicky, but I've got to strongly disagree with your comparison to photoshop / photography. There's been an ongoing debate in the Photography community going back as far as photographic editing has existed, regarding when and how acknowledgement of editing has been performed, whether it be literal, physical cutting and pasting of printed media, or a digital equivalent. Some of the more famous personalities in Photography seem to lean to something like this: "Minor adjustments like removing dust from the lens, adjusting colors to match the 'true' colors as it was taken (or to make it 'more visually pleasing'), cropping and level correction are fine and expected. The use of things like Sky Replacement [1] or content-aware fills [2] should be attributed or referenced somewhere if they are the focal point of the photograph." There are others who do much more "artistic" styles of photography [3] where they argue something like: "No person in their right mind would believe that there isn't some amount of editing going into nearly photograph, so why mention it at all?" or "Because editing has been so heavily used, by so many famous photographers for so long, without mentioning editing, it's fine if people don't mention the use of digital (or physical) editing." or "Popular services like Instagram have built-in filters that users can use to make themselves look different - should the people who post photos that have those filters on also be pressured into mentioning that their photo was edited? If not, what's the difference?" There's really not any solid consensus on what is "correct" when it comes to editing attribution. Most tend to agree that doing the bulk of the work "in-camera" (cropping by moving in closer, for example) will make your life easier later on, but, there are some outliers that strongly advocate that any form of editing in post, is "not photography". Things get even weirder when you get into print. The same photograph printed on different mediums of paper, or with different printers, or different ink, or different DPI settings can come out looking noticeably different even to laymen (imagine a smeared blotchy photo versus a crisp, clear one - there are times where you might want the photo to look somewhat blotchy for aesthetic reasons). In a way, choosing one form of medium over another is a kind of editing, because the end presentation is measurably different from the original. A quick example might be noticing the difference between modern movies that are shot on digital vs traditional film. There is a kind of "feel" to the end product, because of the medium that was used to record and distribute it. TL;DR The photography community is very much not in agreement to what amount of editing is, or is not permissible, or when it should, or should not be attributed or brought up, or even how that information should be conveyed. Some are fine without any mention whatsoever because of historical reasons. Others want it mentioned under the use of certain tools or criteria. Still other don't consider a photographer to be a "real" photographer at all, and instead should be called a "stills editor" (or something like that) if they use any editing at all, outside of "in-camera work". [1] https://helpx.adobe.com/photoshop/using/replace-sky.html https://helpx.adobe.com/photoshop/using/replace-sky.html [2] https://helpx.adobe.com/photoshop/using/content-aware-fill.html https://helpx.adobe.com/photoshop/using/content-aware-fill.h... [3] https://www.lookslikefilm.com/2019/08/15/artistic-photos/ https://www.lookslikefilm.com/2019/08/15/artistic-photos/
- johngossman 3y ago
  And while we're on the topic (Getty Images created a generative AI that gives credit to the source images) https://news.ycombinator.com/item?id=37793579 https://news.ycombinator.com/item?id=37793579
NoMoreNicksLeft 3y ago
The hilarious part isn't that a human didn't write the paper. Instead, consider that in some obscure environmentalism journal, a paper might only ever be read by a few dozen people around the world anyway. Even a good paper on some groundbreaking study might have very narrow interest after all. But we're fast approaching an era where only the LLMs will read the thing, and only to spit out a summary of it. No human wrote it, no humans will read it. The biggest journals will of course avoid this fate the longest, maybe even putting it off indefinitely. And some of the most useless journals have been doing worse for a long time. At some point though, doesn't academia have to wake up and realize how pointless all of this is?
- alchemist1e9 3y ago
  I was thinking something a bit similar about news in the future. I’m already thinking I should try to put together an automated daily briefing for myself using a list of sources and ask LLMs to write a summary briefing. But then also consider in the future the news sources will also likely be generating some of the articles with LLMs. This LLMs reading LLM output is a curious development. Same for emails also btw. I already have an LLM redraft most of my emails as it makes them more readable and concise.
  - daft_pink 3y ago
    Arc browser is already summarizing things this way.
- barrysteve 3y ago
  It's darkening the public usage, the private channels for research and communication will become disproportionately more valuable. It's odd not to see Google or Meta try and make a copyright strike system for written text. Copy and paste LLMs are driving people away from google's ad bucks, filling FB with junk. I guess 'the web' is no longer worth defending, de facto.
  - jprete 3y ago
    It's possible that the open Web simply cannot be defended. The advantage is generally on the attacking side rather than the defending side, after all.
shadowgovt 3y ago
It's easy to forget how many people don't actually have English as a first language. With so many journals demanding papers submitted in a language not native to the author, use of assistive technology to convert the paper so the underlying information can be made known should be expected.
- the_af 3y ago
  If it's just a better Google Translate, then I don't see the problem. If it's used to actually write the first version because people can't write or sort out their ideas... I dunno. Is there any evidence that it's primarily used by non-native English speakers for this purpose, though? (edit: just saw the list of authors, probably this is the case, at least in this context).
  - shadowgovt 3y ago
    > If it's used to actually write the first version because people can't write or sort out their ideas... I dunno. For what it's worth, I'm also 10,000% on board with that use case. I have definitely stared at a blank piece of paper with writer's block because I couldn't write the introduction for a paper before. If ChatGPT gets somebody past that block, more power to it. Every paper is required to start with some novel variant of "from the dawn of time humanity has wondered if..." when all you want to say is "We discovered some stuff, here it is." I am super in favor of automating the step that nobody wants to write, nobody wants to read, and no journal will accept without its presence. Reporting novel scientific findings was never supposed to be an exercise in fluent English authorship mastery.
    - SoftTalker 3y ago
      I haven't written scientific papers, but for other reports or writeups I always start with the meat of what I want to say. I write the introduction later, once I know what it is that I'm introducing.
    - the_af 3y ago
      > I am super in favor of automating the step that nobody wants to write, nobody wants to read, and no journal will accept without its presence. You make a good point. Maybe the bits that can be automated are the zero-value bits, and therefore the best conclusion would be to get rid of them? In other words, if it can be automated, it's not real content. I'd be wary of people writing summaries or conclusions using ChatGPT though, because ChatGPT can write very convincing crap that completely misses the point. Someone who is careless enough to forget removing the "Regenerate response" part of the output may be as careless reviewing whether the conclusion truly says what they meant it to say. As an aside, I've asked ChatGPT 3.5 to provide interpretations of works of fiction of which I know the most common ones (e.g. "this is a parable about [...]", "this is a warning against authoritarianism", etc), and ChatGPT sometimes comes up with stuff that completely misses the point; but if you aren't familiar with the work (haven't read it except for a summary) you could mistake it for a valid interpretation!
iamben 3y ago
Could this just be a translation? A non native speaker pasting their paper into ChatGPT and asking it to translate it to English, and then copying the results and hoping for the best?
daft_pink 3y ago
I think it's okay to use ChatGPT to write and fix your prose, but not to do your thinking or analysis for you. That's how I use it at work.
- RugnirViking 3y ago
  > to write and fix your prose fix prose, absolubtely. To write? Well a lot of the more egrerious errors ive seen when trying to get it to write academic writing comes from it not having a full context of what the problem is/whats been done/ what you are trying to do. Ive been turning over the idea in my head that for one to fully give it all the required context one might as well just use that context as the piece of writing in the first place. There is no advantage into stretching the relevant information to 6 paragraphs instead of 2. Not to say it should never be used but trying to use it while being thorough and actually reading its output has usually led me to extending context more and more and eventually just writing the section myself.
  - daft_pink 3y ago
    I don't think you should just blindly accept the results. I just write what I want to say stream of consciousness, drop it into chatgpt and then review the result to see where it improved and where it didn't. A prompt such as "Why are all the dolphins dying?" would be unacceptable. A prompt such as rewrite "explain that the dolphins are dying because 30 degree rise in ocean temperaturs made their environments unlivable" that reformats the language into a academic style I feel is totally acceptable and efficient if you read what it spits out and agree and edit from there.
- jprete 3y ago
  I am very sympathetic to people who are using it to write better in language X when their native language is Y. I think most other people should not use it for writing, because the hard part is thinking about (A) what you _really_ want to say (B) what your audience cares about (C) mistaken assumptions by you or your audience, which need correction in the text. And I think people using ChatGPT to write will give it a vague idea of (A) and usually nothing about (B) and (C).
the_af 3y ago
If you look at the papers from the list, some of the authors respond to the comment which asks for clarifications on ChatGPT usage (for those who didn't read TFA, this is detected mostly because they cut & pasted ChatGPT's response without removing extraneous parts like "Regenerate response" from the UI! So not only do they rely on ChatGPT, they also don't know how to use it well). What's interesting is that even when they reply, authors don't seem to respond to the ethics/standards issue of actually using ChatGPT. Well, some do ("I will not do it again" is basically what one of the authors says, which reads like a childish response to me), but many claim it was a "mistake" -- the mistake being... forgetting to remove the boilerplate and "regenerate response" garbage! Uh-oh. I think an acceptable use of AI would be in translation, basically a better Google Translate. But is this what's happening here?
- sonicanatidae 3y ago
  I find it helpful in working with technology. I have been experimenting with generating powershell scripts using chatgpt, and aside from a few syntax errors or localization, the scripts are a great starting point. Instead of writing the whole script. I can have ChatGPT write it, then I just have to clean it up, address any errors and localize.
  - sdenton4 3y ago
    Yeah, exactly this. Paragraphs of condensed scientific text after six rounds of collaborator revisions are typically a mess - ChatGPT is great for suggesting a more readable alternative, and then further crafting into something actually-good. I think a lot of the hand-wringing neglects that there's a range of ways to use LLMs. Text-refactor is obviously perfectly ethical, unless you're gunning for the Butlerian Jihad, no worse than the grammar checker in Word.
    - the_af 3y ago
      > I think a lot of the hand-wringing neglects that there's a range of ways to use LLMs. You and the comment you replied to seem to be making an argument about something neither my own comment nor TFA said. Nobody is claiming LLMs or ChatGPT aren't useful. The argument is about using it to write parts of research papers without disclosing this usage. Nobody mentioned using ChatGPT to write code.
deleted 3y ago
[deleted]
artursapek 3y ago
BRB changing my email signature to "Regenerate response" lol
choeger 3y ago
There's a huge fraudulent scene when it comes to scientific papers, IMO largely driven by the "schoolification" of universities. Many authors seem to consider writing and publishing a research paper as just another form of an exam and thus cheating as the logical approach.
- JohnFen 3y ago
  People consider cheating on exams as a "logical approach"? That might be the most depressing thing I've heard today.
hn8305823 3y ago
It's not so much the undeclared use of ChatGPT that bothers me but the egregious sloppiness of it. It was already obvious that writing scientific papers has evolved into a cargo cult, but this shows that it's much worse than previously thought. Authors, referees, and journals go through the motions of what lead to genuine scientific advancement in the past, but no actual value is being created at any step in the process for that vast majority of papers. Of course most papers will never be ground breaking but at least in the past there was some value being created in the process.
crazygringo 3y ago
It seems totally unclear whether ChatGPT is being used to invent fictional methodology or literature review etc., which is extremely unethical, or whether it's simply being used to check and fix grammar and tone for non-native speakers, which is perfectly fine. But regardless, this is extremely troubling in terms of quality. The fact that "regenerate response" gets left in tells us that even if it's just for grammar correction: 1) The authors aren't even minimally reviewing if the ChatGPT version didn't accidentally change the meaning of the sentences 2) Peer reviewers clearly aren't even reading the paper 3) The journal isn't even reading the paper to introduce a minimal level of quality control ChatGPT seems to be the smallest problem here. And thankfully, it is revealing much more serious problems about quality checks in academia at all. The question is, will anything be done about it? Not about ChatGPT, but about the horrifically irresponsible processes that would let these slip through?
- davely 3y ago
  EDIT: Oh, wow. Lack of coffee this morning. I read that as "fictional mythology", NOT "fictional methodology." Ignore me. :) ----- I want to pull apart one specific thing you mentioned, essentially: > It seems totally unclear whether ChatGPT is being used to invent fictional methodology [...], which is extremely unethical In the context of generating something fictional, why is using ChatGPT unethical? One reason I'm asking: I have always been a big fan of NaNoWriMo[1] (National Novel Writing Month, every November). This year, I'm interested in exploring how to use ChatGPT -- not necessarily to generate a ton of prose, but to essentially help flesh out some plot ideas that I've had. Part of the back and forth I've had with it is coming up with some interesting scenarios for my story that I hadn't really thought about and I'm keen on exploring a bit more. [1] https://en.wikipedia.org/wiki/National_Novel_Writing_Month https://en.wikipedia.org/wiki/National_Novel_Writing_Month
  - mola 3y ago
    Because this fiction is presented as part of a scientific paper....
    - 3y ago
ugjka 3y ago
It just gonna get worse as every commercial editor will get some AI completion crap
cm2012 3y ago
"Signs of undeclared Microsoft Word spellcheck use in papers mounting." Such alarmist tones would sound really silly when applied to other dumb productivity tools. In the end, the quality of the paper is what should be judged.
- Valodim 3y ago
  That is true in principle. But as scientific papers, this is supposed to be extremely precise, highly iterated, and thoroughly reviewed content. If there are very obvious mistakes like a leftover pasted in button text, it calls into question whether the authors (or reviewers) were responsibly verifying the content contributed by a tool that is known to sometimes deliver wildly but subtly incorrect results, and thus the quality of the paper content as a whole.
  - cm2012 3y ago
    This just let's you see which writers are incompetent which is a good thing. People who copy and paste chat GPT without checking would have shit papers regardless.
matthewdgreen 3y ago
Four quick thoughts: 1. Bad grammar in a paper is incredibly irritating, which is particularly unfair for people who are non-native English speakers. If ChatGPT can help people to convert awkward English into nicer work without changing the ideas or confabulating, that seems like a win. I don't see why people need to "declare" their usage of ChatGPT for these purposes. Is that a standard now? 2. The major issue here is that the authors copied a response and accidentally also brought some button text with it. This is given to imply that people aren't reading ChatGPT's output. But I think this is just as likely to be benign. People do all sorts of silly nonsense in papers (including me), like leaving author comments in final drafts. It's bad and sloppy, but it isn't usually a crime. 3. Many papers on public servers are junky. Many journals and conferences are equally junky. Search for any bad string and you'll find it everywhere, and non-scientists will be scandalized. 4. I like the people at Retraction Watch and think they're doing the Lord's work. However sometimes I worry that immersing yourself in wickedness makes you see wickedness everywhere, and also creates an (unconscious) incentive to sensationalize bad stuff. I hope they're aware of this.
- stayfrosty420 3y ago
  how is (2) likely to be benign and not plagiarism? It is completely different from leaving an author comment in a draft...such a weird take.
  - matthewdgreen 3y ago
    If I type a series of bullet points into ChatGPT and ask it to re-phrase them in better English prose, am I plagiarizing? If it's not plagiarism but I screw up at copy/pasting, does it become plagiarism? I don't know! It is not something I personally would ever do because I like my own writing. But I'm also not going to throw that accusation at ESL authors who are using a writing tool (like Autocorrect-on-steroids) to convey original ideas to an audience, at least not until we've had a much longer community discussion on using these tools. ETA I would urge you not to throw the death penalty at people whose crime may be more akin to shoplifting. This is bad both for the shoplifters and for people who want to fight serious crime. Plagiarism is one of the most serious accusations in science: casually tossing it around like this risks our ability to enforce it against serious offenders.
blitzar 3y ago
I hope they also crack down on undeclared spell-check use in papers
zingababba 3y ago
Meanwhile students getting zeros on essays for being erroneously flagged as being created by AI, what a fun place modern academia is.
SubiculumCode 3y ago
Using a LLM to expedite writing is not fake science. You can use LLMs to create fake science, but also use it expedite writing up real science. I have experimented with feeding in a paper that I've read, I know is relevant, and direct the LLM to extract this point out and connect it to this other point. I then go through and edit. Its faster to rewrite existing text (for me) than to generate it the first time. We should NOT be demonizing tools that increase our scientists productivity when they use it responsibly, rigorously, and ethicly.
smeej 3y ago
Does anybody still believe in the peer reviewed journal system? After all these years and the countless stories of passing off fake information? We know people make up their data. We know peer reviewers sign off on it without actually looking it over. And we know the journals publish it anyway, and make a lot of money off of it. Why does anybody even care if we've added another layer of fabrication to the mix of something that's already so full of lies?
- ulrashida 3y ago
  This feels very dismissive of the efforts put in by genuine participants into the system. Exceptions and examples of malfeasance are not sufficient to indict all aspects.
  - coolhand2120 3y ago
    What happens when fake/fraud/whatever is > 50% as it is in some domains of science? https://en.m.wikipedia.org/wiki/Replication_crisis https://en.m.wikipedia.org/wiki/Replication_crisis I believe something that is not falsifiable is not science. So what would failure look like to you if it isn’t “> 50% off science is fake”? What do you call failure of this system?
- JohnFen 3y ago
  > Does anybody still believe in the peer reviewed journal system? Sure, why not? Yes, it's imperfect and BS can get through. But on the whole, it works reasonably well.
  - breakwaterlabs 3y ago
    A system that generates plausible, seemingly authoritative information, but often makes hard to detect errors ranging from minor to outright lies is dangeorous. This goes double when the information is either difficult or impossible to verify. This shouldn't be surprising, since the most effective and dangerous liars tell the truth most of the time.
EGreg 3y ago
And articles And comments I predict that by 2030, probably 99.9% of all content and published discoveries will be AI-assisted, and outcompete manual stuff for grants and money. Manual content will be as niche as knitted sweaters on etsy.
devin 3y ago
In cases where it's being used to clean up grammar, perhaps the new standard should be providing a diff between the original paper and the cleaned up version.
dionysus_jon 3y ago
I don’t really think the issue is that fact that ChatGPT (or perhaps another llm) was used. The issue is that it was not checked. Perhaps they used it just to translate as a final step and did not have a fluent person to check it over (with the requisite level of understanding) LLMs are new, and soon to be superseded by multimodal AI, so the understanding of them is currently quite immature, in both usage and consumption of.
mensetmanusman 3y ago
Maybe 50% of science research being reproducible will be the high point.

103 points by williamsmj 3y ago

This is going to be like Photoshop usage. Very quickly it will be expected that papers use AI to write parts, just as fixing white balance in Photoshop doesn’t require alerting the viewer that the photo has been enhanced. More extensive use may need to be acknowledged. But I’m not sure…design tools are rarely credited except in cases like a Fx heavy movie

RugnirViking 3y ago
Absolubtely. Im involved in writing right now and i've caught people using its output. It's hard to make an outright ban, heck ive used it myself for things like tricky LaTeX formatting (make this stuff into a nice table with color boxes corresponding to the rgb values). But as with all papers the issue is just a lack of dilligence, be it people straight up using hallucinated sources or the paper bizzarely fawning over every aspect of the project, even things that are objectively rubbish and author knows that when questioned (such as a paragraph praising the high quality of servo motors used when they are blamed for test failure in a later section). Someone taking the time to use it well will use it well, someone lazy will still be lazy.
- siftrics 3y ago
  > Someone taking the time to use it well will use it well, someone lazy will still be lazy. Exactly. It's a self-solving problem as long as draconian luddites don't outright ban LLMs.
improv 3y ago
Reminds me of when teachers told students not to use Wikipedia for research purposes.
- siftrics 3y ago
  I clearly remember getting in trouble in class each year for using Wikipedia. My podunk teachers couldn't fathom the difference between directly citing Wikipedia (just cite the source it references instead) and merely reading it. Pain.
  - thomasahle 3y ago
    > just cite the source it references instead You also shouldn't cite sources you haven't read yourself.
    - siftrics 3y ago
      I agree?
  - hotnfresh 3y ago
    Some teachers notice if you only cite sources on the Wikipedia page, too. More likely if the whole class is writing on the same topic. The work-around is to write it from Wikipedia, but find sources by searching on Google Books. Being able to read a couple pages around each part you’re citing is enough, so their page allowance for restricted works is usually plenty. Hit your citation count and real-book-source requirement (if they still do that second one) without leaving your chair. Barely more work than writing from Wikipedia and not covering your tracks.
- ProllyInfamous 3y ago
  My wikipedia account is over two decades old, and I often bring this up when talking bee -keeping stuff... them both being "bottom up organization that results in a beautiful exchange of information." This experience "went full circle" for me, January 2023... when ChatGPT cited SOMETHING I HAD WRITTEN into wikipedia (re: transistor density).
jprete 3y ago
This is a more serious problem than that, because AI does in fact make stuff up, and will bullshit logical connections that make superficial sense but don't match the system under discussion. I think a better analogy is airbrushing photos to remove people; it changes the facts of the photo in a way people care about. Likewise, it's a problem if people sloppily use GenAI to fix up text or graphs or whatever, and it creates things that match the paper really well but don't match reality. Those are _exactly_ the kinds of problems that will slip through peer review very easily. Fixing white balance, or a thousand other old-school photo changes, is very different; the underlying structure of the photo is there and nobody cares about white balance or most of those old corrections, because our eyes and brains are constantly mucking with them anyway. People care about airbrushing or GenAI-"airbrushing" parts of the picture, because the information content we care about is changed.
- hn_throwaway_99 3y ago
  > I think a better analogy is airbrushing photos to remove people; it changes the facts of the photo in a way people care about. I was thinking about this a lot recently with the announcement of the Pixel 8. One of the big announced features was "Magic Editor", where you could edit all your photos to automatically put smiles on people who were frowning, remove people altogether like that inconvenient homeless person or garbage truck: https://blog.google/products/pixel/google-pixel-8-pro/ https://blog.google/products/pixel/google-pixel-8-pro/ This may be a bit "get off my lawn", but honestly I was disgusted. Why do people feel their everyday family photos need to match some Insta-bullshit ideal? Some of my favorite old family pics are where something "wrong" or unexpected happens. I don't fault Google for this, and I think it's natural to want to touch up photos when you're taking an important picture. But I feel like we're careening towards a future where we're starting to forget what reality actually is. When TikTok came out with their extremely realistic "makeup filters", I remember seeing a video where a photographer commented that she would take pictures of women who were shocked at how "ugly" they were because they weren't used to seeing "un-Facetuned" photos of themselves. Sorry for the semi-rant, just feels like a situation where any individual, small technological improvement makes sense, but in totality we end up at a place that is overall worse than where we started.
sxg 3y ago
I don't think authors need to credit AI in their written works, but I do think this is the double-edged sword of claiming partly AI-generated content as your own (for reference, the WGA recently won the right to claim AI-inspired and partially AI-created work as a human writer's own work without having to credit the AI). I think it's fair for a scientific human author to take full credit for papers that are partially written by AI, but any logical errors and hallucinations inserted by AI should also be attributed to the human author. It's the human author's responsibility to ensure the final written work is accurate. Edit: for further context, I have published a few scientific papers in recent years, and using AI is not all that different from a senior author relying on junior authors. Early on in my career, I created the first draft of most papers, and my more senior co-authors would verify my work and create direct edits or suggestions for me to revise the draft. The final paper was created after a few rounds of this cycle.
registeredcorn 3y ago
I don't mean to be super nitpicky, but I've got to strongly disagree with your comparison to photoshop / photography. There's been an ongoing debate in the Photography community going back as far as photographic editing has existed, regarding when and how acknowledgement of editing has been performed, whether it be literal, physical cutting and pasting of printed media, or a digital equivalent. Some of the more famous personalities in Photography seem to lean to something like this: "Minor adjustments like removing dust from the lens, adjusting colors to match the 'true' colors as it was taken (or to make it 'more visually pleasing'), cropping and level correction are fine and expected. The use of things like Sky Replacement [1] or content-aware fills [2] should be attributed or referenced somewhere if they are the focal point of the photograph." There are others who do much more "artistic" styles of photography [3] where they argue something like: "No person in their right mind would believe that there isn't some amount of editing going into nearly photograph, so why mention it at all?" or "Because editing has been so heavily used, by so many famous photographers for so long, without mentioning editing, it's fine if people don't mention the use of digital (or physical) editing." or "Popular services like Instagram have built-in filters that users can use to make themselves look different - should the people who post photos that have those filters on also be pressured into mentioning that their photo was edited? If not, what's the difference?" There's really not any solid consensus on what is "correct" when it comes to editing attribution. Most tend to agree that doing the bulk of the work "in-camera" (cropping by moving in closer, for example) will make your life easier later on, but, there are some outliers that strongly advocate that any form of editing in post, is "not photography". Things get even weirder when you get into print. The same photograph printed on different mediums of paper, or with different printers, or different ink, or different DPI settings can come out looking noticeably different even to laymen (imagine a smeared blotchy photo versus a crisp, clear one - there are times where you might want the photo to look somewhat blotchy for aesthetic reasons). In a way, choosing one form of medium over another is a kind of editing, because the end presentation is measurably different from the original. A quick example might be noticing the difference between modern movies that are shot on digital vs traditional film. There is a kind of "feel" to the end product, because of the medium that was used to record and distribute it. TL;DR The photography community is very much not in agreement to what amount of editing is, or is not permissible, or when it should, or should not be attributed or brought up, or even how that information should be conveyed. Some are fine without any mention whatsoever because of historical reasons. Others want it mentioned under the use of certain tools or criteria. Still other don't consider a photographer to be a "real" photographer at all, and instead should be called a "stills editor" (or something like that) if they use any editing at all, outside of "in-camera work". [1] https://helpx.adobe.com/photoshop/using/replace-sky.html https://helpx.adobe.com/photoshop/using/replace-sky.html [2] https://helpx.adobe.com/photoshop/using/content-aware-fill.html https://helpx.adobe.com/photoshop/using/content-aware-fill.h... [3] https://www.lookslikefilm.com/2019/08/15/artistic-photos/ https://www.lookslikefilm.com/2019/08/15/artistic-photos/
johngossman 3y ago
And while we're on the topic (Getty Images created a generative AI that gives credit to the source images) https://news.ycombinator.com/item?id=37793579 https://news.ycombinator.com/item?id=37793579

NoMoreNicksLeft 3y ago

The hilarious part isn't that a human didn't write the paper. Instead, consider that in some obscure environmentalism journal, a paper might only ever be read by a few dozen people around the world anyway. Even a good paper on some groundbreaking study might have very narrow interest after all. But we're fast approaching an era where only the LLMs will read the thing, and only to spit out a summary of it. No human wrote it, no humans will read it. The biggest journals will of course avoid this fate the longest, maybe even putting it off indefinitely. And some of the most useless journals have been doing worse for a long time. At some point though, doesn't academia have to wake up and realize how pointless all of this is?

alchemist1e9 3y ago
I was thinking something a bit similar about news in the future. I’m already thinking I should try to put together an automated daily briefing for myself using a list of sources and ask LLMs to write a summary briefing. But then also consider in the future the news sources will also likely be generating some of the articles with LLMs. This LLMs reading LLM output is a curious development. Same for emails also btw. I already have an LLM redraft most of my emails as it makes them more readable and concise.
- daft_pink 3y ago
  Arc browser is already summarizing things this way.
barrysteve 3y ago
It's darkening the public usage, the private channels for research and communication will become disproportionately more valuable. It's odd not to see Google or Meta try and make a copyright strike system for written text. Copy and paste LLMs are driving people away from google's ad bucks, filling FB with junk. I guess 'the web' is no longer worth defending, de facto.
- jprete 3y ago
  It's possible that the open Web simply cannot be defended. The advantage is generally on the attacking side rather than the defending side, after all.

shadowgovt 3y ago

It's easy to forget how many people don't actually have English as a first language. With so many journals demanding papers submitted in a language not native to the author, use of assistive technology to convert the paper so the underlying information can be made known should be expected.

the_af 3y ago
If it's just a better Google Translate, then I don't see the problem. If it's used to actually write the first version because people can't write or sort out their ideas... I dunno. Is there any evidence that it's primarily used by non-native English speakers for this purpose, though? (edit: just saw the list of authors, probably this is the case, at least in this context).
- shadowgovt 3y ago
  > If it's used to actually write the first version because people can't write or sort out their ideas... I dunno. For what it's worth, I'm also 10,000% on board with that use case. I have definitely stared at a blank piece of paper with writer's block because I couldn't write the introduction for a paper before. If ChatGPT gets somebody past that block, more power to it. Every paper is required to start with some novel variant of "from the dawn of time humanity has wondered if..." when all you want to say is "We discovered some stuff, here it is." I am super in favor of automating the step that nobody wants to write, nobody wants to read, and no journal will accept without its presence. Reporting novel scientific findings was never supposed to be an exercise in fluent English authorship mastery.
  - SoftTalker 3y ago
    I haven't written scientific papers, but for other reports or writeups I always start with the meat of what I want to say. I write the introduction later, once I know what it is that I'm introducing.
  - the_af 3y ago
    > I am super in favor of automating the step that nobody wants to write, nobody wants to read, and no journal will accept without its presence. You make a good point. Maybe the bits that can be automated are the zero-value bits, and therefore the best conclusion would be to get rid of them? In other words, if it can be automated, it's not real content. I'd be wary of people writing summaries or conclusions using ChatGPT though, because ChatGPT can write very convincing crap that completely misses the point. Someone who is careless enough to forget removing the "Regenerate response" part of the output may be as careless reviewing whether the conclusion truly says what they meant it to say. As an aside, I've asked ChatGPT 3.5 to provide interpretations of works of fiction of which I know the most common ones (e.g. "this is a parable about [...]", "this is a warning against authoritarianism", etc), and ChatGPT sometimes comes up with stuff that completely misses the point; but if you aren't familiar with the work (haven't read it except for a summary) you could mistake it for a valid interpretation!

iamben 3y ago

Could this just be a translation? A non native speaker pasting their paper into ChatGPT and asking it to translate it to English, and then copying the results and hoping for the best?

daft_pink 3y ago

I think it's okay to use ChatGPT to write and fix your prose, but not to do your thinking or analysis for you. That's how I use it at work.

RugnirViking 3y ago
> to write and fix your prose fix prose, absolubtely. To write? Well a lot of the more egrerious errors ive seen when trying to get it to write academic writing comes from it not having a full context of what the problem is/whats been done/ what you are trying to do. Ive been turning over the idea in my head that for one to fully give it all the required context one might as well just use that context as the piece of writing in the first place. There is no advantage into stretching the relevant information to 6 paragraphs instead of 2. Not to say it should never be used but trying to use it while being thorough and actually reading its output has usually led me to extending context more and more and eventually just writing the section myself.
- daft_pink 3y ago
  I don't think you should just blindly accept the results. I just write what I want to say stream of consciousness, drop it into chatgpt and then review the result to see where it improved and where it didn't. A prompt such as "Why are all the dolphins dying?" would be unacceptable. A prompt such as rewrite "explain that the dolphins are dying because 30 degree rise in ocean temperaturs made their environments unlivable" that reformats the language into a academic style I feel is totally acceptable and efficient if you read what it spits out and agree and edit from there.
jprete 3y ago
I am very sympathetic to people who are using it to write better in language X when their native language is Y. I think most other people should not use it for writing, because the hard part is thinking about (A) what you _really_ want to say (B) what your audience cares about (C) mistaken assumptions by you or your audience, which need correction in the text. And I think people using ChatGPT to write will give it a vague idea of (A) and usually nothing about (B) and (C).

the_af 3y ago

If you look at the papers from the list, some of the authors respond to the comment which asks for clarifications on ChatGPT usage (for those who didn't read TFA, this is detected mostly because they cut & pasted ChatGPT's response without removing extraneous parts like "Regenerate response" from the UI! So not only do they rely on ChatGPT, they also don't know how to use it well). What's interesting is that even when they reply, authors don't seem to respond to the ethics/standards issue of actually using ChatGPT. Well, some do ("I will not do it again" is basically what one of the authors says, which reads like a childish response to me), but many claim it was a "mistake" -- the mistake being... forgetting to remove the boilerplate and "regenerate response" garbage! Uh-oh. I think an acceptable use of AI would be in translation, basically a better Google Translate. But is this what's happening here?

sonicanatidae 3y ago
I find it helpful in working with technology. I have been experimenting with generating powershell scripts using chatgpt, and aside from a few syntax errors or localization, the scripts are a great starting point. Instead of writing the whole script. I can have ChatGPT write it, then I just have to clean it up, address any errors and localize.
- sdenton4 3y ago
  Yeah, exactly this. Paragraphs of condensed scientific text after six rounds of collaborator revisions are typically a mess - ChatGPT is great for suggesting a more readable alternative, and then further crafting into something actually-good. I think a lot of the hand-wringing neglects that there's a range of ways to use LLMs. Text-refactor is obviously perfectly ethical, unless you're gunning for the Butlerian Jihad, no worse than the grammar checker in Word.
  - the_af 3y ago
    > I think a lot of the hand-wringing neglects that there's a range of ways to use LLMs. You and the comment you replied to seem to be making an argument about something neither my own comment nor TFA said. Nobody is claiming LLMs or ChatGPT aren't useful. The argument is about using it to write parts of research papers without disclosing this usage. Nobody mentioned using ChatGPT to write code.

deleted 3y ago

[deleted]

artursapek 3y ago

BRB changing my email signature to "Regenerate response" lol

choeger 3y ago

There's a huge fraudulent scene when it comes to scientific papers, IMO largely driven by the "schoolification" of universities. Many authors seem to consider writing and publishing a research paper as just another form of an exam and thus cheating as the logical approach.

JohnFen 3y ago
People consider cheating on exams as a "logical approach"? That might be the most depressing thing I've heard today.

hn8305823 3y ago

It's not so much the undeclared use of ChatGPT that bothers me but the egregious sloppiness of it. It was already obvious that writing scientific papers has evolved into a cargo cult, but this shows that it's much worse than previously thought. Authors, referees, and journals go through the motions of what lead to genuine scientific advancement in the past, but no actual value is being created at any step in the process for that vast majority of papers. Of course most papers will never be ground breaking but at least in the past there was some value being created in the process.

crazygringo 3y ago

It seems totally unclear whether ChatGPT is being used to invent fictional methodology or literature review etc., which is extremely unethical, or whether it's simply being used to check and fix grammar and tone for non-native speakers, which is perfectly fine. But regardless, this is extremely troubling in terms of quality. The fact that "regenerate response" gets left in tells us that even if it's just for grammar correction: 1) The authors aren't even minimally reviewing if the ChatGPT version didn't accidentally change the meaning of the sentences 2) Peer reviewers clearly aren't even reading the paper 3) The journal isn't even reading the paper to introduce a minimal level of quality control ChatGPT seems to be the smallest problem here. And thankfully, it is revealing much more serious problems about quality checks in academia at all. The question is, will anything be done about it? Not about ChatGPT, but about the horrifically irresponsible processes that would let these slip through?

davely 3y ago
EDIT: Oh, wow. Lack of coffee this morning. I read that as "fictional mythology", NOT "fictional methodology." Ignore me. :) ----- I want to pull apart one specific thing you mentioned, essentially: > It seems totally unclear whether ChatGPT is being used to invent fictional methodology [...], which is extremely unethical In the context of generating something fictional, why is using ChatGPT unethical? One reason I'm asking: I have always been a big fan of NaNoWriMo[1] (National Novel Writing Month, every November). This year, I'm interested in exploring how to use ChatGPT -- not necessarily to generate a ton of prose, but to essentially help flesh out some plot ideas that I've had. Part of the back and forth I've had with it is coming up with some interesting scenarios for my story that I hadn't really thought about and I'm keen on exploring a bit more. [1] https://en.wikipedia.org/wiki/National_Novel_Writing_Month https://en.wikipedia.org/wiki/National_Novel_Writing_Month
- mola 3y ago
  Because this fiction is presented as part of a scientific paper....
  - 3y ago

ugjka 3y ago

It just gonna get worse as every commercial editor will get some AI completion crap

cm2012 3y ago

"Signs of undeclared Microsoft Word spellcheck use in papers mounting." Such alarmist tones would sound really silly when applied to other dumb productivity tools. In the end, the quality of the paper is what should be judged.

Valodim 3y ago
That is true in principle. But as scientific papers, this is supposed to be extremely precise, highly iterated, and thoroughly reviewed content. If there are very obvious mistakes like a leftover pasted in button text, it calls into question whether the authors (or reviewers) were responsibly verifying the content contributed by a tool that is known to sometimes deliver wildly but subtly incorrect results, and thus the quality of the paper content as a whole.
- cm2012 3y ago
  This just let's you see which writers are incompetent which is a good thing. People who copy and paste chat GPT without checking would have shit papers regardless.

matthewdgreen 3y ago

Four quick thoughts: 1. Bad grammar in a paper is incredibly irritating, which is particularly unfair for people who are non-native English speakers. If ChatGPT can help people to convert awkward English into nicer work without changing the ideas or confabulating, that seems like a win. I don't see why people need to "declare" their usage of ChatGPT for these purposes. Is that a standard now? 2. The major issue here is that the authors copied a response and accidentally also brought some button text with it. This is given to imply that people aren't reading ChatGPT's output. But I think this is just as likely to be benign. People do all sorts of silly nonsense in papers (including me), like leaving author comments in final drafts. It's bad and sloppy, but it isn't usually a crime. 3. Many papers on public servers are junky. Many journals and conferences are equally junky. Search for any bad string and you'll find it everywhere, and non-scientists will be scandalized. 4. I like the people at Retraction Watch and think they're doing the Lord's work. However sometimes I worry that immersing yourself in wickedness makes you see wickedness everywhere, and also creates an (unconscious) incentive to sensationalize bad stuff. I hope they're aware of this.

stayfrosty420 3y ago
how is (2) likely to be benign and not plagiarism? It is completely different from leaving an author comment in a draft...such a weird take.
- matthewdgreen 3y ago
  If I type a series of bullet points into ChatGPT and ask it to re-phrase them in better English prose, am I plagiarizing? If it's not plagiarism but I screw up at copy/pasting, does it become plagiarism? I don't know! It is not something I personally would ever do because I like my own writing. But I'm also not going to throw that accusation at ESL authors who are using a writing tool (like Autocorrect-on-steroids) to convey original ideas to an audience, at least not until we've had a much longer community discussion on using these tools. ETA I would urge you not to throw the death penalty at people whose crime may be more akin to shoplifting. This is bad both for the shoplifters and for people who want to fight serious crime. Plagiarism is one of the most serious accusations in science: casually tossing it around like this risks our ability to enforce it against serious offenders.

blitzar 3y ago

I hope they also crack down on undeclared spell-check use in papers

zingababba 3y ago

Meanwhile students getting zeros on essays for being erroneously flagged as being created by AI, what a fun place modern academia is.

SubiculumCode 3y ago

Using a LLM to expedite writing is not fake science. You can use LLMs to create fake science, but also use it expedite writing up real science. I have experimented with feeding in a paper that I've read, I know is relevant, and direct the LLM to extract this point out and connect it to this other point. I then go through and edit. Its faster to rewrite existing text (for me) than to generate it the first time. We should NOT be demonizing tools that increase our scientists productivity when they use it responsibly, rigorously, and ethicly.

smeej 3y ago

Does anybody still believe in the peer reviewed journal system? After all these years and the countless stories of passing off fake information? We know people make up their data. We know peer reviewers sign off on it without actually looking it over. And we know the journals publish it anyway, and make a lot of money off of it. Why does anybody even care if we've added another layer of fabrication to the mix of something that's already so full of lies?

ulrashida 3y ago
This feels very dismissive of the efforts put in by genuine participants into the system. Exceptions and examples of malfeasance are not sufficient to indict all aspects.
- coolhand2120 3y ago
  What happens when fake/fraud/whatever is > 50% as it is in some domains of science? https://en.m.wikipedia.org/wiki/Replication_crisis https://en.m.wikipedia.org/wiki/Replication_crisis I believe something that is not falsifiable is not science. So what would failure look like to you if it isn’t “> 50% off science is fake”? What do you call failure of this system?
JohnFen 3y ago
> Does anybody still believe in the peer reviewed journal system? Sure, why not? Yes, it's imperfect and BS can get through. But on the whole, it works reasonably well.
- breakwaterlabs 3y ago
  A system that generates plausible, seemingly authoritative information, but often makes hard to detect errors ranging from minor to outright lies is dangeorous. This goes double when the information is either difficult or impossible to verify. This shouldn't be surprising, since the most effective and dangerous liars tell the truth most of the time.

EGreg 3y ago

And articles And comments I predict that by 2030, probably 99.9% of all content and published discoveries will be AI-assisted, and outcompete manual stuff for grants and money. Manual content will be as niche as knitted sweaters on etsy.

devin 3y ago

In cases where it's being used to clean up grammar, perhaps the new standard should be providing a diff between the original paper and the cleaned up version.

dionysus_jon 3y ago

I don’t really think the issue is that fact that ChatGPT (or perhaps another llm) was used. The issue is that it was not checked. Perhaps they used it just to translate as a final step and did not have a fluent person to check it over (with the requisite level of understanding) LLMs are new, and soon to be superseded by multimodal AI, so the understanding of them is currently quite immature, in both usage and consumption of.

mensetmanusman 3y ago

Maybe 50% of science research being reproducible will be the high point.