10 ms·

'Obviously ChatGPT' – how reviewers accused me of scientific fraud

17 points by zettabomb 3y ago

throwitaway1123 3y ago
> I was accused of writing fraud: passing off ‘writing’ produced by artificial intelligence (AI) as my own. That hurts, because — like many people — I find writing a paper to be a somewhat painful process. This is what worries me the most about AI (and large language models in particular). You can summarily dismiss someone's work by simply claiming it was written by ChatGPT.
- czl 3y ago
  Why is scientific writing being judged by the writing style / etc. vs the power and novelty of the ideas it contains? Manuscripts are prepared with spell check / grammar check, with “mechanical” fonts, computation and layout. How is AI helping improve scientific publications different from other ways technology has improved them?
  - throwitaway1123 3y ago
    > Why is scientific writing being judged by the writing style / etc. vs the power and novelty of the ideas it contains? It's not solely being judged on the basis of the writing style, and the author alludes to this in the first paragraph: "I have just been accused of scientific fraud. Not data fraud — no one accused me of fabricating or misleadingly manipulating data or results. This, I suppose, is a relief". I think there is an art to communicating complex ideas clearly, and some people value that ability, and don't like having their rhetorical ability discredited. There are certain scientists who I appreciate almost as much for their ability to communicate science to the public as I do for their ability to conduct scientific research. Richard Feynman for example had an uncanny ability to convey scientific concepts.
    - viraptor 3y ago
      It feels like the wrong criticism to me for that reason. As far as form goes, who cares about GPT/not-GPT? If there's an issue with the form, point out that issue. If there's an issue with content, point out that issue. There's a separate argument about potentially unknowingly copying someone's exact words. But beyond that, it's a lazy criticism that's common on the internet. You could mentally replace "it's GPT" with "it's lame". Ok - but why does the person think it's lame.
      - throwitaway1123 3y ago
        After reading over your response and the other commenter who replied to me (czl), I think to a certain extent we're talking about different things. You both seem to be saying that using a chatbot to write a manuscript shouldn't automatically disqualify your work from being published, which I don't really have any qualms about. What bothers me is the idea that a talented digital artist could spend days crafting a photorealistic painting only to have someone rob them of the credit they're due by attributing the work to Midjourney. It bothers me that a gifted writer could spend hours painstakingly deliberating over the best way to convey an idea only to have the credit for the prose given to OpenAI. This is basically the sentiment that the author of this article was expressing: "That hurts, because — like many people — I find writing a paper to be a somewhat painful process. I read books on how to write — both to be comforted by how much these books stress that writing is generally slow and difficult, and to find ways to improve. My current strategy involves willing myself to write and creating several outlines before the first draft, which is followed by writing and a lot of revising." People generally don't enjoy working hard on something and having the credit stolen by another entity (regardless of whether or not that entity is a machine). I don't have a problem with people using Github Copilot to write code for example, but if I personally write my code by hand it's infuriating if someone gives the credit to Github's AI.
        czl 3y ago
        > What bothers me is the idea that a talented digital artist could spend days crafting a photorealistic painting only to have someone rob them of the credit they're due by attributing the work to Midjourney. I try to impress you with my ability to look and create photo like images. It takes me a few days to do each one. Camera takes a few seconds to do the same thing. Should I be offended when you tell me I must have used a camera to make mine? How much "credit" should my skill get? What value is my skill in the age of cameras?
    - czl 3y ago
      Science > I think there is an art to communicating complex ideas clearly, and some people value that ability, Yes. There is also an art to fine penmanship and the ability to craft a professional manuscript by hand with just ink. This used to be valued now we use mechanical fonts and layout engines etc. Times change, technology changes. The utility / novelty of the ideas vs how they are conveyed: which of those matters for a science publication? If both why not also got back to handwriting and ban the use of fonts / layout engines to increase the challenge required to create good manuscripts? Is science a costly signalling status contest, a process that should be as exclusive as possible? Is science a search for truth, a process that should broadly accessible even to those with poor language skills?
      - throwitaway1123 3y ago
        > now we use mechanical fonts and layout engines etc. Times change, technology changes. Absolutely, and I personally don't design my own fonts. But if I were a professional typographer like Matthew Butterick[1] for example, and someone accused me of using Times New Roman when I actually created my own fonts from scratch, I'd probably be a little perturbed. > Is science a search for truth, a process that should broadly accessible even to those with poor language skills? It's absolutely a search for truth, and like I said elsewhere, I'm not opposed to someone using a large language model to help them document their findings. You can continue using ChatGPT to help you write [2]. I have no issue with that. You're arguing against a straw man. I'm saying if someone had poor English language skills for example, and they spent years acquiring English language proficiency, they'd probably be annoyed if you discredited their years of mastering the language by simply saying "you're not fluent in English, you just used ChatGPT to write everything". That doesn't mean someone else who doesn't have the time or resources to master English can't use ChatGPT as a writing aid. Those two things aren't mutually exclusive. [1] https://practicaltypography.com/about-matthew-butterick.html https://practicaltypography.com/about-matthew-butterick.html [2] https://news.ycombinator.com/item?id=38637667 https://news.ycombinator.com/item?id=38637667
        czl 3y ago
        > if I were a professional typographer like Matthew Butterick[1] for example, and someone accused me of using Times New Roman when I actually created my own fonts from scratch, I'd probably be a little perturbed. A closer analogy with AI for science writing would be to accuse you of getting help from a computer to create the new font and not doing it by hand. When you create fonts goal is the font. When you publish research the writeup is important but goal is useful and new results. Because accuracy and clarity of the writeup matters whatever serves that best should be used. > You can continue using ChatGPT to help you write [2]. > https://news.ycombinator.com/item?id=38637667 https://news.ycombinator.com/item?id=38637667 The example you quoted states the output comes from ChatGPT. It was used to explain some concepts and it did a far better job than I could do for the time it took. > I'm saying if someone had poor English language skills for example, and they spent years acquiring English language proficiency, they'd probably be annoyed if you discredited their years of mastering the language by simply saying "you're not fluent in English, you just used ChatGPT to write everything". Will the situation be the same when AI improves so much that it helps writing much like mechanical fonts regularly help our writing today? Say I took years to develop penmanship skills such that my hand writing looks like mechanical fonts. Should I be annoyed if you say “Your hand writing looks like a mechanical font!”? When it comes to science writing should pride be placed in penmanship skills? Should pride be in English language skills? What matters for science writing? Today AI writing has a mixed reputation. Sometimes the output is fantastic. Sometimes output is fantastic nonsense. To tell someone their manuscript looks like ChatGPT wrote it is likely meant as an insult. In the future do you not expect AIs will be used to judge writing (like Latex style sheets guide journal submissions today)? Will writing not be rejected unless it meets high standards for clarity etc? Perhaps the best way to meet those standards will be to get an AI to help you? How do you suppose the future of science writing will go?
  - bart_spoon 3y ago
    This is the biggest question to me. Who cares even if the reviewer did use ChatGPT to write his findings. The findings themselves are what is important. This is just another example of the abysmal state of affairs that is modern peer-review.
happytiger 3y ago
It’s crazy but angry mobs will attack you about being GPT on virtually anything that is long-form content or uses advanced vocabulary deemed too GPT-ish. I’ve had the experience so much on various forums I now have to write “This is not GPT” on most posts to try to preempt the trolls (surprisingly this works). There is a need to be able to source validate the work, and see the creation of things in order to validate their originality already, and we are barely into the AI era. This obviously leads to a deep breakdown in trust. I suspect the next step will be some kind of Real ID that allows everything you post to be validated and sophisticated technologies to try to counter AI, but it will be in vain as the vast majority of content on the Internet will become AI source generated. It’s an arms race and one side has geometric math advantages. Perhaps eventually, in weird comfort, it will become stylish and trendy to collect “human written” things, like the vinyl records people collect where fully half of the people collecting have no ability to play them. So too, I’m sure we will go full circle with this technology, which will eat and destroy credibility for original work and cast doubt in anything that can be generated, as it becomes so good it cannot be separated from human efforts. https://consequence.net/2023/04/half-vinyl-buyers-record-player-study/amp/ https://consequence.net/2023/04/half-vinyl-buyers-record-pla...
sema4hacker 3y ago
> I now plan to use Git and GitHub for all my writing from day one, and to document changes every day. It’s not an ironclad system, but it has given me some peace of mind — not to mention, a paper trail that clearly shows a manuscript written slowly and painstakingly, and without ChatGPT. It probably won't be too long before someone creates a tool to create fake Git/Github entries to "prove" a document was written by a human after it was actually ChatGPT-generated.
- nomonnai 3y ago
  I am afraid you're right. On the other hand, in my experience, writing an academic text is rarely straightforward, with a lot of forth-and-back involved. Ultimately, a human must decide whether the diffs look more like such a pattern or a script-generated one. Edit. Btw, you can use Git with Word's docx-format out of the box. Docx is a zip folder with XML inside; git-diff will show text changes. Further, there's git-merge-driver for merging various file formats (https://github.com/Praqma/git-merge-driver https://github.com/Praqma/git-merge-driver).
- zettabomb 3y ago
  Unfortunately, as with filesystem dates, git dates are trivially modified. I don't even think writing such a tool would be difficult either - space out the commits semi-randomly within the time of day you might be writing, add a few paragraphs each time (potentially in different sections), occasionally go back and rewrite a sentence or two. It's not proof of anything, but then what do we need to prove that yes, I really wrote this myself? A record of every keystroke and mouse click? A recording of my screen while writing? A recording of my webcam while writing? All seem either fakeable today or very soon. I'm probably just not smart enough but I can't figure out how to truly prove your (human) authorship. To some extent though I'm also not sure how much it matters. If the science is sound, the text is well written, and the paper is well presented, why do I care who wrote the text? I didn't handwrite the graphs or the tables of data, that was a script. Nor did I typeset every letter, that was LaTeX. The text isn't even any different in nature, like the difference between an original oil painting and a print of the same. It's ultimately just a means to communicate ideas from my mind to yours. Perhaps it is ethical to disclose the use of generative AI, but I'm not convinced that would be any better in the minds of the reviewers in question here. I just don't see a substantiative difference if the end product ultimately is high quality. For what it's worth, I wasn't a great student (good at learning, yes, good at going through the motions to get good grades, not so much) and despite what some might call an academic approach to most problems I'm not really a fan of academia itself. There's a lot of stuffy old practices, like p-value being the metric for so many things despite the knowledge that it can be "hacked", and weird, unknowable financial or political motives that you'll never be privy to, which in my opinion compromise the integrity of the work more than use of generative AI does while also being harder to detect. I'm certainly not the first to point these issues and I won't be the one to solve it. I do wonder though if the whole thing is kinda broken, ChatGPT or not.
fragmede 3y ago
https://xkcd.com/810/ https://xkcd.com/810/
nomonnai 3y ago
This is an alarming development. Nevertheless, as a scientist, I believe it's an effect of a misguided culture of trust. This culture has, paradoxically, allowed mistrust of the sort described in the article to fester. > So much of science is built on trust and faith in the ethics and integrity of our colleagues. This is where things went wrong. We need a culture of "show me," not "trust me," the core of critical rationalism: Establishing the convention that checking each other's work is the only way to advance our understanding of the world. Figuring new things out is an error-prone process. Sometimes, these errors are not known to a researcher; sometimes, they are known but deemed non-critical; sometimes, a person has ulterior goals that would be endangered by acknowledging and correcting the error. I don't judge. We've all been there. However, things have been swept under the rug for far too long. If large-scale attempts can only replicate 50% of the studies investigated, published results from psychology cannot and should not be trusted without further checks (https://journals.sagepub.com/doi/10.1177/2515245918810225 https://journals.sagepub.com/doi/10.1177/2515245918810225). The problem may be less acute in other fields, but perhaps only due to a lack of scrutiny (https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008984 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...). Absent an established process to verify a publication's central claims, substitute measures are used, such as "sounds like ChatGPT to me." This is an effect, not a cause, of a culture that values "trust me" over "show me."

17 points by zettabomb 3y ago

> I was accused of writing fraud: passing off ‘writing’ produced by artificial intelligence (AI) as my own. That hurts, because — like many people — I find writing a paper to be a somewhat painful process. This is what worries me the most about AI (and large language models in particular). You can summarily dismiss someone's work by simply claiming it was written by ChatGPT.

czl 3y ago
Why is scientific writing being judged by the writing style / etc. vs the power and novelty of the ideas it contains? Manuscripts are prepared with spell check / grammar check, with “mechanical” fonts, computation and layout. How is AI helping improve scientific publications different from other ways technology has improved them?
- throwitaway1123 3y ago
  > Why is scientific writing being judged by the writing style / etc. vs the power and novelty of the ideas it contains? It's not solely being judged on the basis of the writing style, and the author alludes to this in the first paragraph: "I have just been accused of scientific fraud. Not data fraud — no one accused me of fabricating or misleadingly manipulating data or results. This, I suppose, is a relief". I think there is an art to communicating complex ideas clearly, and some people value that ability, and don't like having their rhetorical ability discredited. There are certain scientists who I appreciate almost as much for their ability to communicate science to the public as I do for their ability to conduct scientific research. Richard Feynman for example had an uncanny ability to convey scientific concepts.
  - viraptor 3y ago
    It feels like the wrong criticism to me for that reason. As far as form goes, who cares about GPT/not-GPT? If there's an issue with the form, point out that issue. If there's an issue with content, point out that issue. There's a separate argument about potentially unknowingly copying someone's exact words. But beyond that, it's a lazy criticism that's common on the internet. You could mentally replace "it's GPT" with "it's lame". Ok - but why does the person think it's lame.
    - throwitaway1123 3y ago
      After reading over your response and the other commenter who replied to me (czl), I think to a certain extent we're talking about different things. You both seem to be saying that using a chatbot to write a manuscript shouldn't automatically disqualify your work from being published, which I don't really have any qualms about. What bothers me is the idea that a talented digital artist could spend days crafting a photorealistic painting only to have someone rob them of the credit they're due by attributing the work to Midjourney. It bothers me that a gifted writer could spend hours painstakingly deliberating over the best way to convey an idea only to have the credit for the prose given to OpenAI. This is basically the sentiment that the author of this article was expressing: "That hurts, because — like many people — I find writing a paper to be a somewhat painful process. I read books on how to write — both to be comforted by how much these books stress that writing is generally slow and difficult, and to find ways to improve. My current strategy involves willing myself to write and creating several outlines before the first draft, which is followed by writing and a lot of revising." People generally don't enjoy working hard on something and having the credit stolen by another entity (regardless of whether or not that entity is a machine). I don't have a problem with people using Github Copilot to write code for example, but if I personally write my code by hand it's infuriating if someone gives the credit to Github's AI.
      - czl 3y ago
        > What bothers me is the idea that a talented digital artist could spend days crafting a photorealistic painting only to have someone rob them of the credit they're due by attributing the work to Midjourney. I try to impress you with my ability to look and create photo like images. It takes me a few days to do each one. Camera takes a few seconds to do the same thing. Should I be offended when you tell me I must have used a camera to make mine? How much "credit" should my skill get? What value is my skill in the age of cameras?
  - czl 3y ago
    Science > I think there is an art to communicating complex ideas clearly, and some people value that ability, Yes. There is also an art to fine penmanship and the ability to craft a professional manuscript by hand with just ink. This used to be valued now we use mechanical fonts and layout engines etc. Times change, technology changes. The utility / novelty of the ideas vs how they are conveyed: which of those matters for a science publication? If both why not also got back to handwriting and ban the use of fonts / layout engines to increase the challenge required to create good manuscripts? Is science a costly signalling status contest, a process that should be as exclusive as possible? Is science a search for truth, a process that should broadly accessible even to those with poor language skills?
    - throwitaway1123 3y ago
      > now we use mechanical fonts and layout engines etc. Times change, technology changes. Absolutely, and I personally don't design my own fonts. But if I were a professional typographer like Matthew Butterick[1] for example, and someone accused me of using Times New Roman when I actually created my own fonts from scratch, I'd probably be a little perturbed. > Is science a search for truth, a process that should broadly accessible even to those with poor language skills? It's absolutely a search for truth, and like I said elsewhere, I'm not opposed to someone using a large language model to help them document their findings. You can continue using ChatGPT to help you write [2]. I have no issue with that. You're arguing against a straw man. I'm saying if someone had poor English language skills for example, and they spent years acquiring English language proficiency, they'd probably be annoyed if you discredited their years of mastering the language by simply saying "you're not fluent in English, you just used ChatGPT to write everything". That doesn't mean someone else who doesn't have the time or resources to master English can't use ChatGPT as a writing aid. Those two things aren't mutually exclusive. [1] https://practicaltypography.com/about-matthew-butterick.html https://practicaltypography.com/about-matthew-butterick.html [2] https://news.ycombinator.com/item?id=38637667 https://news.ycombinator.com/item?id=38637667
      - czl 3y ago
        > if I were a professional typographer like Matthew Butterick[1] for example, and someone accused me of using Times New Roman when I actually created my own fonts from scratch, I'd probably be a little perturbed. A closer analogy with AI for science writing would be to accuse you of getting help from a computer to create the new font and not doing it by hand. When you create fonts goal is the font. When you publish research the writeup is important but goal is useful and new results. Because accuracy and clarity of the writeup matters whatever serves that best should be used. > You can continue using ChatGPT to help you write [2]. > https://news.ycombinator.com/item?id=38637667 https://news.ycombinator.com/item?id=38637667 The example you quoted states the output comes from ChatGPT. It was used to explain some concepts and it did a far better job than I could do for the time it took. > I'm saying if someone had poor English language skills for example, and they spent years acquiring English language proficiency, they'd probably be annoyed if you discredited their years of mastering the language by simply saying "you're not fluent in English, you just used ChatGPT to write everything". Will the situation be the same when AI improves so much that it helps writing much like mechanical fonts regularly help our writing today? Say I took years to develop penmanship skills such that my hand writing looks like mechanical fonts. Should I be annoyed if you say “Your hand writing looks like a mechanical font!”? When it comes to science writing should pride be placed in penmanship skills? Should pride be in English language skills? What matters for science writing? Today AI writing has a mixed reputation. Sometimes the output is fantastic. Sometimes output is fantastic nonsense. To tell someone their manuscript looks like ChatGPT wrote it is likely meant as an insult. In the future do you not expect AIs will be used to judge writing (like Latex style sheets guide journal submissions today)? Will writing not be rejected unless it meets high standards for clarity etc? Perhaps the best way to meet those standards will be to get an AI to help you? How do you suppose the future of science writing will go?
- bart_spoon 3y ago
  This is the biggest question to me. Who cares even if the reviewer did use ChatGPT to write his findings. The findings themselves are what is important. This is just another example of the abysmal state of affairs that is modern peer-review.

happytiger 3y ago

It’s crazy but angry mobs will attack you about being GPT on virtually anything that is long-form content or uses advanced vocabulary deemed too GPT-ish. I’ve had the experience so much on various forums I now have to write “This is not GPT” on most posts to try to preempt the trolls (surprisingly this works). There is a need to be able to source validate the work, and see the creation of things in order to validate their originality already, and we are barely into the AI era. This obviously leads to a deep breakdown in trust. I suspect the next step will be some kind of Real ID that allows everything you post to be validated and sophisticated technologies to try to counter AI, but it will be in vain as the vast majority of content on the Internet will become AI source generated. It’s an arms race and one side has geometric math advantages. Perhaps eventually, in weird comfort, it will become stylish and trendy to collect “human written” things, like the vinyl records people collect where fully half of the people collecting have no ability to play them. So too, I’m sure we will go full circle with this technology, which will eat and destroy credibility for original work and cast doubt in anything that can be generated, as it becomes so good it cannot be separated from human efforts. https://consequence.net/2023/04/half-vinyl-buyers-record-player-study/amp/ https://consequence.net/2023/04/half-vinyl-buyers-record-pla...

sema4hacker 3y ago

> I now plan to use Git and GitHub for all my writing from day one, and to document changes every day. It’s not an ironclad system, but it has given me some peace of mind — not to mention, a paper trail that clearly shows a manuscript written slowly and painstakingly, and without ChatGPT. It probably won't be too long before someone creates a tool to create fake Git/Github entries to "prove" a document was written by a human after it was actually ChatGPT-generated.

nomonnai 3y ago
I am afraid you're right. On the other hand, in my experience, writing an academic text is rarely straightforward, with a lot of forth-and-back involved. Ultimately, a human must decide whether the diffs look more like such a pattern or a script-generated one. Edit. Btw, you can use Git with Word's docx-format out of the box. Docx is a zip folder with XML inside; git-diff will show text changes. Further, there's git-merge-driver for merging various file formats (https://github.com/Praqma/git-merge-driver https://github.com/Praqma/git-merge-driver).
zettabomb 3y ago
Unfortunately, as with filesystem dates, git dates are trivially modified. I don't even think writing such a tool would be difficult either - space out the commits semi-randomly within the time of day you might be writing, add a few paragraphs each time (potentially in different sections), occasionally go back and rewrite a sentence or two. It's not proof of anything, but then what do we need to prove that yes, I really wrote this myself? A record of every keystroke and mouse click? A recording of my screen while writing? A recording of my webcam while writing? All seem either fakeable today or very soon. I'm probably just not smart enough but I can't figure out how to truly prove your (human) authorship. To some extent though I'm also not sure how much it matters. If the science is sound, the text is well written, and the paper is well presented, why do I care who wrote the text? I didn't handwrite the graphs or the tables of data, that was a script. Nor did I typeset every letter, that was LaTeX. The text isn't even any different in nature, like the difference between an original oil painting and a print of the same. It's ultimately just a means to communicate ideas from my mind to yours. Perhaps it is ethical to disclose the use of generative AI, but I'm not convinced that would be any better in the minds of the reviewers in question here. I just don't see a substantiative difference if the end product ultimately is high quality. For what it's worth, I wasn't a great student (good at learning, yes, good at going through the motions to get good grades, not so much) and despite what some might call an academic approach to most problems I'm not really a fan of academia itself. There's a lot of stuffy old practices, like p-value being the metric for so many things despite the knowledge that it can be "hacked", and weird, unknowable financial or political motives that you'll never be privy to, which in my opinion compromise the integrity of the work more than use of generative AI does while also being harder to detect. I'm certainly not the first to point these issues and I won't be the one to solve it. I do wonder though if the whole thing is kinda broken, ChatGPT or not.

fragmede 3y ago

https://xkcd.com/810/ https://xkcd.com/810/

nomonnai 3y ago

This is an alarming development. Nevertheless, as a scientist, I believe it's an effect of a misguided culture of trust. This culture has, paradoxically, allowed mistrust of the sort described in the article to fester. > So much of science is built on trust and faith in the ethics and integrity of our colleagues. This is where things went wrong. We need a culture of "show me," not "trust me," the core of critical rationalism: Establishing the convention that checking each other's work is the only way to advance our understanding of the world. Figuring new things out is an error-prone process. Sometimes, these errors are not known to a researcher; sometimes, they are known but deemed non-critical; sometimes, a person has ulterior goals that would be endangered by acknowledging and correcting the error. I don't judge. We've all been there. However, things have been swept under the rug for far too long. If large-scale attempts can only replicate 50% of the studies investigated, published results from psychology cannot and should not be trusted without further checks (https://journals.sagepub.com/doi/10.1177/2515245918810225 https://journals.sagepub.com/doi/10.1177/2515245918810225). The problem may be less acute in other fields, but perhaps only due to a lack of scrutiny (https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008984 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...). Absent an established process to verify a publication's central claims, substitute measures are used, such as "sounds like ChatGPT to me." This is an effect, not a cause, of a culture that values "trust me" over "show me."