6 ms·
Can AI-Generated Text Be Reliably Detected?
- chrischen 4y agoWhether text is AI generated or not doesn’t matter. It’s whether we can detect if the text is low or high quality that matters.
- nwoli 4y agoYeah I mean if AI HN comments start to appear does that really matter in the end. Ie there’s already a lot of noise in the form of low quality comments, and they get downvoted. If an AI manages to produce high quality comments that people upvote im not sure I care that it wasn’t generated by a human
- jacquesm 4y agoIt would matter to me because I'm here to read what people write, not what bots write. Besides that being against the HN guidelines noise in the form of high quality comments is still noise.
- travisjungroth 4y agoA comment being noise and high quality seems contradictory.
- notahacker 4y ago"High quality" in HN terms means it's not just a couple of words or a meme, isn't glaringly false or advertising or particularly off topic or antagonistic. "Feedback" that isn't actually feedback, syntheses of other people's comments that claim to be a new person adding their agreement, anecdotes that didn't happen, little known facts that aren't true and pseudo-recommendations that are actually derived largely from the product marketing copy will usually pass this filter, but they're definitely noise
- chrischen 4y agoI'm here to read high quality comments. If I'm unable to distinguish between high and low quality comments then I have another problem... maybe I'm the one that's not human.
- p1esk 4y agoI think what he meant was that an AI generated comment might be high quality bullshit, like for example a story from personal experience that has never existed, or a false fact that looks convincing and/or is hard to check, or a personal opinion written by a statistical meat grinder, etc. And these comments might come to dominate any discussion because they are so easy to generate, unlike human high quality comments.
- chrischen 4y agoIf it's now easy to generate nice sounding bullshit then perhaps we should be re-evaluating our goals in reading that. The examples you give is no different from going on TikTok and ingesting mindless entertaining content.
- p1esk 4y agoExactly. We don’t want HN turning into TikTok, do we? What’s worse, there will inevitably appear AI constructed “native” ads or even political propaganda - high quality comments skillfully promoting products or someone’s agenda.
- chrischen 4y agoYes, so downvote comments that are just entertainment. Only keep high quality objective comments. At best problem solved, at worst AI are better at commenting than humans.
- rcme 4y agoIt definitely matters. Like you can play chess against a bot at any level. It’s not nearly as fun as playing another human.
- jacquesm 4y agoIt does matter. For instance, what will you train your future AI on? If AI ends up eating its own tail it will end up amplifying its errors and the end result is that you won't be able to trust anything that wasn't written down before this hit.
- chrischen 4y agoYou shouldn't be trusting random comments as truth in the first place. Maybe learn to identify high value first? Then you can just train on high value targets. Just because (hypothetically) 4-chan is full of humans doesn't make it suddenly a good training target for AI.
- jacquesm 4y agoRandom comments attached to accounts with good reputation count for something. If those get replaced by random comments written by bots attached to accounts with good reputation then that would be a problem. High value has many signals, reputation is one of them. 4chan is/was a very mixed bag, and just like there are garbage HN comments so there were excellent 4chan comments.
- YetAnotherNick 4y agoOpenAI training data oversamples sites which got 3 reddit likes[1], because it improves model. Most large scale training data has some filter on commoncrawl. [1]: https://paperswithcode.com/dataset/webtext https://paperswithcode.com/dataset/webtext
- p1esk 4y agoMy hope is that future AI will become smarter, not just trained on more data, and as a result it will be capable to perhaps “rate” the quality of any text, based on fact checking, reputation of the commenter, content of all their other comments, frequency of the previous comments, consistency of the style, availability of particular LLMs at the time, etc. It won’t be good enough to detect AI of the same level, but might be good enough to filter out most of the earlier AI’s comments. GPT-5 might be able to detect comments generated by GPT-3, even if its training dataset included a small fraction of GPT-3 generated content, or even if it was trained on the same corpus as GPT-4 (up to 2021).
- YPPH 4y agoThere are contexts where it does matter. For instance, plagiarism detection at schools.
- chaxor 4y agoWhy is "at schools" necessarily important though? Wouldn't it also be nice to know if the scientific articles you are reading, or the code a company is based upon, or a new piece of legislature was constructed from someone just asking an LLM to do it for them?
- YPPH 4y agoSure, I was only giving one example to counter the claim, which is not to say there aren't other examples.
- 8note 4y agoAs usual, "just" pulls a lot of weight - if they're putting the straight LLM output, it probably won't be great regardless. If they put effort into making the ouput good, then chances are the content will be good regardless of whether they wrote it by hand, or wrote it using an LLM. Like, I don't need to know whether you used a bic ballpoint pen, or some other ballpoint pen to write your note on paper; the tool is a tool
- chrischen 4y agoPlagiarism can be solved with non-AI methods because plagiarism is defined more or less by exact copying. If you mean detection of AI generated essays at schools, then teachers just have to expect better essays and be able to identify them since students now have a new tool. Think of it like grading math homework before and after the invention of calculators. Once students had calculators at home the problems simply got harder and higher level, since there was no reasonable way to enforce "don't use a calculator at home with these artificially simplistic problems."
- waboremo 4y agoThe more I think about this, the less I agree with it. Think of all the online communication you've had. The good parts. The great but messy blog post that changed your mind about something. Communicating with someone on a social network only to share each other's work and establish a great working relationship. Talking to a friend in DMs sharing your pain. Getting feedback from a mentor. Having people tell you how amazing your art is or how it inspired them. The bad parts too. The negative comments from random people. Others mocking your tweet. People calling your HN comment idiotic. All of these things affect us because we assume a person is behind them, and therefore their intent is clear. Without that, we devalue all communication to compensate. Like a more exaggerated effect of how we view likes on social media, where "20k likes" means nothing. Truly believe we're only ok with it now because it's easy to tell what is AI generated, and they're mostly confined. When AI is creating videos on Twitter, creating threads on Reddit, posting blogs on their personal site to be shared to Mastodon, replying to other bots on HN... I'm not sure we'll feel that it's whether we can detect quality anymore. There's also the differentiation theory where in order to separate human connection from bots, we'll do the opposite. We'll start incorporating prompt injections into everything we post, start purposefully being "low quality" because bots are trained against that. Now we're creating a very particular kind of societal mess.
- YetAnotherNick 4y agoI definitely don't agree with the GP but I don't agree with you either. Human could and absolutely become attached to non humans, ranging from dog to pen to house. And I believe humans might begin to develop deeper relationship with AI bots very soon.
- waboremo 4y agoI don't know. Part of why we anthropomorphize so much is because we can still feel the physical elements. We pet a dog, we live inside our homes, list goes on. When that is not possible, how does this change people and their valuation of online communication? Would you still love "Anne" when she's now rejected your 50th attempt to meet in person? Would you still reply to comments like mine if the default assumption wasn't "that's another human being I can persuade" and instead "that is a bot"? I'm sure in the long term that differentiation alone would change, you know androids and all that jazz - however in the meantime, we're going to be facing a tremendous shift in how we interact online.
- WalterBright 4y agoIf HN develops into a forum dominated by chatbots, I'll leave.
- klipklop 4y agoHow will you know?
- WalterBright 4y agoMaybe I am the only real person that exists, and everyone else is just a simulation.
- itsaquicknote 4y agoCan't remember who said it, but it went something like "any headline phrased as a potentially provocative question means the answer is no". Which is what the paper reduxes to.
- capableweb 4y agoBetteridge's law of headlines > Betteridge's law of headlines is an adage that states: "Any headline that ends in a question mark can be answered by the word no." It is named after Ian Betteridge, a British technology journalist who wrote about it in 2009, although the principle is much older. It is based on the assumption that if the publishers were confident that the answer was yes, they would have presented it as an assertion; by presenting it as a question, they are not accountable for whether it is correct or not. https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
- consumer451 4y agoFor the last few days I have been wondering if there is any analysis to see if this "law" is accurate. Is there anything other than this? > With 46% non-polar and 20% answered “yes”, at least two thirds of our headline sample violates Betteridge’s law. We conclude that it cannot be “mostly correct” either. http://calmerthanyouare.org/2015/03/19/betteridges-law.html#:~:text=Betteridge's%20law%20of%20headlines%20famously,sometimes%20hides%20behind%20dubious%20headlines http://calmerthanyouare.org/2015/03/19/betteridges-law.html#....
- knodi123 4y agoSounds like you should read my article "Is Betteridge's Law Always A Reliable Tool For Summarizing Articles?" Or you can just apply betteridge's law to it. ;-)
- butterguns 4y agohttps://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
- adoxyz 4y agoIn my experience, no. But it's highly dependent on the prompt and the subject matter. Especially with GPT4 and the right prompt.
- rkwasny 4y agoNo.
- rkwasny 4y agoUnless you know the weights.
- sebzim4500 4y agoEven then, you don't know the prompt.
- stu2b50 4y agoI don’t think text is sufficiently unconstrained. It is trivial to get GPT to output “Hi, how are you?” but of course it’s impossible to determine if that common turn of phrase is human or machine generated. There is too much a correlation between the position of tokens and correct meaning. In contrast, in an image, it’s very possible to have pixels shuffle around and not change the look. There, it is more likely that we can determine that the output is statistically more likely to be from a particular model.
- AlotOfReading 4y agoYeah, trivial text samples are obviously insufficient. At the other extreme (say millions of tokens of output), there's probably a reasonable chance that you could reliably distinguish human and LLM output. I suspect there might be some interesting questions down the rabbit hole in the middle though.
- deleted 4y ago[deleted]
- Atlas22 4y agoAmusing that the authors acknowledge ChatGPT for "improving clarity and readability" of the paper.
- wsgeorge 4y agoIt was more fun to see GPT-4 acknowledged in the GPT-4 technical report [0] OpenAI published. [0] https://arxiv.org/abs/2303.08774 https://arxiv.org/abs/2303.08774
- puttycat 4y ago(haven't read the paper) Doesn't it boil down to a halting-problem kind of argument? Assume you have a model that can detect AI-generated text. Then you can build a model that emulates the detection model, finds the nearest non-detected text, and outputs it.
- sebzim4500 4y agoTrue but that could be computationally impractical, depending on how hard the detection process is.
- Atlas22 4y agoCouldnt the same be done with the generation model? I.e, The detection model can emulate the generation model to inform detection.
- brucethemoose2 4y agoWhen I used to finetune ESRGAN on its own seemingly flawless looking output, the "feedback" would create artifacts in the subsequent output. Sometimes I could see why if I went back and pixel peeped the training data, but sometimes I couldn't. I wonder if something similar can be applied to LLMs? Maybe running its own output through the net would "amplify" it in a detectable way.
- ldjkfkdsjnv 4y agoAt this point, I want human text filtered out. ChatGPT-4 spam is better than SEO spam. User created comments on HN/Reddit are still king
- capableweb 4y agoIf you can detect AI-generated text you can detect human-"generated" text, and vice-versa. But currently, there is no way to do either.
- alephxyz 4y agoOnly if your detection system is "perfect". Personally I'd be fine with something with high precision and low recall, even if it means forcing users to rewrite their content to make it less AI-like (like /r9k/ on steroids).
- waboremo 4y agoFunny you mention this because more threads/comments on Reddit are bot created. It's unfortunate that the thing that killed forums is now being filled with bots replying to each other and posting threads. It's alright for now, but not so sure how long that will remain. I already stumbled upon some of them in a niche subreddit. Easy to detect now, due to the uncanny way it interjects "facts" into it's comments that are supposed to mimic a reddit rant. In the long term when this stops being so easy to detect, I think a lot of people will start questioning why use Reddit (or similar places) at all. I think also if you hate SEO spam, you might want to brace yourself for chatGPT enabled SEO spam. It's so "good" you might think "Dr. Brandon Amersmith" is a real person and take horrible advice from him.
- ChatGTP 4y agoI really think our efforts as a species would be really better off going towards climate change mitigation and adaption, not burning the world up faster by running thousands of not millions of GPUs on AI experiments and moonshots. We need “AI” because we’re collectively stupid.
- CatWChainsaw 4y agoGarbage in garbage out wins again!
- SammyNameTaken 4y agocant help the thought that corporation imposed no-no words will become the lingo to be used by people to identify themselves in a sea of adtrash
- samstave 4y agoI believe the most important things to humanity right now, as it pertains to AI ;; 1.) the ability to MATHS in our heads. 2.) Navigation biologically (internal compass) - and map reading ability. 3.) analog communication methods 4.) FOOD (commerce, growing, etc) - even if we are "'This is Fine'" -- The above is still valuable.
- _8j50 4y agoWhat you will instead find is the development of ML assisted stylometry. Not to detect ML but to detect when the writing style is not in line with how some person or persons (known employees or students for example) normally write. Or you might be required to provide access to your social media to compare how you normally post stuff.
- px43 4y agoOne under-discussed component of Snowden's XKEYSCORE leak was the claim made in one of the presentations that the NSA has enough stylometry data collected that, give some piece of text, they can not only tell you who wrote it, but they can also tell you where they wrote it (in 2008). Inferred from the "Document Tracking" slide about 2/3rds of the way in. https://www.theguardian.com/world/interactive/2013/jul/31/nsa-xkeyscore-program-full-presentation https://www.theguardian.com/world/interactive/2013/jul/31/ns... It will be interesting to see if any research projects start seriously looking into styleometry using LLMs, but that's basically what you'd need to do to actually generate something useful detection-wise. Ideally it would light up a document identifying the style of known humans, as well as the style of various LLMs. If teachers actually start cracking down on this sort of thing, I'm sure people will fine tune models on their own text, ensuring that anything written uses their personal style, but we can tackle that once we get there.
- mistymountains 4y agoYou could take a similar approach to detection (hypothetically). You provide writing samples a LLM is fine tuned on, then future texts could possibly be filtered by perplexity or similar to detect obvious differences in style. Of course, there would be side effects, and ideally we don’t start creating incentives where we don’t need to. Man, I’m glad I learned to write before all this.
- polishdude20 4y agoHow does it know where it was written?
- zx8080 4y agoGeo-ip? Joking.
- Atlas22 4y agoJust a guess, but it would not be surprising if there are variations in stylistic writing that are influenced by environment (location, people interacted with, air quality, etc). I have certainly observed it happen with myself based on who I've interacted with recently.
- 1vuio0pswjnm7 4y agoHow will websites and third party intermediaries, e.g., so-called "tech" companies, distinguish users submitting human-generated input from users submitting AI-generated input.
- YetAnotherNick 4y agoFirst party platform like Google, Facebook or office 365 easily could with high accuracy by measuring typing rate of user, recording speech etc. At some point, data in the internet might not be trusted at all and the data about the source of the text could be worth billions.
- 1vuio0pswjnm7 4y agoTyping rate could be controlled using something like a USB Rubber Ducky. A user could emulate someone else's typing rate. As for speech, that could be synthesised with something like VALL-E.
- YetAnotherNick 4y agoI said high enough accuracy. Most people won't want to intentionally do the wrong thing. Simple option is that someone could even handtype GPT content, but less than 0.1% or so will do that.
- 1vuio0pswjnm7 4y agoWhat's "the wrong thing".
- gregw2 4y agoInteresting. I'm curious if -- and to what extent -- these detection techniques overcome the techniques Scott Aaronson and Hendrik Kirchner are working on during Scott's sabbatical at OpenAI (https://www.scottaaronson.com/talks/watermark.ppt https://www.scottaaronson.com/talks/watermark.ppt , referenced at https://scottaaronson.blog/?p=7042 https://scottaaronson.blog/?p=7042 )
- deleted 4y ago[deleted]
- IIAOPSW 4y agoSounds like generator-adversary training with extra steps.
- twawaaay 4y agoNope, it can't. Detecting AI-generated text is an adversarial process. Meaning there are going to exist people interested in avoiding detection. This just ensures there is never going to be a reliable solution. There will be some solutions but none that will be reliable. We still have counterfeit banknotes and email spam. Seems like super easy problem to fix but there are always some banknotes and always some email spam that avoid detection. And this is simply because whenever somebody finds a way to close the current hole, somebody else finds a way to open another.
- nextaccountic 4y ago> email spam Even after a decade, it's really, really rare email spam to get past the gmail filter (at least in my experience)
- capableweb 4y agoI used to agree with you but seems like the gmail filter is getting worse and worse, at least it has for me the last few years. I mark everything that manages to bypass it as spam, but the day after almost identical spam messages manage to break through anyway...
- m00x 4y agoIt has to do more with things outside of the text itself. Usually it's because it's of popular spam subjects, spoofed domains, untrustworthy email servers, etc.
- anonzzzies 4y agoThere is targeted spam now in my own language and location; it gets past the filters because it doesn't look like spam at all. I think most spam gets flagged as spam, not because the text context but because of the from/to server / domain / email address.
- hammyhavoc 4y agoEver since December, I've noticed a massive uptick in spam getting past my Gmail filter. We are talking 100+ emails per day selling sexual enhancement drugs, marital aids, conspiranoid prepper nonsense, money mule scams, crypto extortion and various other drivel. What they all have in common is that they're seemingly going through some kind of spinning algorithm to word them differently enough to avoid the filter picking them up.
- cat_plus_plus 4y agoNope. "Aaaaaaaaaaaaaaaaaaaaaaaa". Is this an AI generated text? ChatGPT is a glorified form letter, and there is no way to detect form letters if you only have one sample. I remember broken form letter e-mails starting with "Dear null" and similarly broken AI responses can be detected. If ChatGPT model never changed, one can eventually infer certain signatures in its responses. But of course it will be technically improved and trained with more data over time. How can one detect an arbitrary moving target?
- paulcole 4y agoThe article argues that the current detection methods for AI-generated text are unreliable, which poses a significant risk for the malicious use of LLMs. This is a controversial topic, as some may argue that the benefits of LLMs outweigh the potential risks. However, as the technology continues to advance, we need to have an honest conversation about the ethical use of AI-generated text and the potential consequences of unregulated use. Ultimately, it's up to society to decide whether the risks are worth taking.
- ShamelessC 4y agoIs that you, ChatGPT?
- cyrialize 4y agoI was thinking the same thing. If you ask ChatGPT something without asking for a change in style of response it seems to always spit out a response like this. A format where there’s an intro, middle sentence, middle sentence that is about the opposing view, concluding sentence. Written in a slightly persuasive non-combatant way.
- atleastoptimal 4y agoIn the future students will deliberately make wildly improbable word choices in their essays so as not to set off false positives for AI checkers.
- bjt2n3904 4y agoCan't wait for this to bring about a fresh hell of emergent human behavior. Hmm. My posts on social media keep getting weighted down because it seems too much like ChatGPT. That means (laughing) I need to start writing in a very strange (weird) style so that I throw off the AI-detection algorithms. Or gosh, students writing papers, and the false positives there... "Sorry kid, Turn-It-In says your paper was written by an AI, try again."
- hardchaos 4y ago"In this paper, we show both theoretically and empirically, that these state-of-the-art detectors cannot reliably detect LLM outputs in practical scenarios." This was my finding as well. I abandoned this effort when I found that any created model would be inaccurate on small texts and could be circumvented with a bit of prompt engineering.
- deleted 4y ago[deleted]
- owenpalmer 4y agoThe only way it's possible is if the LLM that generated the text is proprietary and the company behind it provides a way to see if a given text sample was ever generated by said LLM. Otherwise, no. It's definitionally impossible.
- MizuBRDragon 4y agoIt's already hard to detect intention and sarcasm over the web, imagine identifying AI-generated text.