10 ms·
> What exactly makes anyone think that they can detect an LLM that is outputting text? When you read LLM output, you can often tell. The source of the notion i
by wheybags 3y ago
> What exactly makes anyone think that they can detect an LLM that is outputting text?
When you read LLM output, you can often tell. The source of the notion is that we can do it pretty well ourselves, so if the AIs are so magical then they should be able to do it too (not saying I agree, but it is a pretty clear line of logic).
- alwayslikethis 3y agoI think ChatGPT is trained to speak in a certain tone. I played with other models before, and they seem to not have this issue, and they seem much less distinctive for me.
- hutzlibu 3y agoAlso you can just tell ChatGPT to change style.
- horsawlarway 3y agoThis is the thing that kills me - people who think they can reliably detect LLM output are fooling themselves. At best - you can look for logical fallacies and false facts in the output, and use that to guess, but realistically - people are fucking bad at detecting it, including all these HN posters who keep claiming they can do it reliably...
- iinnPP 3y agoAnd we wonder why LLM's hallucinate.
- zirgs 3y agoThere's plenty of fallacies and false facts in texts written by real humans.
- IIAOPSW 3y agoYes but the sort of logical inconsistency the machine has is idiosyncratic. There can be some types of contradictions few words apart that it doesn't pick up on. For example, asking it to sort a list into n mutually exclusive categories, it often repeats categories or has the same item in more than one category or completely omits an item or even sometimes lists the same item twice within the same category. You might catch bullshitters making up matters of fact, but no person is oblivious to error of basic task like that.
- pmoriarty 3y agoBut do such tell-tale signs that it was written by an LLM exclude the possibility that a human wrote it? Some humans may just write in that style. Why not?
- dmd 3y agoMore to the point, many humans write in that style, which is where GPT picked it up in the first place.
- jacksnipe 3y agoYea, it’s literally the style that is held up as good writing in the usual American high school setting, at least IME.
- SAI_Peregrinus 3y agoYep, GPT's default "mode" seems to be the bland "five paragraph essay" encouraged by high-school courses. First, it tells you what it's about to describe. Then it describes it, inventing some supporting evidence. Then it restates what it just told you. Sadly many students never move beyond this introductory framework, so their professional writing (which GPT trained on) ends up with the same uninspired style. It's a way to automate the creation of content that even the "lowest common denominator" of students can manage, but it doesn't produce good writing.
- foverzar 3y agoI dunno, maybe people feel like they can tell, but I saw how some people pointed at the use of textbook linking words and argued that it's a tell.
- horsawlarway 3y agoAnyone who claims that they can reliably detect LLM text is lying (probably to themselves). People in general are terrible detectors for this.
- arp242 3y agoVerbatim copies are often not that hard to detect in a large number of cases; there is a tendency to waffle excessively in a way that few people do. This is of course not a fool-proof method, but it's "reliable" in the sense of "you can get it right more often than not". Edited LLM text is much harder: using a LLM to generate some text and then edit it in shape (often by removing extraneous paragraphs, maybe rewriting a few things slightly). Those are basically impossible to detect reliably.
- transcriptase 3y agoI don’t disagree with you, though personally I’ve called out two people on Reddit using ChatGPT to farm karma by pretending to be an expert on some topic… and an hour later they deleted all of their GPT generated replies from their profiles. There’s definitely a pattern for some prompts, where the model uses an obvious format of: - Make generalized statement answering prompt question - Support that statement with 3 or 4 discrete paragraphs that don’t exhibit any personal experience, give examples, or cite statistics - Finish off with a high-school essay style conclusion statement reiterating the introduction with different phrasing, generally beginning with “Overall” or “In conclusion” It’s also comically obvious when suddenly someone who has a reply history full of low-effort juvenile, zero grammar, curse laden posts on video game and meme subs is suddenly writing mini-theses on topics ranging from Swedish forestry management to the chemistry of dyes used in Cambodian textile manufacturing.
- 3y ago
- criley2 3y agoI think that vanilla GPT writes in a style that is easy to identify. There's also this "too-perfect" feel where in most forms of communication, most people make mistakes. They splice commas, they use the wrong word, they subtly misspell something, they kinda lose their thought process, etc. Especially as a teacher, you get used to seeing the, uh, quality of work that gets produced. I mean that for college teachers too. At my decent state college, the quality of essay writing was atrocious and I was given 99/100 for basically "not needing any help". In college! When you compare the average GPT output to what many students turn in at all levels, the difference is highly apparent. This also opens up the reality that smart cheaters will prompt engineer. I just tried on GPT4 > You are an average 10th grade student. You read and write at a 9th grade level. You don't edit your work ever and your writing has run-ons, comma splices, misuse of grammar, misspellings, etc. Say OK if you understand > Write a 5 paragraph essay about spring bird patterns in Alabama. Use an intro paragraph and a conclusion paragraph. Keep it around 500 words. > You are doing a good job of being a 10th grade student, however I want you to do the previous command like a 10th grade student who has been trained in basic essay writing and does not write conversationally. And it's already producing something that is far more believable lol.
- hammyhavoc 3y agoThis would be much more interesting if you posted the output directly, or linked to a pastebin service with it on. The context is pretty key to deriving value from the comment.
- criley2 3y agoSure, I didn't want to spam large outputs here. https://pastebin.com/JcSHhzw8 https://pastebin.com/JcSHhzw8 For the record: these tools are open and available and it takes about 10 seconds for you to replicate my experiment. The value you derive from my comment should be the idea I give you, not the results I generated. My results are good but imperfect, and are the result of very little effort. Someone truly trying to cheat the system could produce way more I imagine.
- 3y ago
- quickthrower2 3y agoArrr! When ye lay yer eyes upon LLM's scribblings, ye can oft discern its origins, matey! The notion be rooted in our own knack for the task, makin' us reckon that if them AIs be truly magical, they should fare just as well. Not sayin' I be agreein', mind ye, but 'tis a line o' reasonin' as clear as a starry night on the high seas!
- philipov 3y agoDid you... just ask ChatGPT to translate the parent comment into pirate speak?
- bryanrasmussen 3y agoor maybe they translated the parent comment into pirate speak on their own, which would not be difficult, to see if anyone accused them of being an LLM text.
- quickthrower2 3y agoI first asked it to reply in the style of Paul Graham - the idea being it would be like a good HN reply, but not obviously a bot. But sadly it did a terrible job of that - rambling on, So I got it to do a terse version, and still bad. So I then said "as a pirate" and this is what came out.
- oneTbrain23 3y agoIn schools, my coursemates and I always plagiarised each other work. We also heavily do it via any other means like having a class pooled together funds for buying work done by outsiders. This span anything from quantum mechanics to even science fair projects. We ended passing many college course with near flying colors. What have we learnt by cheating? How to cheat better. We learnt how rephrase, rearrange assembly codes, reformat, etc. Some of us even develop program to do substitution and rephrasing. This was way before chatgpt. We did well with this skills as we apply it in corporate world plagiarising collegues works and even competitor a product. We even glorified how Samsung did it in early days of smartphone war era. I am sure my juniors in my high school and college will do the same with chatgpt. I can assure you we plagiarised and got away with it near 100% (unless some were lazy enough not to clean up their copy of plagiarise source material). Those that get caught will just have to learn to adapt. When you graduate from you college, ability to plagiarise is very important in your career (as long as non academia). You can tell if you know the persons tendency of dishonesty. If purely on work output, you are at best at coin toss or maybe those very low quality students. Back my days, we already have plagiarise detectors and even bots to detect plagiarised programming assignments. It is hard to tell. Chatgpt will adapt the output to be non perfect. Plagiariser will also adapt like my experience to de-chatgpt it. It has been done. It will be co tinue to do happen.
- smcleod 3y agoWhy didn't you and your course mates want to learn?
- visarga 3y agoShort term perspective, easier to play games than work on projects.
- analog31 3y agoToo late, they cheated through all of the prerequisites.
- antifa 3y agoHonestly, some classes are useless """requirements""". Most classes I wouldn't consider cheating though, because in that case the only person I'd be cheating would be myself.
- joseph_grobbles 3y ago[dead]
- horsawlarway 3y agoExcept people are actually terrible at detecting LLM output. Worse - The entire point of the LLM is that it's generating its output by picking the statistically likely next word... So from a "analyse text and detect forgery" side of things... any company that claims they can do it without false positives is fucking lying through their teeth.
- raincole 3y agoVery typical bias tho. When you can tell, you can tell. When you can't tell, you don't even know you can't tell.
- hammyhavoc 3y agoSimultaneously, I'm also noticing that writing styles in general are beginning to look very ChatGPT-like, even if someone isn't using ChatGPT to write with at that moment in time. If they're a user of it at any other time, or consuming a lot of its output, they tend to mimic what they're exposed to. Just an observation.
- 2devnull 3y agoMy theory is that many “educated” people were never taught to write properly. Gpt is teaching people what they should have learned in grade school: how to write clearly.
- hammyhavoc 3y agoI don't know about that. I think ChatGPT's output is very repetitive and fluffy, and their style thus becomes similar—not necessarily a positive habit.
- jonhohle 3y agoAnd that writing will be used to train future models until a stiff, logical proof/Rube-Goldberg style of writing becomes the standard everyone adopts (hopefully not).
- hammyhavoc 3y agoWith logical proof styles, that requires active thought, reasoning and rationalizing, amidst other things. It's easy to blag things with fluff and repetition, which is why so many people are hired out of their depth into inappropriate positions. Average ChatGPT output for the most part reminds me of the average British politician who can waffle on until you stop them, talking about all the aspects of something, but never really answering your question with all that much substance and depth, sometimes not even answering it whatsoever. Very "30,000 foot view".
- seanmcdirmid 3y ago
- usrbinbash 3y ago> The source of the notion is that we can do it pretty well ourselves Well, that notion is wrong, otherwise, what's the explanation for something like this: https://www.theregister.com/2023/05/17/university_chatgpt_grades/ https://www.theregister.com/2023/05/17/university_chatgpt_gr...
- strken 3y agoWe may or may not be able to detect whether LLMs wrote something, but I'd bet my life savings that when an administrator at the department of agricultural sciences and natural resources asks ChatGPT whether a particular essay was written by GPT, there will be significant numbers of false positives and false negatives.
- akiselev 3y agoYou can tell if it’s vanilla ChatGPT but the second you tell it to write in another style and tone, it changes completely. It’s very interesting that it sounds like a generic high school essay by default, but that’s without any extra instruction. For example, ask it to write in the style of Christopher Hitchens, Charles Bukowski, or Hunter S Thompson, let alone more extreme examples like Shakespeare or Dante
- causality0 3y agoIt does have a lot of "shallow wells" where if you ask it to make a random list of something or pick a random example of something it will give similar answers every time. For example, whenever it has to come up with some sort of enemy or attacker or opponent it's very drawn to skeletons and giant spiders regardless of context.
- crooked-v 3y agoAlso, ask it to name a character and you'll see a very repetitive pool of names come up.
- deleted 3y ago[deleted]
- mrangle 3y agoThat LLM's are AI is propaganda. It follows that so is any "magical" property conferred.
- ilyt 3y agoMost of what "you can tell" texts could also pretty convincingly pass as human writing in corporate speech