12 ms·
If a GPT detector has any false positives at all it will disadvantage people who are already disadvantaged. They are the least likely to be capable of defending
by iinnPP 3y ago
If a GPT detector has any false positives at all it will disadvantage people who are already disadvantaged. They are the least likely to be capable of defending themselves and the least likely to be heard if they do try to defend themselves.
Not to mention the fact that being called non-human is most definitely going to offend some people.
What exactly makes anyone think that they can detect an LLM that is outputting text? The notion seems absurd yet it keeps coming up.
- mtlmtlmtlmtl 3y ago>What exactly ... Probably the fact that if they admit the reality, they have to think about some difficult and profound questions. It's much easier just to posit an imaginary future technology and decide that will solve it.
- wheybags 3y ago> What exactly makes anyone think that they can detect an LLM that is outputting text? When you read LLM output, you can often tell. The source of the notion is that we can do it pretty well ourselves, so if the AIs are so magical then they should be able to do it too (not saying I agree, but it is a pretty clear line of logic).
- alwayslikethis 3y agoI think ChatGPT is trained to speak in a certain tone. I played with other models before, and they seem to not have this issue, and they seem much less distinctive for me.
- hutzlibu 3y agoAlso you can just tell ChatGPT to change style.
- horsawlarway 3y agoThis is the thing that kills me - people who think they can reliably detect LLM output are fooling themselves. At best - you can look for logical fallacies and false facts in the output, and use that to guess, but realistically - people are fucking bad at detecting it, including all these HN posters who keep claiming they can do it reliably...
- iinnPP 3y agoAnd we wonder why LLM's hallucinate.
- zirgs 3y agoThere's plenty of fallacies and false facts in texts written by real humans.
- IIAOPSW 3y agoYes but the sort of logical inconsistency the machine has is idiosyncratic. There can be some types of contradictions few words apart that it doesn't pick up on. For example, asking it to sort a list into n mutually exclusive categories, it often repeats categories or has the same item in more than one category or completely omits an item or even sometimes lists the same item twice within the same category. You might catch bullshitters making up matters of fact, but no person is oblivious to error of basic task like that.
- pmoriarty 3y agoBut do such tell-tale signs that it was written by an LLM exclude the possibility that a human wrote it? Some humans may just write in that style. Why not?
- dmd 3y agoMore to the point, many humans write in that style, which is where GPT picked it up in the first place.
- jacksnipe 3y agoYea, it’s literally the style that is held up as good writing in the usual American high school setting, at least IME.
- SAI_Peregrinus 3y agoYep, GPT's default "mode" seems to be the bland "five paragraph essay" encouraged by high-school courses. First, it tells you what it's about to describe. Then it describes it, inventing some supporting evidence. Then it restates what it just told you. Sadly many students never move beyond this introductory framework, so their professional writing (which GPT trained on) ends up with the same uninspired style. It's a way to automate the creation of content that even the "lowest common denominator" of students can manage, but it doesn't produce good writing.
- foverzar 3y agoI dunno, maybe people feel like they can tell, but I saw how some people pointed at the use of textbook linking words and argued that it's a tell.
- horsawlarway 3y agoAnyone who claims that they can reliably detect LLM text is lying (probably to themselves). People in general are terrible detectors for this.
- arp242 3y agoVerbatim copies are often not that hard to detect in a large number of cases; there is a tendency to waffle excessively in a way that few people do. This is of course not a fool-proof method, but it's "reliable" in the sense of "you can get it right more often than not". Edited LLM text is much harder: using a LLM to generate some text and then edit it in shape (often by removing extraneous paragraphs, maybe rewriting a few things slightly). Those are basically impossible to detect reliably.
- transcriptase 3y agoI don’t disagree with you, though personally I’ve called out two people on Reddit using ChatGPT to farm karma by pretending to be an expert on some topic… and an hour later they deleted all of their GPT generated replies from their profiles. There’s definitely a pattern for some prompts, where the model uses an obvious format of: - Make generalized statement answering prompt question - Support that statement with 3 or 4 discrete paragraphs that don’t exhibit any personal experience, give examples, or cite statistics - Finish off with a high-school essay style conclusion statement reiterating the introduction with different phrasing, generally beginning with “Overall” or “In conclusion” It’s also comically obvious when suddenly someone who has a reply history full of low-effort juvenile, zero grammar, curse laden posts on video game and meme subs is suddenly writing mini-theses on topics ranging from Swedish forestry management to the chemistry of dyes used in Cambodian textile manufacturing.
- 3y ago
- criley2 3y agoI think that vanilla GPT writes in a style that is easy to identify. There's also this "too-perfect" feel where in most forms of communication, most people make mistakes. They splice commas, they use the wrong word, they subtly misspell something, they kinda lose their thought process, etc. Especially as a teacher, you get used to seeing the, uh, quality of work that gets produced. I mean that for college teachers too. At my decent state college, the quality of essay writing was atrocious and I was given 99/100 for basically "not needing any help". In college! When you compare the average GPT output to what many students turn in at all levels, the difference is highly apparent. This also opens up the reality that smart cheaters will prompt engineer. I just tried on GPT4 > You are an average 10th grade student. You read and write at a 9th grade level. You don't edit your work ever and your writing has run-ons, comma splices, misuse of grammar, misspellings, etc. Say OK if you understand > Write a 5 paragraph essay about spring bird patterns in Alabama. Use an intro paragraph and a conclusion paragraph. Keep it around 500 words. > You are doing a good job of being a 10th grade student, however I want you to do the previous command like a 10th grade student who has been trained in basic essay writing and does not write conversationally. And it's already producing something that is far more believable lol.
- hammyhavoc 3y agoThis would be much more interesting if you posted the output directly, or linked to a pastebin service with it on. The context is pretty key to deriving value from the comment.
- criley2 3y agoSure, I didn't want to spam large outputs here. https://pastebin.com/JcSHhzw8 https://pastebin.com/JcSHhzw8 For the record: these tools are open and available and it takes about 10 seconds for you to replicate my experiment. The value you derive from my comment should be the idea I give you, not the results I generated. My results are good but imperfect, and are the result of very little effort. Someone truly trying to cheat the system could produce way more I imagine.
- 3y ago
- quickthrower2 3y agoArrr! When ye lay yer eyes upon LLM's scribblings, ye can oft discern its origins, matey! The notion be rooted in our own knack for the task, makin' us reckon that if them AIs be truly magical, they should fare just as well. Not sayin' I be agreein', mind ye, but 'tis a line o' reasonin' as clear as a starry night on the high seas!
- philipov 3y agoDid you... just ask ChatGPT to translate the parent comment into pirate speak?
- bryanrasmussen 3y agoor maybe they translated the parent comment into pirate speak on their own, which would not be difficult, to see if anyone accused them of being an LLM text.
- quickthrower2 3y agoI first asked it to reply in the style of Paul Graham - the idea being it would be like a good HN reply, but not obviously a bot. But sadly it did a terrible job of that - rambling on, So I got it to do a terse version, and still bad. So I then said "as a pirate" and this is what came out.
- oneTbrain23 3y agoIn schools, my coursemates and I always plagiarised each other work. We also heavily do it via any other means like having a class pooled together funds for buying work done by outsiders. This span anything from quantum mechanics to even science fair projects. We ended passing many college course with near flying colors. What have we learnt by cheating? How to cheat better. We learnt how rephrase, rearrange assembly codes, reformat, etc. Some of us even develop program to do substitution and rephrasing. This was way before chatgpt. We did well with this skills as we apply it in corporate world plagiarising collegues works and even competitor a product. We even glorified how Samsung did it in early days of smartphone war era. I am sure my juniors in my high school and college will do the same with chatgpt. I can assure you we plagiarised and got away with it near 100% (unless some were lazy enough not to clean up their copy of plagiarise source material). Those that get caught will just have to learn to adapt. When you graduate from you college, ability to plagiarise is very important in your career (as long as non academia). You can tell if you know the persons tendency of dishonesty. If purely on work output, you are at best at coin toss or maybe those very low quality students. Back my days, we already have plagiarise detectors and even bots to detect plagiarised programming assignments. It is hard to tell. Chatgpt will adapt the output to be non perfect. Plagiariser will also adapt like my experience to de-chatgpt it. It has been done. It will be co tinue to do happen.
- smcleod 3y agoWhy didn't you and your course mates want to learn?
- visarga 3y agoShort term perspective, easier to play games than work on projects.
- analog31 3y agoToo late, they cheated through all of the prerequisites.
- antifa 3y agoHonestly, some classes are useless """requirements""". Most classes I wouldn't consider cheating though, because in that case the only person I'd be cheating would be myself.
- joseph_grobbles 3y ago[dead]
- horsawlarway 3y agoExcept people are actually terrible at detecting LLM output. Worse - The entire point of the LLM is that it's generating its output by picking the statistically likely next word... So from a "analyse text and detect forgery" side of things... any company that claims they can do it without false positives is fucking lying through their teeth.
- raincole 3y agoVery typical bias tho. When you can tell, you can tell. When you can't tell, you don't even know you can't tell.
- hammyhavoc 3y agoSimultaneously, I'm also noticing that writing styles in general are beginning to look very ChatGPT-like, even if someone isn't using ChatGPT to write with at that moment in time. If they're a user of it at any other time, or consuming a lot of its output, they tend to mimic what they're exposed to. Just an observation.
- 2devnull 3y agoMy theory is that many “educated” people were never taught to write properly. Gpt is teaching people what they should have learned in grade school: how to write clearly.
- hammyhavoc 3y agoI don't know about that. I think ChatGPT's output is very repetitive and fluffy, and their style thus becomes similar—not necessarily a positive habit.
- jonhohle 3y agoAnd that writing will be used to train future models until a stiff, logical proof/Rube-Goldberg style of writing becomes the standard everyone adopts (hopefully not).
- hammyhavoc 3y agoWith logical proof styles, that requires active thought, reasoning and rationalizing, amidst other things. It's easy to blag things with fluff and repetition, which is why so many people are hired out of their depth into inappropriate positions. Average ChatGPT output for the most part reminds me of the average British politician who can waffle on until you stop them, talking about all the aspects of something, but never really answering your question with all that much substance and depth, sometimes not even answering it whatsoever. Very "30,000 foot view".
- seanmcdirmid 3y ago
- usrbinbash 3y ago> The source of the notion is that we can do it pretty well ourselves Well, that notion is wrong, otherwise, what's the explanation for something like this: https://www.theregister.com/2023/05/17/university_chatgpt_grades/ https://www.theregister.com/2023/05/17/university_chatgpt_gr...
- strken 3y agoWe may or may not be able to detect whether LLMs wrote something, but I'd bet my life savings that when an administrator at the department of agricultural sciences and natural resources asks ChatGPT whether a particular essay was written by GPT, there will be significant numbers of false positives and false negatives.
- akiselev 3y agoYou can tell if it’s vanilla ChatGPT but the second you tell it to write in another style and tone, it changes completely. It’s very interesting that it sounds like a generic high school essay by default, but that’s without any extra instruction. For example, ask it to write in the style of Christopher Hitchens, Charles Bukowski, or Hunter S Thompson, let alone more extreme examples like Shakespeare or Dante
- causality0 3y agoIt does have a lot of "shallow wells" where if you ask it to make a random list of something or pick a random example of something it will give similar answers every time. For example, whenever it has to come up with some sort of enemy or attacker or opponent it's very drawn to skeletons and giant spiders regardless of context.
- crooked-v 3y agoAlso, ask it to name a character and you'll see a very repetitive pool of names come up.
- deleted 3y ago[deleted]
- mrangle 3y agoThat LLM's are AI is propaganda. It follows that so is any "magical" property conferred.
- ilyt 3y agoMost of what "you can tell" texts could also pretty convincingly pass as human writing in corporate speech
- kvetching 3y agoPrior to GPT, word processors would catch when your grammar was off for multiple words and give you suggestions to fix it. As students begin using the new GPT enhanced word processors, these suggestions will only increase. Professors have a tough job ahead and they aren't always smart enough to use these GPT Detectors[1]. I think one solution would be a word processor that records the process of writing a paper and you need to turn in your paper along with your recording. Of course this is going to create added stress but what else do we do? There's going to be GPT scramblers that remove watermarks. [1] https://www.businessinsider.com/professor-fails-students-after-chatgpt-falsely-said-it-wrote-papers-2023-5 https://www.businessinsider.com/professor-fails-students-aft...
- ThatMedicIsASpy 3y agoNothing stops you from working on document A without help and document B with help and just write/copy back into A.
- kvetching 3y agoYou're right. We need full surveillance, multiple cameras on the student and full recording of the entire computer
- adhesive_wombat 3y agoI knew those Intel Management Engine and TPM things would come in handy one day!
- egeozcan 3y agoThey can also use the computer belonging to any family member. In a better world there should be cameras everywhere so we can detect a student not studying as much as the result they bring would require. Ideally though, mind probes should come into play to find out if they really produced the work.
- 3y ago
- thatsadude 3y ago> What exactly makes anyone think that they can detect an LLM that is outputting text? The notion seems absurd yet it keeps coming up. Publish or perish!
- teeray 3y agoIt’s unfortunate that the economics involved means that the purchasers of such tools will likely want a false negative rate of 0, likely at the expense of victims of such tools suffering the resultant false positive tradeoff.
- Nowado 3y agoYou may want to read on history of polygraph. Notion that a tool giving yes/no answer needs to have it's answers relate to ground truth is very academic and generally frown upon by gatekeepers.
- canadianfella 3y ago[dead]
- jstanley 3y ago> it will disadvantage people who are already disadvantaged. They are the least likely to be capable of defending themselves and the least likely to be heard if they do try to defend themselves. Can you come up with any type of system that this does not apply to? https://www.lesswrong.com/posts/G5eMM3Wp3hbCuKKPE/proving-too-much https://www.lesswrong.com/posts/G5eMM3Wp3hbCuKKPE/proving-to...
- iinnPP 3y agoAny system designed to help disadvantaged people that has hit the mark. The point being made is that the idea of detecting AI text is absurd. It won't work it can't work. The point about it having an increased negative impact on those who are already at a disadvantage is important. Smart people don't need additional barriers to becoming benefits to society and promoting any tool that creates those barriers and does literally nothing else is terrible. And no, not all systems meet this criteria.
- bagacrap 3y agoI'm losing the train of thought here. Smart people are the ones least likely to be able to defend themselves? I think it's enough to say that it harms some people (ESL) more than others and leave it at that.
- lmm 3y ago> Any system designed to help disadvantaged people that has hit the mark. Are there any such systems? Don't all of them have the same problem of the very people who most need the system being the least capable of using it?
- eternalban 3y ago> What exactly makes anyone think that they can detect an LLM that is outputting text? The notion seems absurd yet it keeps coming up. My sense of the general idea (non-authorative): Since the sequence emitted by an LLM is probabilistic completion i.e. predict the next word, the examiner can also do the same by progressively processing the text. Given the assumption that the semantic relations extracted from training corpus should be fairly universal for a given domain at the output level (even though distinct LLMs will likely have distinct embedding spaces), then the examiner LLM should be able to assign probabilities to the predicted words. The idea is that a genuine human produced text will have idiosyncrasies that are -not- probabilistically optimal and the examiner can establish a sort of 'distant from probable mean' measure, with the expectation that LLM produced text should be 'closer' to the examiner's predictions of 'the next word'. The problem (if above is correct) then is the missing 'prompt' and meta-instruction embedded therein. Those should ("engineering") affect the output, possibly skewing the distance measure, thus defeating the examiner. But of course, say in context of academia, the examiner can 'guess' as to some aspects of the prompt as well. For example, if you are examining papers for a specific assignment, the examiner can self-prompt as well. "An essay on Hume's position on the knowledge of the self".
- circuit10 3y agoThat only works if the temperature setting is low. If you set it to 1 the it will pick something that’s 1% likely 1% of the time, for example, which should match human text
- eternalban 3y agoDefinitely, should have mentioned that. However it is interesting that using temperature may lower the quality of the output and you may get a C and not the hoped for A+.
- swores 3y agoCould you run it once to get the A+ version, then feed it back that version saying "change x% of these words to be unlikely choices in a way that keeps all the meanings of the essay" to avoid letting the low-probability words hamper the main contents, just the way it's described?
- onos 3y agoI saw this (now aged by four months) post from openAI on a method of detection. It’s not perfect but there is certainly some signal present. https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text https://openai.com/blog/new-ai-classifier-for-indicating-ai-...
- coconuthacker42 3y agowell the system we live under doesn't give a crap about how absurd a notion is, and surely doens't care about educating people against it. as long as it's profitable it's gonna be made. In multiple scenarios it makes a ton of more sense economically to make the misinformation pool larger, and as a person who values truth and knowledge and the free access to correct information and believes that this will be the force which drives our species further, supporting this system is a contradiction. maybe i am a commie now? How would I know? The west has dismantled all the other systems to the point it is impossible for me to imagine living outside it.
- golergka 3y ago> What exactly makes anyone think that they can detect an LLM that is outputting text? Watermarking. On each word of the output, you randomly split all possible words into two groups and only generate output using one of them. If you get a text of 1000 words that exactly follow the secret sequence of the groups, you can be sure that it's generated by this LLM with one in 2^1000 chance of error.
- dragonwriter 3y ago> Watermarking. Watermarking only works if every LLM system available does it and it is impractical for third parties to spin up and use systems that don’t, otherwise it only detects content that opts in to being detected.
- Der_Einzige 3y agoThis technique is trivially fooled by anything which messes with the Logits, which is what everyone is now doing in the non chatGPT world.
- explorer83 3y agoIf its detecting non-native English as GPT produced then wouldnt that mean GPT is modeled on non-native English to produce similar outputs? If it is NOT detecting native English as GPT produced then wouldn't that mean it's showing up different than the training data outputs?
- explorer83 3y agoIt's funny I'm being downvoted. Probably out of a sense that I'm ignorant. Yet I've worked on large data labeling teams. And if you think anyone outside of the military is paying for very large native English speaking teams to label data I got a bridge to sell you. No one outside of special projects are paying large teams the wages of first world country standard.