3 ms·
Some comments here point at impossibility results, but after screening hundreds of job applications at work, it's not hard to pick out the LLM writing, even wit
by ruuda 2y ago
Some comments here point at impossibility results, but after screening hundreds of job applications at work, it's not hard to pick out the LLM writing, even without watermark. My internal LLM detector is now so sensitive that I can tell when my confirmed-human colleagues used an LLM to rephrase something when it's longer than just one sentence. The writing style is just so different.
Maybe if you prompt it right, it can do a better job of masking itself, but people don't seem to do that.
- auggierose 2y agoSo, how many times did you actually get the confirmation that an LLM has/has not been used? My guess is zero times. So, you are not describing an experiment here, you are just describing how you built up your internal bias.
- tessierashpool9 2y agothis is an important realization!
- kuhewa 2y agoProbably not entirely fair. e.g. After enough sentences it is trivially easy to identify LLM output. So you repeatedly get the opportunity to test a sentence or two, guess the provenance and then realise it is the first sentence in several paragraphs of generated output.
- exe34 2y agowithout external validation, you still just have a gut feeling. it doesn't matter if it's the first or thousandth. I can tell if somebody ate cornflakes or oatmeal for breakfast just by looking at how they walk. I'm always right. you better believe me. I've seen thousands of people walk.
- kuhewa 2y agoIf the cornflakes had been sitting in old milk for a week, then yes, I would believe you could tell by how they walk, either hunched over with a cramping stomach or with shit pants. LLM text is quite similar.
- exe34 2y agoYes because it would be completely different if it were oatmeal.
- kuhewa 2y agoIf the oatmeal hadn't sat in old milk for a week, yes, it would not likely cause gi distress
- exe34 2y agoand yet if it had, it would.
- kuhewa 2y agoLet's make one of your breakfast foods eggs. Sometimes you won't notice from the walk if they just had normal eggs. However, when they are also rotten and were over indulged in, you can tell from the person's walk that they ate them for breakfast and due to the noxious sulfur smell emanating from the diarrhoea in the person's business casual slacks, you can tell with a high degree of confidence it wasn't the rotten milk cornflakes, but one of a very few number of sulfur-rich foods, probably eggs. Bad human resume text and overuse of unmodified LLM output are both detectable, but they are detectable because they are bad in quite different ways. Regarding the original resume reader's notion that they can detect LLM text with a high degree of accuracy, it is not their LLM output detection specificity I would take issue with (similarly, despite stating validation is critical, I would bet you, too, are pretty confident when you see an entire page of blogspam or marketih copy that you regard as LLM generated despite it rarely being marked as such). Rather, it is their sensitivity, as I am sure occasional use and especially slightly modified output from LLMs gets by them now and again without them knowing.