3 ms·
There's a lot of people with strong opinions whether these detectors can or cannot work, but I implore you to do the experiment I just did. Go to the top 10 you
by PumpkinSpice 3y ago
There's a lot of people with strong opinions whether these detectors can or cannot work, but I implore you to do the experiment I just did. Go to the top 10 you find in a Google search and paste a sufficiently long sample of your prose. Not a random HN comment - at least 200-400 words of normal, coherent text.
It's a game of cat and mouse in the sense that you can build LLMs specifically optimized for evading the current crop of detectors, but in my testing, they work pretty darn well in the general case. While they might not reliably pick on all LLM text, and while there's sometimes a couple of words in human-generated writing that causes them to output a low but non-zero probability of LLM content, they do not rate human-generated text as "99% AI". Especially not across multiple writing samples.
The most likely story here, I suspect, is that the person leaned on LLMs for commissioned writing and is now trying to save face. The secrecy of the models works both ways, right? And frankly - how often do you see people in HN, or people who do commissioned writing, admit in private that they're using ChatGPT? It's cropping up all over the place.
Note that I'm not commenting on the ethics, fairness, or transparency of tools like that. I'm just saying they work far better than you might be suspecting.
- YetAnotherNick 3y agoYes, I also would like to find an article pre 2020 where AI detector says 99% AI written, because in my small sample I couldn't find any.
- simonw 3y agoOne flaw in this experiment is that most of us here aren't professional writers, which means that text we produce ourselves is probably less likely to trigger a false positive just because we didn't clean up the spelling, grammar and general writing style to the point that it might look like like an LLM wrote it.
- sebzim4500 3y agoI did the same experiment, was pleased to see that all the human-written articles I copied in were correctly identified as human written. On the other hand, I tried three AI generated texts (>500 words each) and only one was marked as AI generated (it was so obviously AI generated that it would have stood out to me).