6 ms·

AI writing detectors are snake oil

mirrorlake 2y ago
Anecdotally, English/History/Communications professors are confirming cheaters with them because they find it easy to identify false information. The red flags are so obvious that the checker tools are just a formality: student papers now have fake URLs and fake citations. Students will boldly submit college papers which have paragraphs about nonexistent characters, or make false claims about what characters did in a story. The e-mail correspondence goes like this: "Hello Professor, I'd like to meet to discuss my failing grade. I didn't know that using ChatGPT was bad, can I have some points back or rewrite my essay?"
A_D_E_P_T 2y ago
Yeah but they "detect" the characteristic AI style: The limited way it structures sentences, the way it lays out arguments, the way it tends to close with an "in conclusion" paragraph, certain word choices, etc. o1-Pro doesn't do any of that. It writes like a human. Damnit. It's too good. It just saved me ~6 hours in drafting a complicated and bespoke legal document. Before you ask: I know what I'm doing, and it did a better job in five minutes than I could have done over those six hours. Homework is over. Journalism is over. A large slice of the legal profession is over. For real this time.
- dr_dshiv 2y ago
  > Before you ask: I know what I'm doing, and it did a better job in five minutes than I could have done over those six hours. Seems like lawyers could do more faster because they know what they are doing. Experts dont get replaced, they get tools to amplify and extend their expertise
  - energy123 2y ago
    Replacement doesn't happen only if the demand for their services scales proportional to the productivity improvements, which is true sometimes but not always true, and is less likely to be true if the productivity improvements are very large.
- dgacmu 2y ago
  ahh, but: > I know what I'm doing Is exactly the key element in being able to use spicy autocomplete. If you don't know what you're doing, it's going to bite you and you won't know it until it's too late. "GPT messed up the contract" is not an argument I would envy anyone presenting in court or to their employer. :) (I say this mostly from using tools like copilot)
  - Sleaker 2y ago
    Well... Lawyers already got slapped for filings straight from ai output. So not new territory as far as that's concerned :)
- ionwake 2y ago
  sold. Ill buy it, thx for review. Edit> Its good. Thanks again for ur review.
- mongol 2y ago
  Journalism is not only about writing. It is about sources, talking to people, being on the ground, connecting dots, asking the right questions. Journalists can certainly benefit from AI and good journalists will have jobs for a long time still.
  - koyote 2y ago
    While the above is true, I'd say the majority of what passes as journalism these days has none of the above and the writing is below what an AI writer could produce :( It's actually surprising how many articles on 'respected' news websites have typos. You'd think there would be automated spellcheckers and at least one 'peer review' (probably too much to ask an actual editor to review the article these days...).
    - JohnBooty 2y ago
      It's actually surprising how many articles on 'respected' news websites have typos. Well, that's why they're respected! The typos let you know they're not using AI!
    - SoftTalker 2y ago
      Mainstream news today is written for an 8th grade reading ability. Many adults would lose interest otherwise, and the generation that grew up reading little more than social media posts will be even worse. AI can handle that sort of writing just fine, readers won't care about the formulaic writing style.
  - umeshunni 2y ago
    These days, most journalism is turning reddit posts and tweets into long form articles with some additional context.
    - SoftTalker 2y ago
      So AI could actually turn journalism more into what it originally was: reporting what is going on, rather than reading and rewriting information from other sources. Interesting possibility.
- j45 2y ago
  It still needs to be driven by someone who knows what they're doing. Just like when software that was coming out, it may have ended jobs. But it also helped get things done that wouldn't otherwise, or as much. In this case, equipping a capable lawyer to be 20x is more like an iron man suit, which is OK. If you can get more done, wit less effort, you are still critical to what's needed.
- solarkraft 2y ago
  > Homework is over. Journalism is over. A large slice of the legal profession is over. For real this time. It just replaces human slop with automated slop. It doesn't automate finding hidden things out just yet, just automates blogspam.
CharlieDigital 2y ago
Startup I'm at has generated a LOT of content using LLMs and once you've reviewed enough of the output, you can easily see specific patterns in the output. Some words/phrases that, by default, it overuses: "dive into", "delve into", "the world of", and others. You correct it with instructions, but it will then find synonyms so there is also a structural pattern to the output that it favors by default. For example, if we tell it "Don't start your writing with 'dive into'", it will just switch to "delve into" or another synonym. Yes, all of this can be corrected if you put enough effort into the prompt and enough iterations to fix all of these tells.
- ballooney 2y ago
  [flagged]
- dartos 2y ago
  All of what you described can change wildly from model to model. Even across different versions of the same model. Maybe a database could be built with “tells” organized by model.
  - liontwist 2y ago
    Exactly. Fixing the old tells just means there are new ones.
  - handfuloflight 2y ago
    > Maybe a database could be built with “tells” organized by model. Automated by the LLMs themselves.
    - dartos 2y ago
      No thanks, I’d like it to be accurate ;) Regular ol tests would do
      - handfuloflight 2y ago
        I should have been more precise. I meant the LLMs would output their tells for you, naturally. But that's obvious.
daemonologist 2y ago
They're not very accurate, but I think snake oil is a bit too far - they're better than guessing at least for the specific model(s) they're trained on. OpenAI's classifier [0] was at 26% recall, 91% precision when it launched, though I don't know what models created the positives in their test set. (Of course they later withdrew that classifier due to its low accuracy, which I think was the right move. When a company offers both an AI Writer and an AI Writing detector people are going to take its predictions as gospel and _that_ is definitely a problem.) All that aside, most models have had a fairly distinctive writing style, particularly when fed no or the same system prompt every time. If o1-Pro blends in more with human writing that's certainly... interesting. [0] https://openai.com/index/new-ai-classifier-for-indicating-ai-written-text/ https://openai.com/index/new-ai-classifier-for-indicating-ai...
dangdetector 2y ago
Doubtful AI writing is obvious as hell.
efficax 2y ago
of course they are. it’s simple: if they worked they would be incorporated into the loss function of the models and then they would no longer work

Anecdotally, English/History/Communications professors are confirming cheaters with them because they find it easy to identify false information. The red flags are so obvious that the checker tools are just a formality: student papers now have fake URLs and fake citations. Students will boldly submit college papers which have paragraphs about nonexistent characters, or make false claims about what characters did in a story. The e-mail correspondence goes like this: "Hello Professor, I'd like to meet to discuss my failing grade. I didn't know that using ChatGPT was bad, can I have some points back or rewrite my essay?"

A_D_E_P_T 2y ago

Yeah but they "detect" the characteristic AI style: The limited way it structures sentences, the way it lays out arguments, the way it tends to close with an "in conclusion" paragraph, certain word choices, etc. o1-Pro doesn't do any of that. It writes like a human. Damnit. It's too good. It just saved me ~6 hours in drafting a complicated and bespoke legal document. Before you ask: I know what I'm doing, and it did a better job in five minutes than I could have done over those six hours. Homework is over. Journalism is over. A large slice of the legal profession is over. For real this time.

dr_dshiv 2y ago
> Before you ask: I know what I'm doing, and it did a better job in five minutes than I could have done over those six hours. Seems like lawyers could do more faster because they know what they are doing. Experts dont get replaced, they get tools to amplify and extend their expertise
- energy123 2y ago
  Replacement doesn't happen only if the demand for their services scales proportional to the productivity improvements, which is true sometimes but not always true, and is less likely to be true if the productivity improvements are very large.
dgacmu 2y ago
ahh, but: > I know what I'm doing Is exactly the key element in being able to use spicy autocomplete. If you don't know what you're doing, it's going to bite you and you won't know it until it's too late. "GPT messed up the contract" is not an argument I would envy anyone presenting in court or to their employer. :) (I say this mostly from using tools like copilot)
- Sleaker 2y ago
  Well... Lawyers already got slapped for filings straight from ai output. So not new territory as far as that's concerned :)
ionwake 2y ago
sold. Ill buy it, thx for review. Edit> Its good. Thanks again for ur review.
mongol 2y ago
Journalism is not only about writing. It is about sources, talking to people, being on the ground, connecting dots, asking the right questions. Journalists can certainly benefit from AI and good journalists will have jobs for a long time still.
- koyote 2y ago
  While the above is true, I'd say the majority of what passes as journalism these days has none of the above and the writing is below what an AI writer could produce :( It's actually surprising how many articles on 'respected' news websites have typos. You'd think there would be automated spellcheckers and at least one 'peer review' (probably too much to ask an actual editor to review the article these days...).
  - JohnBooty 2y ago
    It's actually surprising how many articles on 'respected' news websites have typos. Well, that's why they're respected! The typos let you know they're not using AI!
  - SoftTalker 2y ago
    Mainstream news today is written for an 8th grade reading ability. Many adults would lose interest otherwise, and the generation that grew up reading little more than social media posts will be even worse. AI can handle that sort of writing just fine, readers won't care about the formulaic writing style.
- umeshunni 2y ago
  These days, most journalism is turning reddit posts and tweets into long form articles with some additional context.
  - SoftTalker 2y ago
    So AI could actually turn journalism more into what it originally was: reporting what is going on, rather than reading and rewriting information from other sources. Interesting possibility.
j45 2y ago
It still needs to be driven by someone who knows what they're doing. Just like when software that was coming out, it may have ended jobs. But it also helped get things done that wouldn't otherwise, or as much. In this case, equipping a capable lawyer to be 20x is more like an iron man suit, which is OK. If you can get more done, wit less effort, you are still critical to what's needed.
solarkraft 2y ago
> Homework is over. Journalism is over. A large slice of the legal profession is over. For real this time. It just replaces human slop with automated slop. It doesn't automate finding hidden things out just yet, just automates blogspam.

CharlieDigital 2y ago

Startup I'm at has generated a LOT of content using LLMs and once you've reviewed enough of the output, you can easily see specific patterns in the output. Some words/phrases that, by default, it overuses: "dive into", "delve into", "the world of", and others. You correct it with instructions, but it will then find synonyms so there is also a structural pattern to the output that it favors by default. For example, if we tell it "Don't start your writing with 'dive into'", it will just switch to "delve into" or another synonym. Yes, all of this can be corrected if you put enough effort into the prompt and enough iterations to fix all of these tells.

ballooney 2y ago
[flagged]
dartos 2y ago
All of what you described can change wildly from model to model. Even across different versions of the same model. Maybe a database could be built with “tells” organized by model.
- liontwist 2y ago
  Exactly. Fixing the old tells just means there are new ones.
- handfuloflight 2y ago
  > Maybe a database could be built with “tells” organized by model. Automated by the LLMs themselves.
  - dartos 2y ago
    No thanks, I’d like it to be accurate ;) Regular ol tests would do
    - handfuloflight 2y ago
      I should have been more precise. I meant the LLMs would output their tells for you, naturally. But that's obvious.

daemonologist 2y ago

They're not very accurate, but I think snake oil is a bit too far - they're better than guessing at least for the specific model(s) they're trained on. OpenAI's classifier [0] was at 26% recall, 91% precision when it launched, though I don't know what models created the positives in their test set. (Of course they later withdrew that classifier due to its low accuracy, which I think was the right move. When a company offers both an AI Writer and an AI Writing detector people are going to take its predictions as gospel and _that_ is definitely a problem.) All that aside, most models have had a fairly distinctive writing style, particularly when fed no or the same system prompt every time. If o1-Pro blends in more with human writing that's certainly... interesting. [0] https://openai.com/index/new-ai-classifier-for-indicating-ai-written-text/ https://openai.com/index/new-ai-classifier-for-indicating-ai...

dangdetector 2y ago

Doubtful AI writing is obvious as hell.

efficax 2y ago

of course they are. it’s simple: if they worked they would be incorporated into the loss function of the models and then they would no longer work