4 ms·

People who frequently use ChatGPT for writing tasks can detect AI-generated text

12 points by gaws 1y ago

magicalhippo 1y ago
This looks like AI slop, I can tell from the em dash and from seeing quite a lot of AI slop in my time I mean, not a very surprising result? At work I can almost always immediately see who wrote which lines of code, since we don't enforce code formatting. From what I can gather most authors do have a distinctive writing style as well, which can be detected. Why would LLMs be different?
alganet 1y ago
Why would I investigate the labyrinth floor if the walls are shorter than my eyesight? I feel like this is a fools errand. Let's use the marketing terms to frame this problem. Companies are saying LLMs are like "the new printing press". Should I worry about detecting if something was printed or written by hand? No, I should worry about the volume and contents. That's what matters.
deleted 1y ago
[deleted]
jasonthorsness 1y ago
LLMs by default use the same style every time and don't know have any drive to differentiate. I just wrote an article about this in the "web site design" space where the designed sites ended up looking all alike (https://www.jasonthorsness.com/29 https://www.jasonthorsness.com/29). It makes complete sense that people will start to recognize the "default style" of each LLM. I wonder whether this style is specific to the LLM (Grok vs. ChatGPT) or if it will somehow arise from the raining data itself that they all share and be sort of a permanent "accent" the LLMs have.
- twilight-code 1y ago
  I think you don't even need to use chatGPT, who was using make-me-sound smart verbs like 'delve' frequently before? Now, in some publications; I see it regularly.
  - jasonthorsness 1y ago
    You're absolutely right!
  - ahazred8ta 1y ago
    It's like the magic sunglasses in They Live. You're just seeing the intricate tapestry of nuances.
- yencabulator 1y ago
  > I wonder whether this style is specific to the LLM (Grok vs. ChatGPT) or if it will somehow arise from the raining data itself that they all share and be sort of a permanent "accent" the LLMs have. It's very different. I used LLMs to brainstorm a business plan write-up hypothetical startup and Claude/Gemini/qwen3:30b-a3b, one-shot with a long background text. They all generated similar-but-slightly-non-overlapping ideas in very different language. If I remember right it went something like this: Gemini used stilted business-speak and bold everywhere, but had the best structure for the document. Claude gave surprisingly good one-paragraph intros to every section. Claude and Gwen3 were roughly tied to how nicely written their bullet point content was. I made a new document with Gemini's base structure, Claude's intros, and bullet points mashed from Claude and Gwen, making sure I covered all the good ideas from Gemini that weren't mentioned. Then I edited everything for homogeneity and style. I recommend experimenting with combining their work. (Also, qwen3:30b-a3b is amazing for local LLM work, the MoE architecture makes it have the speed of a 3B parameter model!)

12 points by gaws 1y ago

This looks like AI slop, I can tell from the em dash and from seeing quite a lot of AI slop in my time I mean, not a very surprising result? At work I can almost always immediately see who wrote which lines of code, since we don't enforce code formatting. From what I can gather most authors do have a distinctive writing style as well, which can be detected. Why would LLMs be different?

alganet 1y ago

Why would I investigate the labyrinth floor if the walls are shorter than my eyesight? I feel like this is a fools errand. Let's use the marketing terms to frame this problem. Companies are saying LLMs are like "the new printing press". Should I worry about detecting if something was printed or written by hand? No, I should worry about the volume and contents. That's what matters.

deleted 1y ago

[deleted]

jasonthorsness 1y ago

LLMs by default use the same style every time and don't know have any drive to differentiate. I just wrote an article about this in the "web site design" space where the designed sites ended up looking all alike (https://www.jasonthorsness.com/29 https://www.jasonthorsness.com/29). It makes complete sense that people will start to recognize the "default style" of each LLM. I wonder whether this style is specific to the LLM (Grok vs. ChatGPT) or if it will somehow arise from the raining data itself that they all share and be sort of a permanent "accent" the LLMs have.

twilight-code 1y ago
I think you don't even need to use chatGPT, who was using make-me-sound smart verbs like 'delve' frequently before? Now, in some publications; I see it regularly.
- jasonthorsness 1y ago
  You're absolutely right!
- ahazred8ta 1y ago
  It's like the magic sunglasses in They Live. You're just seeing the intricate tapestry of nuances.
yencabulator 1y ago
> I wonder whether this style is specific to the LLM (Grok vs. ChatGPT) or if it will somehow arise from the raining data itself that they all share and be sort of a permanent "accent" the LLMs have. It's very different. I used LLMs to brainstorm a business plan write-up hypothetical startup and Claude/Gemini/qwen3:30b-a3b, one-shot with a long background text. They all generated similar-but-slightly-non-overlapping ideas in very different language. If I remember right it went something like this: Gemini used stilted business-speak and bold everywhere, but had the best structure for the document. Claude gave surprisingly good one-paragraph intros to every section. Claude and Gwen3 were roughly tied to how nicely written their bullet point content was. I made a new document with Gemini's base structure, Claude's intros, and bullet points mashed from Claude and Gwen, making sure I covered all the good ideas from Gemini that weren't mentioned. Then I edited everything for homogeneity and style. I recommend experimenting with combining their work. (Also, qwen3:30b-a3b is amazing for local LLM work, the MoE architecture makes it have the speed of a 3B parameter model!)