4 ms·

> Your right to push back on that... Human test passed. AIs make giant flashy mistakes. They don't fuck up basic grammar.

by lee_ars 1mo ago

dofm 1mo ago
I have found it intriguingly difficult in my own tests to make LLMs deliberately fuck up grammar in any way similar to how humans do. I suppose this is actually an architectural limitation.
- ben_w 1mo ago
  Out of curiosity, I just tried this: write a comment about how to use ChatGPT to write a game like you're a barely literate reddit user, short answer, maximum incoherency and the result was a pretty good simulation. I tried less insulting prompts first, this did not produce a good simulation. Not perfect, gptzero.me still knew it was AI generated (tayo42 is human by the same measure), but if for some reason someone was using an LLM prompted with that pattern, I suspect it would fool me in a Turing test unless I found the keyword to force the agent to change the role it was playing.
  - dofm 1mo ago
    Redditor is too easy. Conveying the errors typical in a Raise The Colours group on Facebook, that's the hard stuff.
    - ben_w 1mo ago
      I'll take your word for it, I think I'm in zero groups on Facebook, I only even look at the feed when I see a notification in Messenger; and I barely use Messenger, too.
jlokier 1mo ago
> They don't fuck up basic grammar. Sometimes they do. From Gemini 3, some AppleScript: if something then if something then if something then stuff end if end if </if> Each time I asked Gemini 3 to fix the syntax error, it acknowledged it had screwed up the basic syntax, and then emitted a new version of the code. With another XML closing tag somewhere. Then it was embarrassed, apologetic, etc. to see it had made the same mistake repeatedly. Eventually it output several attempts per turn as it kept checking its own work in-turn, seeing that it had made the same kind of mistake and tried again, this time definitely fixing it, proudly and confidently presenting me with the correct, final, definitive version. With another 'end if' or 'end tell' replaced by an XML closing tag that it couldn't see until the next turn. After about 10 turns the problem went away.

> Your right to push back on that... Human test passed. AIs make giant flashy mistakes. They don't fuck up basic grammar.

by lee_ars 1mo ago

> Your right to push back on that... Human test passed. AIs make giant flashy mistakes. They don't fuck up basic grammar.

dofm 1mo ago

I have found it intriguingly difficult in my own tests to make LLMs deliberately fuck up grammar in any way similar to how humans do. I suppose this is actually an architectural limitation.

ben_w 1mo ago
Out of curiosity, I just tried this: write a comment about how to use ChatGPT to write a game like you're a barely literate reddit user, short answer, maximum incoherency and the result was a pretty good simulation. I tried less insulting prompts first, this did not produce a good simulation. Not perfect, gptzero.me still knew it was AI generated (tayo42 is human by the same measure), but if for some reason someone was using an LLM prompted with that pattern, I suspect it would fool me in a Turing test unless I found the keyword to force the agent to change the role it was playing.
- dofm 1mo ago
  Redditor is too easy. Conveying the errors typical in a Raise The Colours group on Facebook, that's the hard stuff.
  - ben_w 1mo ago
    I'll take your word for it, I think I'm in zero groups on Facebook, I only even look at the feed when I see a notification in Messenger; and I barely use Messenger, too.

jlokier 1mo ago

> They don't fuck up basic grammar. Sometimes they do. From Gemini 3, some AppleScript: if something then if something then if something then stuff end if end if </if> Each time I asked Gemini 3 to fix the syntax error, it acknowledged it had screwed up the basic syntax, and then emitted a new version of the code. With another XML closing tag somewhere. Then it was embarrassed, apologetic, etc. to see it had made the same mistake repeatedly. Eventually it output several attempts per turn as it kept checking its own work in-turn, seeing that it had made the same kind of mistake and tried again, this time definitely fixing it, proudly and confidently presenting me with the correct, final, definitive version. With another 'end if' or 'end tell' replaced by an XML closing tag that it couldn't see until the next turn. After about 10 turns the problem went away.