3 ms·

We're on Hacker News. Do I really have to point out the existence of social engineering to you? Or that scamming old people out of their life savings is a profi

We're on Hacker News. Do I really have to point out the existence of social engineering to you? Or that scamming old people out of their life savings is a profitable enough activity that there are entire call centers dedicated to the task? Humans keep overestimating just how high the bar of "human-like intelligence" is.

jacobgold 3mo ago
Drawing the conclusion that "humans fail" and "models fail", so they must be similar, is very wrong. You could have humans calculate 2+2 all day and get a surprisingly high error rate. That reveals a flaw in how humans operate. LLMs fail for entirely different reasons. Their mistakes don't imply they're human-like at all. It's not about the error rate.
- ACCount37 3mo ago
  You're saying that a class of mistakes points out a "major (possibly fundamental) flaw". I'm pointing out some very similar classes of mistakes in humans - well known, well documented and widely exploited. They just keep paying the "IRS" in gift cards, buying lottery tickets and getting the captain's age wrong. If you're using the existence of flaws in LLMs to deny the claim of intelligence to them, then why do "generally intelligent" humans exhibit some impressively similar-looking flaws? And, if we're talking about that conspicuous similarity - do they actually fail "for entirely different reasons"? Or do you just want the reasons to be "entirely different" - and not the same reasons viewed at a different angle? Because the similarities between humans falling for trick questions or scams, and LLMs falling for adversarial questions or prompt injections don't look coincidental to me at all. One of the oldest patterns in scamming is overwhelming and confusing the victim. Numerous prompt injection methods seek to overwhelm and confuse an LLM - if an LLM can't keep track of things, can't grasp what's going on, it's far more likely to lose track of what's a prompt and what's data, overlook past instructions or go past its behavioral guardrails. And humans who fall for trick questions like "1kg of feathers" or "captain's age" due to shallow attention and naive pattern matching? They fail in surprisingly similar ways to how LLMs fail on SimpleBench tasks that are filled with overwhelming adversarial distractors. Many "trick questions" are tricky to humans and LLMs alike - to the point that it's unlikely to be coincidental.
  - jacobgold 3mo ago
    > If you're using the existence of flaws in LLMs to deny the claim of intelligence to them... That's not the point at all. It's the fact that they fail in ways completely unlike humans. You also have the burden of proof reversed. Its on you to prove these LLM agents are human-like intelligences if that's your claim. No one can prove this because it's false.
    - ACCount37 3mo ago
      You are the one claiming that "they fail in ways completely unlike humans" insistently. Now go cough up some proof. I'll wait.
      - jacobgold 3mo ago
        This isn't even controversial. The proof is available to anyone who uses these systems: They hallucinate tool state, drift from the objective while seeming to comply, switch languages randomly (Cyrillic or Japanese characters in output), confuse tasks they've planned for completed ones, and of course follow prompt injections embedded in files or web pages.
        TeMPOraL 3mo ago
        So just like me, including the prompt injections if you count "nerd sniping" as such? (And in particular, switching languages on the fly is normal for people who speak more than one well, it's something you learn not to do for the sake of people less comfortable with the languages involved.)
        jacobgold 3mo ago
        > So just like me, including the prompt injections if you count "nerd sniping" as such? Who would count that as prompt injection? It's a superficial analogy. If you were vulnerable to prompt injection, I could order you to do absolutely anything you're capable of doing and you would be helpless to do otherwise.

jacobgold 3mo ago

Drawing the conclusion that "humans fail" and "models fail", so they must be similar, is very wrong. You could have humans calculate 2+2 all day and get a surprisingly high error rate. That reveals a flaw in how humans operate. LLMs fail for entirely different reasons. Their mistakes don't imply they're human-like at all. It's not about the error rate.

ACCount37 3mo ago
You're saying that a class of mistakes points out a "major (possibly fundamental) flaw". I'm pointing out some very similar classes of mistakes in humans - well known, well documented and widely exploited. They just keep paying the "IRS" in gift cards, buying lottery tickets and getting the captain's age wrong. If you're using the existence of flaws in LLMs to deny the claim of intelligence to them, then why do "generally intelligent" humans exhibit some impressively similar-looking flaws? And, if we're talking about that conspicuous similarity - do they actually fail "for entirely different reasons"? Or do you just want the reasons to be "entirely different" - and not the same reasons viewed at a different angle? Because the similarities between humans falling for trick questions or scams, and LLMs falling for adversarial questions or prompt injections don't look coincidental to me at all. One of the oldest patterns in scamming is overwhelming and confusing the victim. Numerous prompt injection methods seek to overwhelm and confuse an LLM - if an LLM can't keep track of things, can't grasp what's going on, it's far more likely to lose track of what's a prompt and what's data, overlook past instructions or go past its behavioral guardrails. And humans who fall for trick questions like "1kg of feathers" or "captain's age" due to shallow attention and naive pattern matching? They fail in surprisingly similar ways to how LLMs fail on SimpleBench tasks that are filled with overwhelming adversarial distractors. Many "trick questions" are tricky to humans and LLMs alike - to the point that it's unlikely to be coincidental.
- jacobgold 3mo ago
  > If you're using the existence of flaws in LLMs to deny the claim of intelligence to them... That's not the point at all. It's the fact that they fail in ways completely unlike humans. You also have the burden of proof reversed. Its on you to prove these LLM agents are human-like intelligences if that's your claim. No one can prove this because it's false.
  - ACCount37 3mo ago
    You are the one claiming that "they fail in ways completely unlike humans" insistently. Now go cough up some proof. I'll wait.
    - jacobgold 3mo ago
      This isn't even controversial. The proof is available to anyone who uses these systems: They hallucinate tool state, drift from the objective while seeming to comply, switch languages randomly (Cyrillic or Japanese characters in output), confuse tasks they've planned for completed ones, and of course follow prompt injections embedded in files or web pages.
      - TeMPOraL 3mo ago
        So just like me, including the prompt injections if you count "nerd sniping" as such? (And in particular, switching languages on the fly is normal for people who speak more than one well, it's something you learn not to do for the sake of people less comfortable with the languages involved.)
        jacobgold 3mo ago
        > So just like me, including the prompt injections if you count "nerd sniping" as such? Who would count that as prompt injection? It's a superficial analogy. If you were vulnerable to prompt injection, I could order you to do absolutely anything you're capable of doing and you would be helpless to do otherwise.