3 ms·

I mostly use Gemini 2.5 Pro. I have a “you are my editor” prompt asking it to proofread my texts. Recently it pointed out two typos in two different words that

I mostly use Gemini 2.5 Pro. I have a “you are my editor” prompt asking it to proofread my texts. Recently it pointed out two typos in two different words that just weren’t there. Indeed, the two words each had a typo but not the one pointed out by Gemini. The real typos were random missing letters. But the typos Gemini hallucinated were ones that are very common typos made in those words. The only thing transformer based LLMs can ever do is _faking_ intelligence. Which for many tasks is good enough. Even in my example above, the corrected text was flawless. But for a whole category of tasks, LLMs without oversight will never be good enough because there simply is no real intelligence in them.

shmel 1y ago
do you really think that an architecture that struggles to count r in strawberry is a good choice for proofreading? It perceives words very differently from us.
- maoberlehner 1y ago
  Yes, actually I think it works really well for me considering that I’m not a native speaker and one thing I’m after is correcting technical correct but non-idiomatic wording.
- simonw 1y ago
  Counting letters in words and identifying when words are misspelled are two different tasks - it can be good at one and bad at the other. Interestingly, spell checking is something models have been surprisingly bad at in the past - I remember being shocked at how bad Claude 3 was at spotting typos. This has changed with Claude 4 and o3 from what I've seen - another example of incremental model improvements swinging over a line in terms of things they can now be useful for.
  - slagsnag 1y ago
    Shill harder, Simon! Otherwise they may refuse to ask you back for their next PR chucklefest.
    - simonw 1y ago
      Wasn't expecting a "you're a shill" accusation to show up on a comment where I say that LLMs used to suck at spell check but now they can just about do it.
      - gnowlescentic 1y ago
        So 2 trillion dollars to do what Word could do in 1995... and trying to promote that as an advancement is not propaganda? Sure let's double the amount of resources a couple more times who knows what it will be able to take on after mastering spelling.
butler14 1y ago
I had this too last week. It pointed out two errors that simply weren’t there. Then completely refused to back down and doubled down on its own certainty, until I sent it a screenshot of the original prompt. Kind of funny.
- BartjeD 1y ago
  [flagged]
morganf 1y ago
One of my main uses for LLMs is copy editing and it is incredible to me how terrible all of them are at that.
awestroke 1y ago
I'll show you a few misspelled words and you tell me (without using any tools or thinking it through) which bits in the utf8 encoded bytes are incorrect. If you're wrong, I'll conclude you are not intelligent. LLMs don't see letters, they see tokens. This is a foundational attribute of LLMs. When you point out that the LLM does not know the number of R's in the word "Strawberry", you are not exposing the LLM as some kind of sham, you're just admitting to being a fool.
- hnfong 1y ago
  Being confused as to how LLMs see tokens is just a factual error. I think the more concerning error GP makes is how he makes deductions on fundamental nature of the intelligence of LLMs by looking at "bugs" in current iterations of LLMs. It's like looking at a child struggling to learn how to spell, and making broad claims like "look at the mistakes this child made, humans will never attain any __real__ intelligence!" So yeah at this point I'm often pessimistic whether humans have "real" intelligence or not. Pretty sure LLMs can spot the logical mistakes in his claims easily.
  - maoberlehner 1y ago
    Your explanation perfectly captures another big differences between human / mammal intelligence and LLM intelligence: A child can make mistakes and (few shot) learn. A LLM can’t. And even a child struggling with spelling won’t make a mistake like the one I have described. It will spell things wrong and not even catch the spelling mistake. But it won’t pretend and insist there is a mistake where there isn’t (okay, maybe it will, but only to troll you). Maybe talking about “real” intelligence was not precise enough and it’s better to talk about “mammal like intelligence.” I guess there is a chance LLMs can be trained to a level where all the questions where there is a correct answer for (basically everything that can be benchmarked) will be answered correctly. Would this be incredibly useful and make a lot of jobs obsolete? Yes. Still a very different form of intelligence.
    - LordDragonfang 1y ago
      > A child can make mistakes and (few shot) learn. A LLM can’t. Considering that we literally call the process of giving an llm several attempts at a problem "few-shot reasoning", I do not understand your reasoning here. And LLM absolutely can "gain acquire knowledge of or skill in (something)" of things within its context window (i.e. learning). And then you can bake those understandings in by making a LoRa, or further training. If this is really your distinction that makes intelligence, the only difference between llms and human brains is that human brains have a built-in mechanism to convert short-term memory to long-term, and llms haven't fully evolved that.

shmel 1y ago

do you really think that an architecture that struggles to count r in strawberry is a good choice for proofreading? It perceives words very differently from us.

maoberlehner 1y ago
Yes, actually I think it works really well for me considering that I’m not a native speaker and one thing I’m after is correcting technical correct but non-idiomatic wording.
simonw 1y ago
Counting letters in words and identifying when words are misspelled are two different tasks - it can be good at one and bad at the other. Interestingly, spell checking is something models have been surprisingly bad at in the past - I remember being shocked at how bad Claude 3 was at spotting typos. This has changed with Claude 4 and o3 from what I've seen - another example of incremental model improvements swinging over a line in terms of things they can now be useful for.
- slagsnag 1y ago
  Shill harder, Simon! Otherwise they may refuse to ask you back for their next PR chucklefest.
  - simonw 1y ago
    Wasn't expecting a "you're a shill" accusation to show up on a comment where I say that LLMs used to suck at spell check but now they can just about do it.
    - gnowlescentic 1y ago
      So 2 trillion dollars to do what Word could do in 1995... and trying to promote that as an advancement is not propaganda? Sure let's double the amount of resources a couple more times who knows what it will be able to take on after mastering spelling.

butler14 1y ago

I had this too last week. It pointed out two errors that simply weren’t there. Then completely refused to back down and doubled down on its own certainty, until I sent it a screenshot of the original prompt. Kind of funny.

BartjeD 1y ago
[flagged]

morganf 1y ago

One of my main uses for LLMs is copy editing and it is incredible to me how terrible all of them are at that.

awestroke 1y ago

I'll show you a few misspelled words and you tell me (without using any tools or thinking it through) which bits in the utf8 encoded bytes are incorrect. If you're wrong, I'll conclude you are not intelligent. LLMs don't see letters, they see tokens. This is a foundational attribute of LLMs. When you point out that the LLM does not know the number of R's in the word "Strawberry", you are not exposing the LLM as some kind of sham, you're just admitting to being a fool.

hnfong 1y ago
Being confused as to how LLMs see tokens is just a factual error. I think the more concerning error GP makes is how he makes deductions on fundamental nature of the intelligence of LLMs by looking at "bugs" in current iterations of LLMs. It's like looking at a child struggling to learn how to spell, and making broad claims like "look at the mistakes this child made, humans will never attain any __real__ intelligence!" So yeah at this point I'm often pessimistic whether humans have "real" intelligence or not. Pretty sure LLMs can spot the logical mistakes in his claims easily.
- maoberlehner 1y ago
  Your explanation perfectly captures another big differences between human / mammal intelligence and LLM intelligence: A child can make mistakes and (few shot) learn. A LLM can’t. And even a child struggling with spelling won’t make a mistake like the one I have described. It will spell things wrong and not even catch the spelling mistake. But it won’t pretend and insist there is a mistake where there isn’t (okay, maybe it will, but only to troll you). Maybe talking about “real” intelligence was not precise enough and it’s better to talk about “mammal like intelligence.” I guess there is a chance LLMs can be trained to a level where all the questions where there is a correct answer for (basically everything that can be benchmarked) will be answered correctly. Would this be incredibly useful and make a lot of jobs obsolete? Yes. Still a very different form of intelligence.
  - LordDragonfang 1y ago
    > A child can make mistakes and (few shot) learn. A LLM can’t. Considering that we literally call the process of giving an llm several attempts at a problem "few-shot reasoning", I do not understand your reasoning here. And LLM absolutely can "gain acquire knowledge of or skill in (something)" of things within its context window (i.e. learning). And then you can bake those understandings in by making a LoRa, or further training. If this is really your distinction that makes intelligence, the only difference between llms and human brains is that human brains have a built-in mechanism to convert short-term memory to long-term, and llms haven't fully evolved that.