3 ms·

I'll show you a few misspelled words and you tell me (without using any tools or thinking it through) which bits in the utf8 encoded bytes are incorrect. If you

I'll show you a few misspelled words and you tell me (without using any tools or thinking it through) which bits in the utf8 encoded bytes are incorrect. If you're wrong, I'll conclude you are not intelligent. LLMs don't see letters, they see tokens. This is a foundational attribute of LLMs. When you point out that the LLM does not know the number of R's in the word "Strawberry", you are not exposing the LLM as some kind of sham, you're just admitting to being a fool.

hnfong 1y ago
Being confused as to how LLMs see tokens is just a factual error. I think the more concerning error GP makes is how he makes deductions on fundamental nature of the intelligence of LLMs by looking at "bugs" in current iterations of LLMs. It's like looking at a child struggling to learn how to spell, and making broad claims like "look at the mistakes this child made, humans will never attain any __real__ intelligence!" So yeah at this point I'm often pessimistic whether humans have "real" intelligence or not. Pretty sure LLMs can spot the logical mistakes in his claims easily.
- maoberlehner 1y ago
  Your explanation perfectly captures another big differences between human / mammal intelligence and LLM intelligence: A child can make mistakes and (few shot) learn. A LLM can’t. And even a child struggling with spelling won’t make a mistake like the one I have described. It will spell things wrong and not even catch the spelling mistake. But it won’t pretend and insist there is a mistake where there isn’t (okay, maybe it will, but only to troll you). Maybe talking about “real” intelligence was not precise enough and it’s better to talk about “mammal like intelligence.” I guess there is a chance LLMs can be trained to a level where all the questions where there is a correct answer for (basically everything that can be benchmarked) will be answered correctly. Would this be incredibly useful and make a lot of jobs obsolete? Yes. Still a very different form of intelligence.
  - LordDragonfang 1y ago
    > A child can make mistakes and (few shot) learn. A LLM can’t. Considering that we literally call the process of giving an llm several attempts at a problem "few-shot reasoning", I do not understand your reasoning here. And LLM absolutely can "gain acquire knowledge of or skill in (something)" of things within its context window (i.e. learning). And then you can bake those understandings in by making a LoRa, or further training. If this is really your distinction that makes intelligence, the only difference between llms and human brains is that human brains have a built-in mechanism to convert short-term memory to long-term, and llms haven't fully evolved that.
LetsGetTechnicl 1y ago
Damn, if only something called a "language model" could model language accurately, let alone live up to its creators' claims that it possesses near-human intelligence. But yeah we can call getting some basic facts a "feature not a bug" if you want
- awestroke 1y ago
  So people that can't read or write have no language? If you don't know an alphabet and its rules, you won't know how many letters are in words. Does that make you unable to model language accurately?
  - maoberlehner 1y ago
    So LLMs don’t know the alphabet and its rules?
    - awestroke 1y ago
      You know about ultraviolet but that doesn't help you see ultraviolet light
      - maoberlehner 1y ago
        Yes. And I know I can’t see it and don’t pretend I can, and that it in fact is green.
        LordDragonfang 1y ago
        Not green, no. But actually, you can see an intense enough source of (monochromatic) near-UV light, our lenses only filter out the majority of it. And if you did, your brain would hallucinate it as purplish-blueish white. Because that's the closest color to those inputs based on your what your neural network (brain) was trained on. It's encountering something uncommon, so it guesses and present it as fact. From this, we can determine either that you (and indeed all humans) are not actually intelligent, or alternatively, intelligence and cognition are complicated and you can't conclude its absence from the first time someone behaves in a way you're not trained to expect from your experience of intelligence.
ncallaway 1y ago
> When you point out that the LLM does not know the number of R's in the word "Strawberry", you are not exposing the LLM as some kind of sham, you're just admitting to being a fool. I'm sorry but that's not reasonable. Yes, I understand what you mean on an architectural level, but if a product is being deployed to the masses you are the fool if you expect every user to have a deep architectural understanding of it. If it's being sold as "this model is a PhD-level expert on every topic in your pocket", then the underlying technical architecture and its specific foibles are irrelevant. What matters is the claims about what it's capable of doing and its actual performance. Would it matter if GPT-5 couldn't count the number of r's in a specific word if the marketing claims being made around it were more grounded? Probably not. But that's not what's happening.
- withinboredom 1y ago
  > If it's being sold as "this model is a PhD-level expert on every topic in your pocket", The thing that pissed me off about them using this line is that they prevented the people who actually pull that off one day from using it.
maoberlehner 1y ago
I think we’re saying the same thing using different words. What LLMs do and what human brains do are very different things. Therefore human / biological intelligence is a different thing than LLM intelligence. Is this phrasing something you can agree with?
tripzilch 1y ago
If I had learned to read utf8 bytes instead of Latin alphabet, this would be trivial. In fact give me a (paid) week to study utf8 for reading and I am sure I could do it. (yes I already know how utf8 works) And the token/strawberry thing is a non-excuse. They just can't count. I can count the number of syllables in a word, regardless of how it's spelled, that's also not based on letters. Or if you want a sub-letter equivalent, I could also count the number of serifs, dots or curves in a word. It's really not so much that the strawberry thing is a "gotcha", or easily explained by "they see tokens instead", because the same reasoning errors happen all the time in LLMs also in places where "it's because of tokens" can't possibly be the explanation. It's just that the strawberry thing is one of the easiest ways to show it just can't reason reliably.

hnfong 1y ago

Being confused as to how LLMs see tokens is just a factual error. I think the more concerning error GP makes is how he makes deductions on fundamental nature of the intelligence of LLMs by looking at "bugs" in current iterations of LLMs. It's like looking at a child struggling to learn how to spell, and making broad claims like "look at the mistakes this child made, humans will never attain any __real__ intelligence!" So yeah at this point I'm often pessimistic whether humans have "real" intelligence or not. Pretty sure LLMs can spot the logical mistakes in his claims easily.

maoberlehner 1y ago
Your explanation perfectly captures another big differences between human / mammal intelligence and LLM intelligence: A child can make mistakes and (few shot) learn. A LLM can’t. And even a child struggling with spelling won’t make a mistake like the one I have described. It will spell things wrong and not even catch the spelling mistake. But it won’t pretend and insist there is a mistake where there isn’t (okay, maybe it will, but only to troll you). Maybe talking about “real” intelligence was not precise enough and it’s better to talk about “mammal like intelligence.” I guess there is a chance LLMs can be trained to a level where all the questions where there is a correct answer for (basically everything that can be benchmarked) will be answered correctly. Would this be incredibly useful and make a lot of jobs obsolete? Yes. Still a very different form of intelligence.
- LordDragonfang 1y ago
  > A child can make mistakes and (few shot) learn. A LLM can’t. Considering that we literally call the process of giving an llm several attempts at a problem "few-shot reasoning", I do not understand your reasoning here. And LLM absolutely can "gain acquire knowledge of or skill in (something)" of things within its context window (i.e. learning). And then you can bake those understandings in by making a LoRa, or further training. If this is really your distinction that makes intelligence, the only difference between llms and human brains is that human brains have a built-in mechanism to convert short-term memory to long-term, and llms haven't fully evolved that.

LetsGetTechnicl 1y ago

Damn, if only something called a "language model" could model language accurately, let alone live up to its creators' claims that it possesses near-human intelligence. But yeah we can call getting some basic facts a "feature not a bug" if you want

awestroke 1y ago
So people that can't read or write have no language? If you don't know an alphabet and its rules, you won't know how many letters are in words. Does that make you unable to model language accurately?
- maoberlehner 1y ago
  So LLMs don’t know the alphabet and its rules?
  - awestroke 1y ago
    You know about ultraviolet but that doesn't help you see ultraviolet light
    - maoberlehner 1y ago
      Yes. And I know I can’t see it and don’t pretend I can, and that it in fact is green.
      - LordDragonfang 1y ago
        Not green, no. But actually, you can see an intense enough source of (monochromatic) near-UV light, our lenses only filter out the majority of it. And if you did, your brain would hallucinate it as purplish-blueish white. Because that's the closest color to those inputs based on your what your neural network (brain) was trained on. It's encountering something uncommon, so it guesses and present it as fact. From this, we can determine either that you (and indeed all humans) are not actually intelligent, or alternatively, intelligence and cognition are complicated and you can't conclude its absence from the first time someone behaves in a way you're not trained to expect from your experience of intelligence.

ncallaway 1y ago

> When you point out that the LLM does not know the number of R's in the word "Strawberry", you are not exposing the LLM as some kind of sham, you're just admitting to being a fool. I'm sorry but that's not reasonable. Yes, I understand what you mean on an architectural level, but if a product is being deployed to the masses you are the fool if you expect every user to have a deep architectural understanding of it. If it's being sold as "this model is a PhD-level expert on every topic in your pocket", then the underlying technical architecture and its specific foibles are irrelevant. What matters is the claims about what it's capable of doing and its actual performance. Would it matter if GPT-5 couldn't count the number of r's in a specific word if the marketing claims being made around it were more grounded? Probably not. But that's not what's happening.

withinboredom 1y ago
> If it's being sold as "this model is a PhD-level expert on every topic in your pocket", The thing that pissed me off about them using this line is that they prevented the people who actually pull that off one day from using it.

maoberlehner 1y ago

I think we’re saying the same thing using different words. What LLMs do and what human brains do are very different things. Therefore human / biological intelligence is a different thing than LLM intelligence. Is this phrasing something you can agree with?

tripzilch 1y ago

If I had learned to read utf8 bytes instead of Latin alphabet, this would be trivial. In fact give me a (paid) week to study utf8 for reading and I am sure I could do it. (yes I already know how utf8 works) And the token/strawberry thing is a non-excuse. They just can't count. I can count the number of syllables in a word, regardless of how it's spelled, that's also not based on letters. Or if you want a sub-letter equivalent, I could also count the number of serifs, dots or curves in a word. It's really not so much that the strawberry thing is a "gotcha", or easily explained by "they see tokens instead", because the same reasoning errors happen all the time in LLMs also in places where "it's because of tokens" can't possibly be the explanation. It's just that the strawberry thing is one of the easiest ways to show it just can't reason reliably.