6 ms·

This is one of the least interesting questions to ask LLMs. I wish it wasn't so popular.

by alach11 2y ago

moralestapia 2y ago
And yet ... all they have to do is answer it properly.
- sfink 2y ago
  It did! How many R's do you think are in "strawberry"? I get 3, and it got 3. I'm with the LLM. ...oh, you disagree with its intermediate reasoning? You think it should follow a series of logical steps that are each individually correct, rather than pulling wildly incorrect intermediate steps out of its butt and synthesizing them together in the end into a coherent answer that happens to be correct? But that's what an LLM is! Why complain about an LLM being an (unusually transparent) LLM? I think this example serves as a beautiful illustration of how LLMs work, and are supposed to work—even if the correspondence is inexact between (1) incorrect and irrelevant-to-us intermediate steps and (2) internal matrix multiplications that represent incorrect or invalid logic. The correspondence is inexact and probably mostly figurative, but it's still a great example of how internal nonsense can still lead to externally "correct" answers. ("Nonsense" is underselling the complex and highly sophisticated internal state that leads to "correct" answers a stunningly high percentage of the time; I'm just trying to distinguish it from formal logic or the type of reasoning that we think we do and sometimes actually do. And would do, in this example problem.)
  - moralestapia 2y ago
    That's why I used the word properly, to imply that the whole answer should make sense.
- qeternity 2y ago
  Training on the test set is all you need.
NoCoooode 2y ago
That they can't do this sort of simple question speaks volumes to the entire approach. I don't think generative AI will ever be able to reach AGI, and most people selling LLM today pretend it is AGI
- GaggiX 2y ago
  The fact that LLMs are usually trained on tokens and not on characters, doesn't really speak about what generative AI is going to reach or not. >most people selling LLM today pretend it is AGI Who are these "most people"?
  - smokedetector1 2y ago
    ELI5 why are tokens not a single letter?
    - WhitneyLand 2y ago
      Such an architecture could be implemented, it could use one token per letter, or one token per word, instead of the typical 0.75 per word we see. The choice just comes with trade-offs in memory usage, compute, and effectiveness of the model in various scenarios. So what we ended up with was a pragmatic/engineering decision rather than a theoretical or fundamental constraint.
- alach11 2y ago
  All it speaks to is that tokenization is weird and introduces artifacts to LLM performance. Counting letters is a trivial task when you're staring at words on a screen. It's much harder when you're perceiving vectors based on parts of words. The fact that LLMs find certain things easier/harder than humans is completely unsurprising, and there are much more interesting benchmarks to use to compare one LLM to another.
  - jrflowers 2y ago
    This is a good point. While LLMs being incapable of reliably doing a simple task that’s been doable by computers since the punch card days is an important consideration for anyone that might be thinking about using them for anything other than as a toy, this fact is uninteresting because of Reasons
xanderlewis 2y ago
The point is that it’s trivial (or should be). Not that it’s interesting. It’s certainly interesting that they can’t answer it properly, though.
WhitneyLand 2y ago
I find it fascinating. To me it grounds expectations and motivates discussions about the jagged boundaries of LLM capabilities.
- hn_throwaway_99 2y ago
  Why do you find it fascinating? I have the most rudimentary understanding of LLMs and it seems to me the least fascinating thing about LLM limitations. That is, all LLMs look at language as a series of opaque, independent tokens, e.g. strawberry might be a single token (say <5678>), or probably two (e.g. straw and berry, say <123><789>). But in no way will it represent it like we will, with letters. So if you ask it "how many R's are in strawberry", it fundamentally can't do any "counting", it just basically has to rely on whether that question (or similar questions about related words) has been asked before so it can predict the next token in its output correctly. I suppose with enough data and LLM could learn to associate character counts with tokens (e.g. with the right training set it could learn metadata about the tokens). My point being that with only the most basic understanding of how LLMs function, this "character counting" limitation seems blatantly obvious.
  - WhitneyLand 2y ago
    I don’t think we’ve yet come to the point where, how an LLM end to end goes from prompt to output is blatantly obvious. LLMs operate with language at multiple levels of abstraction and tokens are not the only way to have character level knowledge. For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge. And of course LLMs famously have emergent properties, for which there’s not yet a precise mechanism to illuminate the results. We’re dealing with very complex systems that are still relatively poorly understood, and I believe the pool of concepts understood to the point of being blatantly obvious is still small.
    - hn_throwaway_99 2y ago
      The fact that large parts of how LLMs work is poorly understood doesn't mean that that we should brush the whole thing with "it's a complex mystery!" We know exactly how LLMs tokenize their input - OpenAI even has a public tool you can test out, https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer . When you say "For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge." - yes, that's why I explicitly said "e.g. with the right training set it could learn metadata about the tokens". But the point was still put perfectly by another commenter: "How many 1 bits are there in strawberry?" When nearly all humans can't answer that, we're very unsurprised; we see it immediately as a simple difference in how humans encode that string vs. how computers do it. We don't say "Well, the human mind is so complex and filled with emergent properties that the reason for this human limitation is a big mystery". And we also know that if we teach a human the encoding rules for characters (i.e. the bit pattern of each letter), they could answer this question, and similarly that's the analogous thing to training an LLM to learn metadata about the tokens. Now, what I think is very interesting is why it's so hard to teach an LLM to say "I don't know" when asked to count characters. In my opinion, that's a much more interesting limitation that gets at some of the root, fundamental differences in how LLMs function.
anonzzzies 2y ago
Yep, people find this interesting, but many (many) people get this wrong as well; it has something to do with how many letters it is and how many you can keep in your head I think. We are not very good counters or calculators or computers and, even though I am not saying llms are the same or close to humans, we tried to model their substrates after biology and are surprised we get something that cannot count or calculate very well. I find it a dumb test that proves nothing imho. However, not very impressed with this model.
- nateglims 2y ago
  > many (many) people get this wrong as well Many people can't count letters in a word?
  - roywiggins 2y ago
    I messed up spelling "juggler" in a spelling bee once (I was 10). The confusing thing about LLMs is that they speak tokens, not written text, so it's a lot more like interrogating someone who is functionally illiterate- of course they're going to be a terrible speller. You can find much weirder things that LLMs are absurdly bad at, like "A farmer needs to get himself and a chicken across a river. His boat can hold the farmer and one piece of cargo. How does he do this?" 9 times out of 10 LLMs will pattern match this to the classic puzzle (there's usually also a sack of grain and a wolf) and start insisting on extra trips and inventing wolves. Even if a human makes the same mistake, they almost certainly would realize it after being reminded that there is no wolf, but LLMs often insist there is. o1-preview (but not -mini) seems to have cracked it, though.
  - anonzzzies 2y ago
    Yep. Try it with your neighbours. It's not bad or offensive; it is just what it is.
- marcosdumay 2y ago
  It's a magician-like trick. The double-r gets all the attention because one needs to consciously remember it, so nobody remembers to look at the other single-r. The LLMs probably get it wrong because people get it wrong.
cynicalpeace 2y ago
Something I like to tell it to do is actually to respond using only a certain number of words. Sorta like min token length rather than max token length. Haven't found a model that does it reliably yet.
roywiggins 2y ago
I also think it's kind of a silly example- smart people can be functionally illiterate, after all. It doesn't tell you that much. My favorite LLM stumper is asking them to solve the farmer/wolf/chicken/grain river-crossing puzzle but with only a chicken. They usually either insist on pointless extra trips or hallucinate a wolf or grain. Liquid somehow does both and also loses track of what's where. > The farmer can take the chicken across the river first. Then, he can go back to the original side and bring the boat back. He can then take the chicken back to the original side and leave it there. Next, he can take a bag of grain across the river. He can then go back to the original side and bring the chicken across the river. Finally, he can go back to the original side one last time and bring the chicken across the river. One generation ended like this: > the farmer and the chicken are both on the other side of the river, and the chicken is safe from the grain.
nateglims 2y ago
Being openly annoyed by it will just lead to it being used more. That's sort of a fundamental behavior of the internet.
viccis 2y ago
It's also the case that the more people talk about it, the more future models will "solve" it just by having learned from discussion about it.

This is one of the least interesting questions to ask LLMs. I wish it wasn't so popular.

moralestapia 2y ago

And yet ... all they have to do is answer it properly.

sfink 2y ago
It did! How many R's do you think are in "strawberry"? I get 3, and it got 3. I'm with the LLM. ...oh, you disagree with its intermediate reasoning? You think it should follow a series of logical steps that are each individually correct, rather than pulling wildly incorrect intermediate steps out of its butt and synthesizing them together in the end into a coherent answer that happens to be correct? But that's what an LLM is! Why complain about an LLM being an (unusually transparent) LLM? I think this example serves as a beautiful illustration of how LLMs work, and are supposed to work—even if the correspondence is inexact between (1) incorrect and irrelevant-to-us intermediate steps and (2) internal matrix multiplications that represent incorrect or invalid logic. The correspondence is inexact and probably mostly figurative, but it's still a great example of how internal nonsense can still lead to externally "correct" answers. ("Nonsense" is underselling the complex and highly sophisticated internal state that leads to "correct" answers a stunningly high percentage of the time; I'm just trying to distinguish it from formal logic or the type of reasoning that we think we do and sometimes actually do. And would do, in this example problem.)
- moralestapia 2y ago
  That's why I used the word properly, to imply that the whole answer should make sense.
qeternity 2y ago
Training on the test set is all you need.

NoCoooode 2y ago

That they can't do this sort of simple question speaks volumes to the entire approach. I don't think generative AI will ever be able to reach AGI, and most people selling LLM today pretend it is AGI

GaggiX 2y ago
The fact that LLMs are usually trained on tokens and not on characters, doesn't really speak about what generative AI is going to reach or not. >most people selling LLM today pretend it is AGI Who are these "most people"?
- smokedetector1 2y ago
  ELI5 why are tokens not a single letter?
  - WhitneyLand 2y ago
    Such an architecture could be implemented, it could use one token per letter, or one token per word, instead of the typical 0.75 per word we see. The choice just comes with trade-offs in memory usage, compute, and effectiveness of the model in various scenarios. So what we ended up with was a pragmatic/engineering decision rather than a theoretical or fundamental constraint.
alach11 2y ago
All it speaks to is that tokenization is weird and introduces artifacts to LLM performance. Counting letters is a trivial task when you're staring at words on a screen. It's much harder when you're perceiving vectors based on parts of words. The fact that LLMs find certain things easier/harder than humans is completely unsurprising, and there are much more interesting benchmarks to use to compare one LLM to another.
- jrflowers 2y ago
  This is a good point. While LLMs being incapable of reliably doing a simple task that’s been doable by computers since the punch card days is an important consideration for anyone that might be thinking about using them for anything other than as a toy, this fact is uninteresting because of Reasons

xanderlewis 2y ago

The point is that it’s trivial (or should be). Not that it’s interesting. It’s certainly interesting that they can’t answer it properly, though.

WhitneyLand 2y ago

I find it fascinating. To me it grounds expectations and motivates discussions about the jagged boundaries of LLM capabilities.

hn_throwaway_99 2y ago
Why do you find it fascinating? I have the most rudimentary understanding of LLMs and it seems to me the least fascinating thing about LLM limitations. That is, all LLMs look at language as a series of opaque, independent tokens, e.g. strawberry might be a single token (say <5678>), or probably two (e.g. straw and berry, say <123><789>). But in no way will it represent it like we will, with letters. So if you ask it "how many R's are in strawberry", it fundamentally can't do any "counting", it just basically has to rely on whether that question (or similar questions about related words) has been asked before so it can predict the next token in its output correctly. I suppose with enough data and LLM could learn to associate character counts with tokens (e.g. with the right training set it could learn metadata about the tokens). My point being that with only the most basic understanding of how LLMs function, this "character counting" limitation seems blatantly obvious.
- WhitneyLand 2y ago
  I don’t think we’ve yet come to the point where, how an LLM end to end goes from prompt to output is blatantly obvious. LLMs operate with language at multiple levels of abstraction and tokens are not the only way to have character level knowledge. For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge. And of course LLMs famously have emergent properties, for which there’s not yet a precise mechanism to illuminate the results. We’re dealing with very complex systems that are still relatively poorly understood, and I believe the pool of concepts understood to the point of being blatantly obvious is still small.
  - hn_throwaway_99 2y ago
    The fact that large parts of how LLMs work is poorly understood doesn't mean that that we should brush the whole thing with "it's a complex mystery!" We know exactly how LLMs tokenize their input - OpenAI even has a public tool you can test out, https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer . When you say "For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge." - yes, that's why I explicitly said "e.g. with the right training set it could learn metadata about the tokens". But the point was still put perfectly by another commenter: "How many 1 bits are there in strawberry?" When nearly all humans can't answer that, we're very unsurprised; we see it immediately as a simple difference in how humans encode that string vs. how computers do it. We don't say "Well, the human mind is so complex and filled with emergent properties that the reason for this human limitation is a big mystery". And we also know that if we teach a human the encoding rules for characters (i.e. the bit pattern of each letter), they could answer this question, and similarly that's the analogous thing to training an LLM to learn metadata about the tokens. Now, what I think is very interesting is why it's so hard to teach an LLM to say "I don't know" when asked to count characters. In my opinion, that's a much more interesting limitation that gets at some of the root, fundamental differences in how LLMs function.

anonzzzies 2y ago

Yep, people find this interesting, but many (many) people get this wrong as well; it has something to do with how many letters it is and how many you can keep in your head I think. We are not very good counters or calculators or computers and, even though I am not saying llms are the same or close to humans, we tried to model their substrates after biology and are surprised we get something that cannot count or calculate very well. I find it a dumb test that proves nothing imho. However, not very impressed with this model.

nateglims 2y ago
> many (many) people get this wrong as well Many people can't count letters in a word?
- roywiggins 2y ago
  I messed up spelling "juggler" in a spelling bee once (I was 10). The confusing thing about LLMs is that they speak tokens, not written text, so it's a lot more like interrogating someone who is functionally illiterate- of course they're going to be a terrible speller. You can find much weirder things that LLMs are absurdly bad at, like "A farmer needs to get himself and a chicken across a river. His boat can hold the farmer and one piece of cargo. How does he do this?" 9 times out of 10 LLMs will pattern match this to the classic puzzle (there's usually also a sack of grain and a wolf) and start insisting on extra trips and inventing wolves. Even if a human makes the same mistake, they almost certainly would realize it after being reminded that there is no wolf, but LLMs often insist there is. o1-preview (but not -mini) seems to have cracked it, though.
- anonzzzies 2y ago
  Yep. Try it with your neighbours. It's not bad or offensive; it is just what it is.
marcosdumay 2y ago
It's a magician-like trick. The double-r gets all the attention because one needs to consciously remember it, so nobody remembers to look at the other single-r. The LLMs probably get it wrong because people get it wrong.

cynicalpeace 2y ago

Something I like to tell it to do is actually to respond using only a certain number of words. Sorta like min token length rather than max token length. Haven't found a model that does it reliably yet.

roywiggins 2y ago

I also think it's kind of a silly example- smart people can be functionally illiterate, after all. It doesn't tell you that much. My favorite LLM stumper is asking them to solve the farmer/wolf/chicken/grain river-crossing puzzle but with only a chicken. They usually either insist on pointless extra trips or hallucinate a wolf or grain. Liquid somehow does both and also loses track of what's where. > The farmer can take the chicken across the river first. Then, he can go back to the original side and bring the boat back. He can then take the chicken back to the original side and leave it there. Next, he can take a bag of grain across the river. He can then go back to the original side and bring the chicken across the river. Finally, he can go back to the original side one last time and bring the chicken across the river. One generation ended like this: > the farmer and the chicken are both on the other side of the river, and the chicken is safe from the grain.

nateglims 2y ago

Being openly annoyed by it will just lead to it being used more. That's sort of a fundamental behavior of the internet.

viccis 2y ago

It's also the case that the more people talk about it, the more future models will "solve" it just by having learned from discussion about it.