6 ms·

If it truly understood what things mean, then it would be able to tell me how many r's are in the word strawberry. But it messes something so simple up because

If it truly understood what things mean, then it would be able to tell me how many r's are in the word strawberry. But it messes something so simple up because it doesn't actually understand things. It's just doing math, and the math has holes and limitations in how it works that causes simple errors like this. If it was truly understanding, then it should be able to understand and figure out how to work around these such limitations in the math. At least in my opinion.

brookst 2y ago
The limitations on processing letters aren’t in the math, they are in the encoding. Language is the map, and concepts are the territory. You may as well complain that someone doesn’t really understand their neighborhood if they can’t find it on a map.
- SirMaster 2y ago
  >they are in the encoding Is encoding not math?
  - TeMPOraL 2y ago
    It's math, but specifically an independent piece you could swap out for a different one that does much better on this problem (e.g. use characters instead of tokens) - it's just doing so would make training and inference much more expensive (read: much worse model performance for a given training/compute budget), so it's not worth the trade-off. It's not like humans read letter by letter either, at least not past the age of 6 or such. They can, if needed, but it requires extra effort. Same is true with LLMs.
    - SirMaster 2y ago
      But that's really what I meant. When you say the limitation on processing is not in the math. I would say it is a mathematical limitation of processing because they had to choose a math that works on parts of words instead of letters due to the limitation of the power of the math that can be done for training and inference. They chose to use some limiting math which prevents the LLM from being able to easily answer questions like this. It's not a limitation of math in general. It's a limitation of the math they chose to build the LLM on which is what was going through my head when I was writing it.
      - brookst 2y ago
        The LLM only sees tokens. The limitation is in the E2E product because of the encoder chosen. Change the encoder, keep the LLM, different limitations appear. Perhaps it’s a pedantic difference, but to someone in the field the complaint reads like saying TCP/IP is deficient because it doesn’t support encryption: technically true but missing context about the whole stack.
ben_w 2y ago
That's like saying I don't understand what vanilla flavour means just because I can't tell you how many hydrogen atoms vanillin contains — my sense of smell just doesn't do that, and an LLM just isn't normally tokenised in a way to count letters. What I can do, is google it. And an LLM trained on an appropriate source that creates a mapping from nearly-a-whole-word tokens into letter-tokens, that model can (in principle) learn to count the letters in some word.
- rootusrootus 2y ago
  I think it's closer to giving you a diagram of the vanillin molecule and then asking you how many hydrogen atoms you see.
  - ben_w 2y ago
    I'm not clear why you think that's closer? The very first thing that happens in most LLMs is that information getting deleted by the letters getting converted into a token stream.
    - kaoD 2y ago
      That doesn't explain why LLMs can't understand how many letters are in their tokens.
      - Terr_ 2y ago
        If I may, I think you both may be talking slightly past one another. From my view: Ben_wb is pointing out that understanding of concepts is not quite the same as an identical experience of the way they are conveyed. I can use a translation app to to correspond with someone who only knows Mandarin, and they'll understand the concept of vowels, even if they can't tell me how many vowels are in the original sentence I wrote, because that sentence was lost in translation. KaoD is pointing out that if the system really understands anything nearly as well as it first appears, it should still perform better than it does. My hypothetical Chinese pen-pal would at least be able to recognize and explain why they can't answer me, even if they don't have all the information to solve it.
orangecat 2y ago
But it messes something so simple up because it doesn't actually understand things. Meanwhile on the human side: https://neuroscienceresearch.wustl.edu/how-your-mind-plays-tricks-on-you/ https://neuroscienceresearch.wustl.edu/how-your-mind-plays-t...
CamperBob2 2y ago
If it truly understood what things mean, then it would be able to tell me how many r's are in the word strawberry. How about if it recognized its limitations with regard to introspecting its tokenization process, and wrote and ran a Python program to count the r's? Would that change your opinion? Why or why not?
- SirMaster 2y ago
  Certainly a step in the right direction. For an entity to understand the context and its limitations and find a way to work with what it can do.
  - CamperBob2 2y ago
    Right, and that's basically what it does in plenty of other domains now, when you ask it to deal with something quantitative. Pretty cool.
- dizhn 2y ago
  Would it even know when to resort to this ?

brookst 2y ago

The limitations on processing letters aren’t in the math, they are in the encoding. Language is the map, and concepts are the territory. You may as well complain that someone doesn’t really understand their neighborhood if they can’t find it on a map.

SirMaster 2y ago
>they are in the encoding Is encoding not math?
- TeMPOraL 2y ago
  It's math, but specifically an independent piece you could swap out for a different one that does much better on this problem (e.g. use characters instead of tokens) - it's just doing so would make training and inference much more expensive (read: much worse model performance for a given training/compute budget), so it's not worth the trade-off. It's not like humans read letter by letter either, at least not past the age of 6 or such. They can, if needed, but it requires extra effort. Same is true with LLMs.
  - SirMaster 2y ago
    But that's really what I meant. When you say the limitation on processing is not in the math. I would say it is a mathematical limitation of processing because they had to choose a math that works on parts of words instead of letters due to the limitation of the power of the math that can be done for training and inference. They chose to use some limiting math which prevents the LLM from being able to easily answer questions like this. It's not a limitation of math in general. It's a limitation of the math they chose to build the LLM on which is what was going through my head when I was writing it.
    - brookst 2y ago
      The LLM only sees tokens. The limitation is in the E2E product because of the encoder chosen. Change the encoder, keep the LLM, different limitations appear. Perhaps it’s a pedantic difference, but to someone in the field the complaint reads like saying TCP/IP is deficient because it doesn’t support encryption: technically true but missing context about the whole stack.

ben_w 2y ago

That's like saying I don't understand what vanilla flavour means just because I can't tell you how many hydrogen atoms vanillin contains — my sense of smell just doesn't do that, and an LLM just isn't normally tokenised in a way to count letters. What I can do, is google it. And an LLM trained on an appropriate source that creates a mapping from nearly-a-whole-word tokens into letter-tokens, that model can (in principle) learn to count the letters in some word.

rootusrootus 2y ago
I think it's closer to giving you a diagram of the vanillin molecule and then asking you how many hydrogen atoms you see.
- ben_w 2y ago
  I'm not clear why you think that's closer? The very first thing that happens in most LLMs is that information getting deleted by the letters getting converted into a token stream.
  - kaoD 2y ago
    That doesn't explain why LLMs can't understand how many letters are in their tokens.
    - Terr_ 2y ago
      If I may, I think you both may be talking slightly past one another. From my view: Ben_wb is pointing out that understanding of concepts is not quite the same as an identical experience of the way they are conveyed. I can use a translation app to to correspond with someone who only knows Mandarin, and they'll understand the concept of vowels, even if they can't tell me how many vowels are in the original sentence I wrote, because that sentence was lost in translation. KaoD is pointing out that if the system really understands anything nearly as well as it first appears, it should still perform better than it does. My hypothetical Chinese pen-pal would at least be able to recognize and explain why they can't answer me, even if they don't have all the information to solve it.

orangecat 2y ago

But it messes something so simple up because it doesn't actually understand things. Meanwhile on the human side: https://neuroscienceresearch.wustl.edu/how-your-mind-plays-tricks-on-you/ https://neuroscienceresearch.wustl.edu/how-your-mind-plays-t...

CamperBob2 2y ago

If it truly understood what things mean, then it would be able to tell me how many r's are in the word strawberry. How about if it recognized its limitations with regard to introspecting its tokenization process, and wrote and ran a Python program to count the r's? Would that change your opinion? Why or why not?

SirMaster 2y ago
Certainly a step in the right direction. For an entity to understand the context and its limitations and find a way to work with what it can do.
- CamperBob2 2y ago
  Right, and that's basically what it does in plenty of other domains now, when you ask it to deal with something quantitative. Pretty cool.
dizhn 2y ago
Would it even know when to resort to this ?