4 ms·

No. We know that LLMs make world models. (Often literally: https://twitter.com/wesg52/status/1709551516577902782 https://twitter.com/wesg52/status/1709551516577

by wilg 3y ago

No. We know that LLMs make world models. (Often literally: https://twitter.com/wesg52/status/1709551516577902782 https://twitter.com/wesg52/status/1709551516577902782). I should also point out that your tone is aggressive, so it makes me think you're not interested in learning. But I'm going to proceed anyway. It cannot understand these properties of words well because it does not operate on words, it operates on tokens. You can see examples here: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer This makes it much more difficult to learn how to reason about particular data that appears inside the token, because it does not ever receive that information. The only way it could get that information is if the training data explicitly attempted to work around this limitation. But I would also appreciate insight from someone with a deeper understanding of the internals of these big LLMs.

Jensson 3y ago
> I should also point out that your tone is aggressive, so it makes me think you're not interested in learning Why do you think that? Aggressiveness is the best way to get responses, people don't like it but it makes people respond to you. And for that matter I have a pretty good understanding about this topic, it is kind of annoying when people try to school you then. The "it gets tokenized data" is just a cop out response by people who don't understand the problem. > It cannot understand these properties of words without them being in the training data because it does not operate on words, it operates on tokens But those properties are in the training data. We know the LLM can answer these sort of questions when asked directly for simpler cases. But when asked to do something that requires it to draw from many different parts it fails. It didn't fail due to the data not being there, it failed due to not understanding that it should use that data. > This makes it much more difficult to learn how to reason about particular data that appears _inside the token_, because _it does not ever receive that information_ Right, the structure of LLM makes these sort of questions harder for it. But they aren't impossible or unfair, nothing prevents an LLM model from solving this sort of question. The main reason it fails is that it tries to write it like a human would, as it isn't trained to solve problems it is trained to mimic humans, it is too dumb to figure out ways to solve it on its own. And to show you I understand how these models works, the most efficient way to solve this question would be to solve it like a human. If it wrote out the steps "try next word as 'Blah'" and then verify those words one at a time, likely it would succeed. And since we know LLMs work like that we could try to make it output the results in that way, and that would improve performance. However, a smart agent would understand that its answer was wrong in that case by itself, and change its answering style on its own to fit the problem. But it can't think like that, it just tries to write it like answers it has seen, it doesn't do any verification since the answers it saw didn't verify, it doesn't spell things out here since the answers it saw didn't spell things out etc.
- wilg 3y ago
  I’m not interested in continuing to talk to you.
  - Jensson 3y ago
    If you don't want to talk more stop posting instead of posting that you want to stop posting. If everyone who wanted to stop posting posted that they want to stop posting it would clutter the board too much.
    - wilg 3y ago
      Let me be clearer, you’re frustrating to engage with on this topic, which you should reconsider. And other posters should consider avoiding engaging with you on this.
      - Jensson 3y ago
        Well, I am trying to explain why your original meme post "it just reads tokens, it can't do that" is wrong. That meme is tiring, and the most tiring with trying to fight such memes is that so many comes and repeats it and it is very hard to change their mind. But yeah, I should have been more clear to target that meme so that you didn't feel it was about you. It is easy to accidentally make things a bit too personal.
        wilg 3y ago
        OK, thanks. I'm not claiming its impossible for an LLM to learn this, just that it appears to be much harder (perhaps by requiring more data) when the task involves fighting against tokenization. For example, it can base64 encode your phrase "What was once dormant may soon angrily awake" correctly, and it can reverse the letters correctly. But it fails to do the same on the example text from the OpenAI tokenizer.
telotortium 3y ago
> This makes it much more difficult to learn how to reason about particular data that appears inside the token, because it does not ever receive that information. More difficult, but it shouldn't be impossible. The Chinese managed to make rhyming dictionaries over 1000 years ago despite using a non-phonetic writing system. More relevantly, GPT-4 has definitely ingested both regular and rhyming dictionaries.
- wilg 3y ago
  I agree it should be possible.
- vidarh 3y ago
  Possible, certainly, but given the difficulties it has with it, it appears it likely requires more training. Unlike with Chinese, the LLMs will not benefit from extensive training data mapping its tokens to and from sounds or rhyming with other words, but need to derive during training from data that rarely mentions these tokens at all.
maxdoop 3y ago
What does it mean to understand?

by wilg 3y ago

Jensson 3y ago

> I should also point out that your tone is aggressive, so it makes me think you're not interested in learning Why do you think that? Aggressiveness is the best way to get responses, people don't like it but it makes people respond to you. And for that matter I have a pretty good understanding about this topic, it is kind of annoying when people try to school you then. The "it gets tokenized data" is just a cop out response by people who don't understand the problem. > It cannot understand these properties of words without them being in the training data because it does not operate on words, it operates on tokens But those properties are in the training data. We know the LLM can answer these sort of questions when asked directly for simpler cases. But when asked to do something that requires it to draw from many different parts it fails. It didn't fail due to the data not being there, it failed due to not understanding that it should use that data. > This makes it much more difficult to learn how to reason about particular data that appears _inside the token_, because _it does not ever receive that information_ Right, the structure of LLM makes these sort of questions harder for it. But they aren't impossible or unfair, nothing prevents an LLM model from solving this sort of question. The main reason it fails is that it tries to write it like a human would, as it isn't trained to solve problems it is trained to mimic humans, it is too dumb to figure out ways to solve it on its own. And to show you I understand how these models works, the most efficient way to solve this question would be to solve it like a human. If it wrote out the steps "try next word as 'Blah'" and then verify those words one at a time, likely it would succeed. And since we know LLMs work like that we could try to make it output the results in that way, and that would improve performance. However, a smart agent would understand that its answer was wrong in that case by itself, and change its answering style on its own to fit the problem. But it can't think like that, it just tries to write it like answers it has seen, it doesn't do any verification since the answers it saw didn't verify, it doesn't spell things out here since the answers it saw didn't spell things out etc.

wilg 3y ago
I’m not interested in continuing to talk to you.
- Jensson 3y ago
  If you don't want to talk more stop posting instead of posting that you want to stop posting. If everyone who wanted to stop posting posted that they want to stop posting it would clutter the board too much.
  - wilg 3y ago
    Let me be clearer, you’re frustrating to engage with on this topic, which you should reconsider. And other posters should consider avoiding engaging with you on this.
    - Jensson 3y ago
      Well, I am trying to explain why your original meme post "it just reads tokens, it can't do that" is wrong. That meme is tiring, and the most tiring with trying to fight such memes is that so many comes and repeats it and it is very hard to change their mind. But yeah, I should have been more clear to target that meme so that you didn't feel it was about you. It is easy to accidentally make things a bit too personal.
      - wilg 3y ago
        OK, thanks. I'm not claiming its impossible for an LLM to learn this, just that it appears to be much harder (perhaps by requiring more data) when the task involves fighting against tokenization. For example, it can base64 encode your phrase "What was once dormant may soon angrily awake" correctly, and it can reverse the letters correctly. But it fails to do the same on the example text from the OpenAI tokenizer.

telotortium 3y ago

> This makes it much more difficult to learn how to reason about particular data that appears inside the token, because it does not ever receive that information. More difficult, but it shouldn't be impossible. The Chinese managed to make rhyming dictionaries over 1000 years ago despite using a non-phonetic writing system. More relevantly, GPT-4 has definitely ingested both regular and rhyming dictionaries.

wilg 3y ago
I agree it should be possible.
vidarh 3y ago
Possible, certainly, but given the difficulties it has with it, it appears it likely requires more training. Unlike with Chinese, the LLMs will not benefit from extensive training data mapping its tokens to and from sounds or rhyming with other words, but need to derive during training from data that rarely mentions these tokens at all.

maxdoop 3y ago

What does it mean to understand?