3 ms·

Asking an LLM about letters is like asking a human about the individual wavelength that make up color. They might be able to infer it from text they read, but t

Asking an LLM about letters is like asking a human about the individual wavelength that make up color. They might be able to infer it from text they read, but their sensory system is simply incapable of detecting any of that directly.

Jensson 3y ago
Not comparable at all, humans cant determine wavelength by sight since it isn't a perfect representation of the data. LLMs however can determine the letters based on its tokens, since those tokens are a perfect representation of the data. We already know they can do this, just ask it to translate to Base64 or something similar, it knows perfectly well what letters the words are made up of.
- vidarh 3y ago
  LLMs need to learn those mappings, because they're not inherent in the tokens, the same way humans need to learn mappings from colours to wavelengths. That one is not a perfect one to one representation of the data is entirely irrelevant. That they can learn it in a given context does not mean they have learnt every relationship the same way humans miss all kinds of relationships as well without being pointed right at them. There's no doubt that LLMs are "dumb" in many aspects, but without exploring why they fail that doesn't tell us whether this is some inherent lack of ability to represent reasoning or understanding, or simply holes in their training. There's no reason to assume the training data LLMs currently learn on is in any way equivalent to the data a human child is exposed to growing up, so there is no reason to assume it will have the same strengths and weaknesses in what it is able to reason about and that we can therefore draw conclusions about it's overall ability just from probing something that appears like it should be simple to a human. (and incidentally, I think people here will overestimate massively how well humans would do on the consonant test above; even native English speakers without any recent exposure to being taught rules as opposed to "just" using the language - many would be able to do it, but I'd be able to bet many would struggle, though most who'd struggle would express that doubt)

Jensson 3y ago

Not comparable at all, humans cant determine wavelength by sight since it isn't a perfect representation of the data. LLMs however can determine the letters based on its tokens, since those tokens are a perfect representation of the data. We already know they can do this, just ask it to translate to Base64 or something similar, it knows perfectly well what letters the words are made up of.

vidarh 3y ago
LLMs need to learn those mappings, because they're not inherent in the tokens, the same way humans need to learn mappings from colours to wavelengths. That one is not a perfect one to one representation of the data is entirely irrelevant. That they can learn it in a given context does not mean they have learnt every relationship the same way humans miss all kinds of relationships as well without being pointed right at them. There's no doubt that LLMs are "dumb" in many aspects, but without exploring why they fail that doesn't tell us whether this is some inherent lack of ability to represent reasoning or understanding, or simply holes in their training. There's no reason to assume the training data LLMs currently learn on is in any way equivalent to the data a human child is exposed to growing up, so there is no reason to assume it will have the same strengths and weaknesses in what it is able to reason about and that we can therefore draw conclusions about it's overall ability just from probing something that appears like it should be simple to a human. (and incidentally, I think people here will overestimate massively how well humans would do on the consonant test above; even native English speakers without any recent exposure to being taught rules as opposed to "just" using the language - many would be able to do it, but I'd be able to bet many would struggle, though most who'd struggle would express that doubt)