2 ms·

Language Models don't see words. They see tokens. So asking them to perform minute operations on letters/syllables/words etc don't work well. Doesn't really hav

Language Models don't see words. They see tokens. So asking them to perform minute operations on letters/syllables/words etc don't work well. Doesn't really have anything to do with Intelligence. Sentences look something like this to the model, [754 76 4752 67 456]

woolion 4y ago
Here you have two predicates on words P: Sigma* -> N and T: Sigma* -> N. The question is give a list of words w such that for all w, P(w) and T(w) are both true. It has access to the correct values for P(w) and T(w), so there is a simple algorithm that can produce all words, you just enumerate them each one and verify that the word satisfies both predicates. An intelligent agent should be able to understand this without any understanding of the underlying domain. If I try to reformulate your answer, this is like words are a different domain than what GPT has been trained on (which is tokens), and so it is not unexpected that GPT cannot generalize to this other domain. Is that a correct interpretation of your statement? In my view failing to see the similarity in the relationship between class objects is a sign of a lack of 'intelligence', whatever that may be in a very precise sense.
- famouswaffles 4y ago
  >In my view failing to see the similarity in the relationship between class objects is a sign of a lack of 'intelligence', whatever that may be in a very precise sense. Ok but not all relationship similarities transfer across representations like that. It's like you give an instruction in English, "Find a word that rhymes with ingenious" to a french man. The french man sees, "Trouver un mot qui rime avec ingénieux". Guess what ? His response to "Trouver un mot qui rime avec ingénieux" would not fulfill the request in english that it might in French. Ingenious and heterogeneous rhyme. ingénieux and hétérogène have no such property The french man is not unintelligent to be unable to fulfill your request.

woolion 4y ago

Here you have two predicates on words P: Sigma* -> N and T: Sigma* -> N. The question is give a list of words w such that for all w, P(w) and T(w) are both true. It has access to the correct values for P(w) and T(w), so there is a simple algorithm that can produce all words, you just enumerate them each one and verify that the word satisfies both predicates. An intelligent agent should be able to understand this without any understanding of the underlying domain. If I try to reformulate your answer, this is like words are a different domain than what GPT has been trained on (which is tokens), and so it is not unexpected that GPT cannot generalize to this other domain. Is that a correct interpretation of your statement? In my view failing to see the similarity in the relationship between class objects is a sign of a lack of 'intelligence', whatever that may be in a very precise sense.

famouswaffles 4y ago
>In my view failing to see the similarity in the relationship between class objects is a sign of a lack of 'intelligence', whatever that may be in a very precise sense. Ok but not all relationship similarities transfer across representations like that. It's like you give an instruction in English, "Find a word that rhymes with ingenious" to a french man. The french man sees, "Trouver un mot qui rime avec ingénieux". Guess what ? His response to "Trouver un mot qui rime avec ingénieux" would not fulfill the request in english that it might in French. Ingenious and heterogeneous rhyme. ingénieux and hétérogène have no such property The french man is not unintelligent to be unable to fulfill your request.