3 ms·
How would you represent or interpret the "next concept" if not with some kind of token, though? Language is a communicated abstraction of concepts, and it woul
by niam 3y ago
How would you represent or interpret the "next concept" if not with some kind of token, though?
Language is a communicated abstraction of concepts, and it would seem that internal representations of those concepts can emerge from something optimized for token prediction. Or at least: internal representations of the speaker to be predicted, including the knowledge they may possess.
- niemandhier 3y agoLanguage is indeed communicated abstraction of concepts, but it emerged under a lot of constraints (our auditory system, our brains inherent bias towards visual stimuli etc. ). Predicting in this constrained system most likely is suboptimal. Imagine translating language into an optimized representation free from human constraints, doing autoregressive prediction in this domain and only than translate back. As far as I understand current models, this is not yet how they work.