3 ms·
Between the input tokens and output tokens there are like 10 layers of mystery meat, though. I think while at the end of it all it's a fancy autocomplete, there
by vore 4y ago
Between the input tokens and output tokens there are like 10 layers of mystery meat, though. I think while at the end of it all it's a fancy autocomplete, there is a lot going on inside the model before it spits out the next token.
- tjr 4y agoMy current opinion is that it's not so much that GPT is utterly amazing, as it is that language is even more amazing than we previously realized, and GPT happens to have been trained on a huge quantity of language material.
- Sharlin 4y agoI was just thinking about this, and the implications it could have on our understanding of the origin and development of human cognitive capabilities, both in the evolutionary and developmental psychology senses.
- dwaltrip 4y agoLanguage, we are learning, happens to correlate extremely well with the actual structure of the world. In hindsight, it’s not that surprising. The entire purpose of language is to help humans send information about the world to each other! To predict the next word very accurately for a huge variety of texts, a model must have some amount of real understanding. Sure, it may look different than human understanding. But there’s no way around this. GPT could not predict so accurately if it didn’t have any understanding the world.
- tjr 4y agoThese have historically often been thought of separately: knowledge representation was the data, and language was the interface. To at least some degree, maybe language is enough for both. Language doesn't just express concepts, but it contains them.