2 ms·
> If it had no model of what it was writing about it wouldn’t be able to write anything coherent. It has a model of the structure of writing on the internet. I
by JohnFen 4y ago
> If it had no model of what it was writing about it wouldn’t be able to write anything coherent.
It has a model of the structure of writing on the internet. It can produce text that adheres to that model. That's all that's required for coherent writing.
- circuit10 4y agoAnd the structure of writing in the Internet includes a lot about the world, so a good model of that must contain some form of world model to make accurate predictions
- JohnFen 4y agoThe only model necessary is that of the writing. So if the writing so far is "He went to the bar", the LLM can know just from the language model that a reasonable next fragment could be "...to have a drink". That doesn't require or imply that the concept of "bar", "drink", or even "He" or the concept of motion has been modelled. The model is of the patterns of tokens (words/characters/whatever).