4 ms·

Well, this has been claimed often, namely by some of the people who have developed statistical language models [1] but it's really not obvious how that should w

Well, this has been claimed often, namely by some of the people who have developed statistical language models [1] but it's really not obvious how that should work. Where would a language model find the world model? Where would it store it? And why would it even need it? Obviously I don't know how the human linguistic ability works but it's clear that for us, text, words, language, isn't carrying around with it a representation of the entire world. So where would a language model find it in its training text? For example if I say "I sat by the sea today", there's no representation of the sea in the word "sea", rather, the representation is inside my head and yours, and the word "sea" is only used to point to that representation [2]. Why would it ever be possible to derive the representation just from the pointer? Like I say, it's really not obvious and I don't think anyone has given a convincing explanation - it's usually words all the way down when people try. In any case, it seems that humans had most of the capabilities conferred by a world model (which we share with other animals) before we had language (which we do not - share with other animals), so it seems likely that a world model must be developed first, before linguistic ability can be built on top of it. But that's a conjecture, of course. I guess maybe, in principle, it should be possible to treat all of human cognition as some kind of hidden (latent) variable and then train a model to represent it only from its expression in text, but we don't really have any kind of modelling technique with that power. That would be a truly omnipotent model. So powerful, indeed, that it wouldn't need to do that in the first place- it would already be a kind of machine god. ______________ [1] e.g. I believe Tomáš Mikolov, one of the creators of word2vec, has claimed something like that but I'm not sure if I have a source for my own claim; more recently Ilya Sutskever, of OpenAI, has said words to that effect- I have a link somewhere I think, if you are curious. [2] That's obvious because the word "sea" doesn't mean anything to someone who doesn't know English, but they'll probably have a word for the sea in their language, so they'll have the concept, without needing a specific word for it.

circuit10 3y ago
No one can experience the real world directly. Whether it’s through language, vision (which feels quite “real” but is really just a 2D projection of light that we interpret), sound or anything else, it’s all just some byproduct of the world that we nonetheless can make useful predictions with. LLMs are more limited in their input data, but I don’t think the difference is fundamental; it’s all just different abstract representations of the world. Like how by looking at the sea we can gain some useful understanding of how it works, an LLM can, via descriptions of the sea, pick up enough information about it to answer questions about it or write text about it. It wouldn’t “find” a world model, it learns it based on its training data, it gets stored in the weights (because there’s nowhere else to store it), and it needs it because it’s making predictions about text, which often describes the world. If you don’t have a basic idea of things like cause and effect it’s difficult to write coherent text
xcv123 3y ago
> Where would a language model find the world model? Where would it store it? And why would it even need it? It is pattern recognition with many layers of abstraction. Obviously it will infer semantic relations at some level. It is a type of machine learning. The entire point of machine learning is to generate a model of the data which can be generalized to new inputs. It would find the world model in semantic relations in text. It would store it in its vast neural network. It would need a world model in order to perform natural language understanding tasks, which is exactly what it was designed to do. When asked about the world it needs a model of the world in order to generate useful answers. GPT-4 took months to train on a supercomputer and it generated a neural network of hundreds of gigabytes. What exactly was that supercomputer doing for several months and what exactly would the neural network represent if not a world model? Are you thinking it rote learned answers to every possible question? > so it seems likely that a world model must be developed first It is "a" model. It can be completely alien to whatever exists in a human mind but still function and exist as some type of model. > Why would it ever be possible to derive the representation just from the pointer? It only needs to have a sufficient representation of the Sea that is useful for the tasks we have trained it to do. It doesn't need to derive "the" one true representation that humans have of the Sea, whatever that is.
- YeGoblynQueenne 3y ago
  >> GPT-4 took months to train on a supercomputer and it generated a neural network of hundreds of gigabytes. What exactly was that supercomputer doing for several months and what exactly would the neural network represent if not a world model? I believe GPT-4 was trained on a server farm, not a single computer. In any case what it was doing all that time was going over and over the text in its gigantic training corpus, which was of petabyte size, and optimising the objective P(tₖ|tₖ₋ₙ , ..., tₖ₋₁) i.e. the probability of token k given a "sliding window" of the n preceding (or surrounding) tokens. There is nothing in this objective that needs a world model, and it is really not obvious why optimising this objective should lead to development of a world model, rather than, or in addition to, a model of the training corpus. It is easy to see how this stuff works. You can train your own language model easily, although of course it would have to be a smaller language model. For example, you can train a Hidden Markov Model on the text of freely available literary works on project Guttenberg, or on wikipedia pages, and without too much compute (an ordinary laptop will do). In fact, I recommend that as an exercise and as an experiment to gain a better understanding in how language modelling works, for those who are curious about questions regarding their ability to model something beyond text. A good textbook to begin with statistical language modelling is "Foundations of Statistical Natural Language Processing" by Manning and Schűtze: https://nlp.stanford.edu/fsnlp/ https://nlp.stanford.edu/fsnlp/ Or, just as good, "Speech and Language Processing" by Jurafsky and Martin: https://web.stanford.edu/~jurafsky/slp3/ https://web.stanford.edu/~jurafsky/slp3/ Or, if you don't have the time for an entire textbook, "Statistical Language Learning" by Eugene Charniak is an excellent, concise introduction to the subject: https://archive.org/details/statisticallangu0000char https://archive.org/details/statisticallangu0000char
xcv123 3y ago
> Why would it ever be possible to derive the representation just from the pointer? Of course it cannot derive a representation from just a pointer. Neither humans or machines can do that. The words are not pointers in isolation. They are connected in a network of semantic relations. The model is in the relationships between pointers. Artificial neural network.
- YeGoblynQueenne 3y ago
  The question then becomes where do the "semantic" relations in the network come from. Unless there is some way to extract the meaning of words with respect to the real-world concepts they refer to, from text, any relation between words that your model, models is not going to be semantic (i.e. it won't represent meaning). Like I say in another comment, this kind of explanation is "words all the way down", but it really doesn't explain where the process ends (or begins) and where the meaning comes into the words. To give you an example, here is a small semantic network that I just created by hand, that represents is-a and has-a relations: abc32 is-a c356 7yt4 is-a c356 c356 has-a 902a c356 has-a 8773 The names of entities in the network, like abc32, replace real English words that I just changed manually (or, well, vim-illy). You can use the is-a and has-a relations in the network to infer e.g. that abc32 has-a 902a, and that 7yt4 and abc32 are things of the same kind, but go ahead and try to understand what the real-world entities I renamed were. You can't- and neither can any model that is trained on text consisting of words whose meaning it understands as much as you understand the meaning of abc32 and 7yt4. Wikipedia has an article on semantic networks: https://en.wikipedia.org/wiki/Semantic_network https://en.wikipedia.org/wiki/Semantic_network I haven't read it but from a quick glance it looks informative.
  - xcv123 3y ago
    > but go ahead and try to understand what the real-world entities I renamed were. Obviously not. Insufficient data. Try that substitution exercise again with terabytes of data (Common Crawl dataset) then feed it into a neural network and see if it can find any patterns.
    - deleted 3y ago

circuit10 3y ago

No one can experience the real world directly. Whether it’s through language, vision (which feels quite “real” but is really just a 2D projection of light that we interpret), sound or anything else, it’s all just some byproduct of the world that we nonetheless can make useful predictions with. LLMs are more limited in their input data, but I don’t think the difference is fundamental; it’s all just different abstract representations of the world. Like how by looking at the sea we can gain some useful understanding of how it works, an LLM can, via descriptions of the sea, pick up enough information about it to answer questions about it or write text about it. It wouldn’t “find” a world model, it learns it based on its training data, it gets stored in the weights (because there’s nowhere else to store it), and it needs it because it’s making predictions about text, which often describes the world. If you don’t have a basic idea of things like cause and effect it’s difficult to write coherent text

xcv123 3y ago

> Where would a language model find the world model? Where would it store it? And why would it even need it? It is pattern recognition with many layers of abstraction. Obviously it will infer semantic relations at some level. It is a type of machine learning. The entire point of machine learning is to generate a model of the data which can be generalized to new inputs. It would find the world model in semantic relations in text. It would store it in its vast neural network. It would need a world model in order to perform natural language understanding tasks, which is exactly what it was designed to do. When asked about the world it needs a model of the world in order to generate useful answers. GPT-4 took months to train on a supercomputer and it generated a neural network of hundreds of gigabytes. What exactly was that supercomputer doing for several months and what exactly would the neural network represent if not a world model? Are you thinking it rote learned answers to every possible question? > so it seems likely that a world model must be developed first It is "a" model. It can be completely alien to whatever exists in a human mind but still function and exist as some type of model. > Why would it ever be possible to derive the representation just from the pointer? It only needs to have a sufficient representation of the Sea that is useful for the tasks we have trained it to do. It doesn't need to derive "the" one true representation that humans have of the Sea, whatever that is.

YeGoblynQueenne 3y ago
>> GPT-4 took months to train on a supercomputer and it generated a neural network of hundreds of gigabytes. What exactly was that supercomputer doing for several months and what exactly would the neural network represent if not a world model? I believe GPT-4 was trained on a server farm, not a single computer. In any case what it was doing all that time was going over and over the text in its gigantic training corpus, which was of petabyte size, and optimising the objective P(tₖ|tₖ₋ₙ , ..., tₖ₋₁) i.e. the probability of token k given a "sliding window" of the n preceding (or surrounding) tokens. There is nothing in this objective that needs a world model, and it is really not obvious why optimising this objective should lead to development of a world model, rather than, or in addition to, a model of the training corpus. It is easy to see how this stuff works. You can train your own language model easily, although of course it would have to be a smaller language model. For example, you can train a Hidden Markov Model on the text of freely available literary works on project Guttenberg, or on wikipedia pages, and without too much compute (an ordinary laptop will do). In fact, I recommend that as an exercise and as an experiment to gain a better understanding in how language modelling works, for those who are curious about questions regarding their ability to model something beyond text. A good textbook to begin with statistical language modelling is "Foundations of Statistical Natural Language Processing" by Manning and Schűtze: https://nlp.stanford.edu/fsnlp/ https://nlp.stanford.edu/fsnlp/ Or, just as good, "Speech and Language Processing" by Jurafsky and Martin: https://web.stanford.edu/~jurafsky/slp3/ https://web.stanford.edu/~jurafsky/slp3/ Or, if you don't have the time for an entire textbook, "Statistical Language Learning" by Eugene Charniak is an excellent, concise introduction to the subject: https://archive.org/details/statisticallangu0000char https://archive.org/details/statisticallangu0000char

xcv123 3y ago

> Why would it ever be possible to derive the representation just from the pointer? Of course it cannot derive a representation from just a pointer. Neither humans or machines can do that. The words are not pointers in isolation. They are connected in a network of semantic relations. The model is in the relationships between pointers. Artificial neural network.

YeGoblynQueenne 3y ago
The question then becomes where do the "semantic" relations in the network come from. Unless there is some way to extract the meaning of words with respect to the real-world concepts they refer to, from text, any relation between words that your model, models is not going to be semantic (i.e. it won't represent meaning). Like I say in another comment, this kind of explanation is "words all the way down", but it really doesn't explain where the process ends (or begins) and where the meaning comes into the words. To give you an example, here is a small semantic network that I just created by hand, that represents is-a and has-a relations: abc32 is-a c356 7yt4 is-a c356 c356 has-a 902a c356 has-a 8773 The names of entities in the network, like abc32, replace real English words that I just changed manually (or, well, vim-illy). You can use the is-a and has-a relations in the network to infer e.g. that abc32 has-a 902a, and that 7yt4 and abc32 are things of the same kind, but go ahead and try to understand what the real-world entities I renamed were. You can't- and neither can any model that is trained on text consisting of words whose meaning it understands as much as you understand the meaning of abc32 and 7yt4. Wikipedia has an article on semantic networks: https://en.wikipedia.org/wiki/Semantic_network https://en.wikipedia.org/wiki/Semantic_network I haven't read it but from a quick glance it looks informative.
- xcv123 3y ago
  > but go ahead and try to understand what the real-world entities I renamed were. Obviously not. Insufficient data. Try that substitution exercise again with terabytes of data (Common Crawl dataset) then feed it into a neural network and see if it can find any patterns.
  - deleted 3y ago