5 ms·

Since language often describes the world, I think a good language model must include a world model

YeGoblynQueenne 3y ago
Well, this has been claimed often, namely by some of the people who have developed statistical language models [1] but it's really not obvious how that should work. Where would a language model find the world model? Where would it store it? And why would it even need it? Obviously I don't know how the human linguistic ability works but it's clear that for us, text, words, language, isn't carrying around with it a representation of the entire world. So where would a language model find it in its training text? For example if I say "I sat by the sea today", there's no representation of the sea in the word "sea", rather, the representation is inside my head and yours, and the word "sea" is only used to point to that representation [2]. Why would it ever be possible to derive the representation just from the pointer? Like I say, it's really not obvious and I don't think anyone has given a convincing explanation - it's usually words all the way down when people try. In any case, it seems that humans had most of the capabilities conferred by a world model (which we share with other animals) before we had language (which we do not - share with other animals), so it seems likely that a world model must be developed first, before linguistic ability can be built on top of it. But that's a conjecture, of course. I guess maybe, in principle, it should be possible to treat all of human cognition as some kind of hidden (latent) variable and then train a model to represent it only from its expression in text, but we don't really have any kind of modelling technique with that power. That would be a truly omnipotent model. So powerful, indeed, that it wouldn't need to do that in the first place- it would already be a kind of machine god. ______________ [1] e.g. I believe Tomáš Mikolov, one of the creators of word2vec, has claimed something like that but I'm not sure if I have a source for my own claim; more recently Ilya Sutskever, of OpenAI, has said words to that effect- I have a link somewhere I think, if you are curious. [2] That's obvious because the word "sea" doesn't mean anything to someone who doesn't know English, but they'll probably have a word for the sea in their language, so they'll have the concept, without needing a specific word for it.
- circuit10 3y ago
  No one can experience the real world directly. Whether it’s through language, vision (which feels quite “real” but is really just a 2D projection of light that we interpret), sound or anything else, it’s all just some byproduct of the world that we nonetheless can make useful predictions with. LLMs are more limited in their input data, but I don’t think the difference is fundamental; it’s all just different abstract representations of the world. Like how by looking at the sea we can gain some useful understanding of how it works, an LLM can, via descriptions of the sea, pick up enough information about it to answer questions about it or write text about it. It wouldn’t “find” a world model, it learns it based on its training data, it gets stored in the weights (because there’s nowhere else to store it), and it needs it because it’s making predictions about text, which often describes the world. If you don’t have a basic idea of things like cause and effect it’s difficult to write coherent text
- xcv123 3y ago
  > Where would a language model find the world model? Where would it store it? And why would it even need it? It is pattern recognition with many layers of abstraction. Obviously it will infer semantic relations at some level. It is a type of machine learning. The entire point of machine learning is to generate a model of the data which can be generalized to new inputs. It would find the world model in semantic relations in text. It would store it in its vast neural network. It would need a world model in order to perform natural language understanding tasks, which is exactly what it was designed to do. When asked about the world it needs a model of the world in order to generate useful answers. GPT-4 took months to train on a supercomputer and it generated a neural network of hundreds of gigabytes. What exactly was that supercomputer doing for several months and what exactly would the neural network represent if not a world model? Are you thinking it rote learned answers to every possible question? > so it seems likely that a world model must be developed first It is "a" model. It can be completely alien to whatever exists in a human mind but still function and exist as some type of model. > Why would it ever be possible to derive the representation just from the pointer? It only needs to have a sufficient representation of the Sea that is useful for the tasks we have trained it to do. It doesn't need to derive "the" one true representation that humans have of the Sea, whatever that is.
corethree 3y ago
A good language model IS a world model. They can be one in the same. Very likely what's going on with something like chatGPT. The world model is simply encoded as text.

Well, this has been claimed often, namely by some of the people who have developed statistical language models [1] but it's really not obvious how that should work. Where would a language model find the world model? Where would it store it? And why would it even need it? Obviously I don't know how the human linguistic ability works but it's clear that for us, text, words, language, isn't carrying around with it a representation of the entire world. So where would a language model find it in its training text? For example if I say "I sat by the sea today", there's no representation of the sea in the word "sea", rather, the representation is inside my head and yours, and the word "sea" is only used to point to that representation [2]. Why would it ever be possible to derive the representation just from the pointer? Like I say, it's really not obvious and I don't think anyone has given a convincing explanation - it's usually words all the way down when people try. In any case, it seems that humans had most of the capabilities conferred by a world model (which we share with other animals) before we had language (which we do not - share with other animals), so it seems likely that a world model must be developed first, before linguistic ability can be built on top of it. But that's a conjecture, of course. I guess maybe, in principle, it should be possible to treat all of human cognition as some kind of hidden (latent) variable and then train a model to represent it only from its expression in text, but we don't really have any kind of modelling technique with that power. That would be a truly omnipotent model. So powerful, indeed, that it wouldn't need to do that in the first place- it would already be a kind of machine god. ______________ [1] e.g. I believe Tomáš Mikolov, one of the creators of word2vec, has claimed something like that but I'm not sure if I have a source for my own claim; more recently Ilya Sutskever, of OpenAI, has said words to that effect- I have a link somewhere I think, if you are curious. [2] That's obvious because the word "sea" doesn't mean anything to someone who doesn't know English, but they'll probably have a word for the sea in their language, so they'll have the concept, without needing a specific word for it.

circuit10 3y ago
No one can experience the real world directly. Whether it’s through language, vision (which feels quite “real” but is really just a 2D projection of light that we interpret), sound or anything else, it’s all just some byproduct of the world that we nonetheless can make useful predictions with. LLMs are more limited in their input data, but I don’t think the difference is fundamental; it’s all just different abstract representations of the world. Like how by looking at the sea we can gain some useful understanding of how it works, an LLM can, via descriptions of the sea, pick up enough information about it to answer questions about it or write text about it. It wouldn’t “find” a world model, it learns it based on its training data, it gets stored in the weights (because there’s nowhere else to store it), and it needs it because it’s making predictions about text, which often describes the world. If you don’t have a basic idea of things like cause and effect it’s difficult to write coherent text
xcv123 3y ago
> Where would a language model find the world model? Where would it store it? And why would it even need it? It is pattern recognition with many layers of abstraction. Obviously it will infer semantic relations at some level. It is a type of machine learning. The entire point of machine learning is to generate a model of the data which can be generalized to new inputs. It would find the world model in semantic relations in text. It would store it in its vast neural network. It would need a world model in order to perform natural language understanding tasks, which is exactly what it was designed to do. When asked about the world it needs a model of the world in order to generate useful answers. GPT-4 took months to train on a supercomputer and it generated a neural network of hundreds of gigabytes. What exactly was that supercomputer doing for several months and what exactly would the neural network represent if not a world model? Are you thinking it rote learned answers to every possible question? > so it seems likely that a world model must be developed first It is "a" model. It can be completely alien to whatever exists in a human mind but still function and exist as some type of model. > Why would it ever be possible to derive the representation just from the pointer? It only needs to have a sufficient representation of the Sea that is useful for the tasks we have trained it to do. It doesn't need to derive "the" one true representation that humans have of the Sea, whatever that is.

corethree 3y ago

A good language model IS a world model. They can be one in the same. Very likely what's going on with something like chatGPT. The world model is simply encoded as text.