5 ms·

They do not model the world. They present a statistical model of an existing corpus of text. If this existing corpus includes useful information it can regurg

by tovej 9mo ago

They do not model the world. They present a statistical model of an existing corpus of text. If this existing corpus includes useful information it can regurgitate that. It cannot, however, synthesize new facts by combining information from this corpus. The strongest thing you could feasibly claim is that the corpus itself models the world, and that the LLM is a surrogate for that model. But this is not true either. The corpus of human produced text is messy, containing mistakes, contradictions, and propaganda; it has to be interpreted by someone with an actual world model (a human) in order for it to be applied to any scrnario; your typical corpus is also biased towards internet discussions, the english language, and western prejudices.

aixpert 9mo ago
they do model the world. Watch Noble price winner Hinton or let's admit that this is more of a religious question then the technical.
- D-Machine 9mo ago
  They model the part of the world that (linguistic models of the world posted on the internet) try to model. But what is posted on the internet is not IRL. So, to be glib: LLMs trained on the internet do not model IRL, they model talking about IRL.
- imtringued 9mo ago
  His point is that human language and the written record is a model of the world, so if you train an LLM you're training a model of a model of the world. That sounds highly technical if you ask me. People complain if you recompress music or images with lossy codecs, but when an LLM does that suddenly it's religious?
  - hackinthebochs 9mo ago
    A model of a model of X is a model of X, albeit extra lossy.
    - tovej 9mo ago
      In this case this is not so. The primary model is not a model at all, and the surrogate has bias added to it. It's also missing any way to actually check the internal consistency of statements or otherwise combine information from its corpus, so it fails as a world model.
    - D-Machine 9mo ago
      An LLM has an internal linguistic model (i.e. it knows token patterns), and that linguistic model models humans' linguistic models (a stream of tokens) of their actual world models (which involve far, far more than linguistics and tokens, such as logical relations beyond mere semantic relations, sensory representations like imagery and sounds, and, yes, words and concepts). So LLMs are linguistic (token pattern) models of linguistic models (streams of tokens) describing world models (more than tokens). It thus does not in fact follow that LLMs model the world (as they are missing everything that is not encoded in non-linguistic semantics).
      - hackinthebochs 9mo ago
        At this point, anyone claiming that LLMs are "just" language models aren't arguing in good faith. LLMs are a general purpose computing paradigm. LLMs are circuit builders, the converged parameters define pathways through the architecture that pick out specific programs. Or as Karpathy puts it, LLMs are a differentiable computer[1]. Training LLMs discovers programs that well reproduce the input sequence. Tokens can represent anything, not just words. Roughly the same architecture can generate passable images, music, or even video. [1] https://x.com/karpathy/status/1582807367988654081 https://x.com/karpathy/status/1582807367988654081
        tovej 9mo ago
        If it's an LLM it's a (large) language model. If you use ideas from LLM architecture in other non-language models, they are not language models. But it is extremely silly to say that "large language models are language models" is a bad faith argument.
aurareturn 9mo ago
It cannot, however, synthesize new facts by combining information from this corpus. Are we sure? Why can't the LLM use tools, run experiments, and create new facts like humans?
- tovej 9mo ago
  Then the LLM is not actually modelling the world, but using other tools that do. The LLM is not the main component in such a system.
  - aurareturn 9mo ago
    So do we expect real world models to just regurgitate new facts from their training data?
    - tovej 9mo ago
      Regurgitating facts kind of assumes it is a language model, as you're assuming a language interface. I would assume a real "world model" or digital twin to be able to reliably model relationships between phenomena in whatever context is being modeled. Validation would probably require experts in whatever thing is being modeled to confirm that the model captures phenomena to some standard of fidelity. Not sure if that's regurgitating facts to you -- it isn't to me. But I don't know what you're asking exactly. Maybe you could specify what it is you mean by "real world model" and what you take fact-regurgitating to mean.
      - aurareturn 9mo ago
        But I don't know what you're asking exactly. Maybe you could specify what it is you mean by "real world model" and what you take fact-regurgitating to mean. You said this: If this existing corpus includes useful information it can regurgitate that.It cannot, however, synthesize new facts by combining information from this corpus. So I'm wondering if you think world models can synthesize new facts.
wongarsu 9mo ago
If we focus on base models and ignore the tuning steps after that, then LLMs are "just" a token predictor. But we know that pure statistical models aren't very good at this. After all we tried for decades to get Markov chains to generate text, and it always became a mess after a couple of words. If you tried to come up with the best way to actually predict the next token, a world model seems like an incredibly strong component. If you know what the sentence so far means, and how it relates to the world, human perception of the world and human knowledge, that makes guessing the next word/token much more reliable than just looking at statistical distributions. The bet OpenAI has made is that if this is the optimal final form, then given enough data and training, gradient descent will eventually build it. And I don't think that's entirely unreasonable, even if we haven't quite reached that point yet. The issues are more in how language is an imperfect description of the world. LLMs seems to be able to navigate the mistakes, contradictions and propaganda with some success, but fail at things like spatial awareness. That's why OpenAI is pushing image models and 3d world models, despite making very little money from them: they are working towards LLMs with more complete world models unchained by language I'm not sure if they are on the right track, but from a theoretical point I don't see an inherent fault
- tovej 9mo ago
  There's plenty of faults in this idea. First, the subjectivity of language. 1) People only speak or write down information that needs to be added to a base "world model" that a listener or receiver already has. This context is extremely important to any form of communication and is entirely missing when you train a pure language model. The subjective experience required to parse the text is missing. 2) When people produce text, there is always a motive to do so which influences the contents of the text. This subjective information component of producing the text is interpreted no different from any "world model" information. A world model should be as objective as possible. Using language, the most subjective form of information is a bad fit. The other issue in this argument is that you're inverting the implication. You say an accurate world model will produce the best word model, but then suddenly this is used to imply that any good word model is a useful world model. This does not compute.
  - 9mo ago
EMM_386 9mo ago
> It cannot, however, synthesize new facts by combining information from this corpus. That would be like saying studying mathematics can't lead to someone discovering new things in mathematics. Nothing would ever be "novel" if studying the existing knowledge could not lead to novel solutions. GPT 5.2 Thinking is solving Erdős Problems that had no prior solution - with a proof.
- tovej 9mo ago
  The Erdos problem was solved by interacting with a formal proof tool, and the problem was trivial. I also don't recall if this was the problem someone had already solved prior but not reported, but that does not matter. The point is that the LLM did not model maths to do this, made calls to a formal proof tool that did model maths, and was essentially working as the step function to a search algorithm, iterating until it found the zero in the function. That's clever use of the LLM as a component in a search algorithm, but the secret sauce here is not the LLM but the middleware that operated both the LLM and the formal proof tool. That middleware was the search tool that a human used to find the solution. This is not the same as a synthesis of information from the corpus of text.

by tovej 9mo ago

aixpert 9mo ago

they do model the world. Watch Noble price winner Hinton or let's admit that this is more of a religious question then the technical.

D-Machine 9mo ago
They model the part of the world that (linguistic models of the world posted on the internet) try to model. But what is posted on the internet is not IRL. So, to be glib: LLMs trained on the internet do not model IRL, they model talking about IRL.
imtringued 9mo ago
His point is that human language and the written record is a model of the world, so if you train an LLM you're training a model of a model of the world. That sounds highly technical if you ask me. People complain if you recompress music or images with lossy codecs, but when an LLM does that suddenly it's religious?
- hackinthebochs 9mo ago
  A model of a model of X is a model of X, albeit extra lossy.
  - tovej 9mo ago
    In this case this is not so. The primary model is not a model at all, and the surrogate has bias added to it. It's also missing any way to actually check the internal consistency of statements or otherwise combine information from its corpus, so it fails as a world model.
  - D-Machine 9mo ago
    An LLM has an internal linguistic model (i.e. it knows token patterns), and that linguistic model models humans' linguistic models (a stream of tokens) of their actual world models (which involve far, far more than linguistics and tokens, such as logical relations beyond mere semantic relations, sensory representations like imagery and sounds, and, yes, words and concepts). So LLMs are linguistic (token pattern) models of linguistic models (streams of tokens) describing world models (more than tokens). It thus does not in fact follow that LLMs model the world (as they are missing everything that is not encoded in non-linguistic semantics).
    - hackinthebochs 9mo ago
      At this point, anyone claiming that LLMs are "just" language models aren't arguing in good faith. LLMs are a general purpose computing paradigm. LLMs are circuit builders, the converged parameters define pathways through the architecture that pick out specific programs. Or as Karpathy puts it, LLMs are a differentiable computer[1]. Training LLMs discovers programs that well reproduce the input sequence. Tokens can represent anything, not just words. Roughly the same architecture can generate passable images, music, or even video. [1] https://x.com/karpathy/status/1582807367988654081 https://x.com/karpathy/status/1582807367988654081
      - tovej 9mo ago
        If it's an LLM it's a (large) language model. If you use ideas from LLM architecture in other non-language models, they are not language models. But it is extremely silly to say that "large language models are language models" is a bad faith argument.

aurareturn 9mo ago

It cannot, however, synthesize new facts by combining information from this corpus. Are we sure? Why can't the LLM use tools, run experiments, and create new facts like humans?

tovej 9mo ago
Then the LLM is not actually modelling the world, but using other tools that do. The LLM is not the main component in such a system.
- aurareturn 9mo ago
  So do we expect real world models to just regurgitate new facts from their training data?
  - tovej 9mo ago
    Regurgitating facts kind of assumes it is a language model, as you're assuming a language interface. I would assume a real "world model" or digital twin to be able to reliably model relationships between phenomena in whatever context is being modeled. Validation would probably require experts in whatever thing is being modeled to confirm that the model captures phenomena to some standard of fidelity. Not sure if that's regurgitating facts to you -- it isn't to me. But I don't know what you're asking exactly. Maybe you could specify what it is you mean by "real world model" and what you take fact-regurgitating to mean.
    - aurareturn 9mo ago
      But I don't know what you're asking exactly. Maybe you could specify what it is you mean by "real world model" and what you take fact-regurgitating to mean. You said this: If this existing corpus includes useful information it can regurgitate that.It cannot, however, synthesize new facts by combining information from this corpus. So I'm wondering if you think world models can synthesize new facts.

wongarsu 9mo ago

If we focus on base models and ignore the tuning steps after that, then LLMs are "just" a token predictor. But we know that pure statistical models aren't very good at this. After all we tried for decades to get Markov chains to generate text, and it always became a mess after a couple of words. If you tried to come up with the best way to actually predict the next token, a world model seems like an incredibly strong component. If you know what the sentence so far means, and how it relates to the world, human perception of the world and human knowledge, that makes guessing the next word/token much more reliable than just looking at statistical distributions. The bet OpenAI has made is that if this is the optimal final form, then given enough data and training, gradient descent will eventually build it. And I don't think that's entirely unreasonable, even if we haven't quite reached that point yet. The issues are more in how language is an imperfect description of the world. LLMs seems to be able to navigate the mistakes, contradictions and propaganda with some success, but fail at things like spatial awareness. That's why OpenAI is pushing image models and 3d world models, despite making very little money from them: they are working towards LLMs with more complete world models unchained by language I'm not sure if they are on the right track, but from a theoretical point I don't see an inherent fault

tovej 9mo ago
There's plenty of faults in this idea. First, the subjectivity of language. 1) People only speak or write down information that needs to be added to a base "world model" that a listener or receiver already has. This context is extremely important to any form of communication and is entirely missing when you train a pure language model. The subjective experience required to parse the text is missing. 2) When people produce text, there is always a motive to do so which influences the contents of the text. This subjective information component of producing the text is interpreted no different from any "world model" information. A world model should be as objective as possible. Using language, the most subjective form of information is a bad fit. The other issue in this argument is that you're inverting the implication. You say an accurate world model will produce the best word model, but then suddenly this is used to imply that any good word model is a useful world model. This does not compute.
- 9mo ago

EMM_386 9mo ago

> It cannot, however, synthesize new facts by combining information from this corpus. That would be like saying studying mathematics can't lead to someone discovering new things in mathematics. Nothing would ever be "novel" if studying the existing knowledge could not lead to novel solutions. GPT 5.2 Thinking is solving Erdős Problems that had no prior solution - with a proof.

tovej 9mo ago
The Erdos problem was solved by interacting with a formal proof tool, and the problem was trivial. I also don't recall if this was the problem someone had already solved prior but not reported, but that does not matter. The point is that the LLM did not model maths to do this, made calls to a formal proof tool that did model maths, and was essentially working as the step function to a search algorithm, iterating until it found the zero in the function. That's clever use of the LLM as a component in a search algorithm, but the secret sauce here is not the LLM but the middleware that operated both the LLM and the formal proof tool. That middleware was the search tool that a human used to find the solution. This is not the same as a synthesis of information from the corpus of text.