5 ms·
There is some evidence from Anthropic that LLMs do model the world. This paper[0] tracing their "thought" is fascinating. Basically an LLM translating across la
by tyre 1y ago
There is some evidence from Anthropic that LLMs do model the world. This paper[0] tracing their "thought" is fascinating. Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages.
It does have clusters of parameters that correlate with concepts, not just randomly "after X word tends to have Y word." Otherwise you would expect all of Chinese to be grouped in one place, all of French in another, all of English in another. This is empirically not the case.
I don't know whether to understand knowledge you have to have a model of the world, but at least as far as language, LLMs very much do seem to have modeling.
[0]: https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language...
- manmal 1y ago> Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages I thought that’s the basic premise of how transformers work - they encode concepts into high dimensional space, and similar concepts will be clustered together. I don’t think it models the world, but just the texts it ingested. It’s observation and regurgitation, not understanding. I do use agents a lot (soon on my second codex subscription), so I don’t think that’s a bad thing. But I’m firmly in the “they are useful tools” camp.
- bryanlarsen 1y agoThat's a model. Not a higher-order model like most humans use, but it's still a model.
- manmal 1y agoYes, not of the world, but of the ingested text. Almost verbatim what I wrote.
- timschmidt 1y agoThe ingested text itself contains a model of the world which we have encoded in it. That's what language is. Therefore by the transitive property...
- manmal 1y agoThat‘s quite a big leap, and sounds like a philosophical question. But many philosophers like late Wittgenstein or Heidegger disagreed with this idea. On more practical terms, maybe you‘ve experienced the following: You read a manual of a device on how to do something with it; but only actually using it for a few times gives you the intuition on how to use it _well_. Text is just very lossy, because not every aspect of the world, and factors in your personal use, are described. Many people rather watch YouTube videos for eg repairs. But those are very lossy as well - they don’t cover the edge cases usually. And there is often just no video on the repair you need to do. BTW, have you ever tried ChatGPT for advice on home improvement? It sucks _hard_ sometimes, hallucinating advice that doesn’t make any sense. And making up tools that don’t exist. There‘s no real commonsense to be had from it. Because it’s all just pieces of text that fight with each other for being the next token. When using Claude Code or codex to write Swift code, I need to be very careful to provide all the APIs that are relevant in context (or let it web search), or garbage will be the result. There is no real understanding of how Swift („the world“) works.
- timschmidt 1y agoNone of your examples refute the direct evidence of internal world model building which has been demonstrated (for example: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html https://adamkarvonen.github.io/machine_learning/2024/01/03/c... ). Instead you have retreated to qualia like "well" and "sucks hard". > hallucinating Literally every human memory. They may seem tangible to you, but they're all in your head. The result of neurons behaving in ways which have directly inspired ML algorithms for nearly a century. Further, history is rife with examples of humans learning from books and other written words. And also of humans thinking themselves special and unique in ways we are not. > When using Claude Code or codex to write Swift code, I need to be very careful to provide all the APIs that are relevant in context (or let it web search), or garbage will be the result. Yep. And humans often need to reference the documentation to get details right as well.
- sleepyams 1y agoWhat does "higher-order" mean?
- dgfitz 1y agoI believe that the M in LLM stands for model. It is a statistical model, as it always has been.
- SR2Z 1y agoRight, but modeling the structure of language is a question of modeling word order and binding affinities. It's the Chinese Room thought experiment - can you get away with a form of "understanding" which is fundamentally incomplete but still produces reasonable outputs? Language in itself attempts to model the world and the processes by which it changes. Knowing which parts-of-speech about sunrises appear together and where is not the same as understanding a sunrise - but you could make a very good case, for example, that understanding the same thing in poetry gets an LLM much closer.
- hackinthebochs 1y agoLLMs aren't just modeling word co-occurrences. They are recovering the underlying structure that generates word sequences. In other words, they are modeling the world. This model is quite low fidelity, but it should be very clear that they go beyond language modeling. We all know of the pelican riding a bicycle test [1]. Here's another example of how various language models view the world [2]. At this point it's just bad faith to claim LLMs aren't modeling the world. [1] https://simonwillison.net/2025/Aug/7/gpt-5/#and-some-svgs-of-pelicans https://simonwillison.net/2025/Aug/7/gpt-5/#and-some-svgs-of... [2] https://www.lesswrong.com/posts/xwdRzJxyqFqgXTWbH/how-does-a-blind-model-see-the-earth https://www.lesswrong.com/posts/xwdRzJxyqFqgXTWbH/how-does-a...
- homarp 1y agoand we can say that a bastardized version of the Sapir-Worf hypothesis applies: what's in the training set shapes or limits LLM's view of the world
- moron4hire 1y agoNeither Sapir nor Whorf presented Linguistic Relativism as their own hypothesis and they never published together. The concept, if it exists at all, is a very weak effect, considering it doesn't reliably replicate.
- 1y ago
- overfeed 1y ago> Basically an LLM translating across languages will "light up" for the same concepts across languages Which is exactly what they are trained to do. Translation models wouldn't be functional if they are unable to correlate an input to specific outputs. That some hiddel-layer neurons fire for the same concept shouldn't come as a surprise, and is a basic feature required for the core functionality.
- balder1991 1y agoAnd if it is true that the language is just the last step after the answer is already conceptualized, why do models perform differently in different languages? If it was just a matter of language, they’d have the same answer but just with a broken grammar, no?
- kaibee 1y agoIf you suddenly had to do all your mental math in base-7, do you think you'd be just as fast and accurate as you are at math in base-10? Is that because you don't have an internal world-model of mathematics? or is it because language and world-model are dependently linked?
- jhanschoo 1y agoLet's make this more concrete than talking about "understanding knowledge". Oftentimes I want to know something that cannot feasibly be arrived at by reasoning, only empirically. Remaining within the language domain, LLMs get so much more useful when they can search the web for news, or your codebase to know how it is organized. Similarly, you need a robot that can interact with the world and reason from newly collected empirical data in order to answer these empirical questions, if the work had not already been done previously.
- awesome_dude 1y agoI know the attributes of an Apple, i know the attributes of a Pear. As does a computer. But only i can bite into one and know without any doubt what it is and how it feels emotionally.
- zaphirplane 1y agoWe segued to conscience and individuality.
- scrubs 1y agoYou have half a point. "Without any doubt" is merely the apex of a huge undefined iceberg. I write half .. eating is multi modal and consequential. The llm can read the menu, but it didn't eat the meal. Even humans are bounded. Feeling, licking, smelling, or eating the menu still is not eating the meal. There is an insuperable gap in the analogy ... a gap in the concept and of sensory data doing it. Back to first point: what one knows through that sensory data ... is not clear at present or even possible with llms.
- awesome_dude 1y agoI think more, also, how i feel about the taste.
- skydhash 1y ago> LLMs get so much more useful when they can search the web for news, or your codebase to know how it is organized But their usefulness is only surface-deep. The news that matters to you is always deeply contextual, it's not only things labelled as breaking news or happening near you. Same thing happens with code organization. The reason is more human nature (how we think and learn) than machine optimization (the compiler usually don't care).
- bravura 1y agoHow large is a lion? Learning the size of objects using pure text analysis requires significant gymnastics. Vision demonstrates physical size more easily. Multimodal learning is important. Full stop. Purely textual learning is not sample efficient for world modeling and the optimization can get stuck in local optima that are easily escaped through multimodal evidence. ("How large are lions? inducing distributions over quantitative attributes", Elazar et al 2019)
- latentsea 1y ago> How large is a lion? Twice of half of its size.
- johnisgood 1y agoCan you be more specific about "size" here? (Do not tell me the definition of size though). You are not wrong though, just very incomplete. Your response is a food for thought, IMO.
- EMM_386 1y ago> How large is a lion? Ask a blind person that question - they can answer it. Too many people think you need to "see" as in human sight to understand things like this. You obviously don't. The massive training data these models ingest is more than sufficient to answer this question - and not just by looking up "dimensions of a lion" in the high-dimensional space. The patterns in that space are what generates the concept of what a lion is. You don't need to physically see a lion to know those things.
- vlovich123 1y agoIf it was modeling the world you’d expect “give me a picture of a glass filled to the brim” to actually do that. It’s inability to correctly and accurately combine concepts indicates it’s probably not building a model of the real world.
- p1esk 1y agoI just gave chatgpt this prompt - it produced a picture of a glass filled to the brim with water.
- jdiff 1y agoLike most quirks that spread widely, a bandaid is swiftly applied. This is also why they now know how many r's are in "strawberry." But we don't get any closer to useful general intelligence by cobbling together thousands of hasty patches.
- llbbdd 1y agoSeems to have worked fine for humans so far.
- bigstrat2003 1y agoNo, humans are not a series of band-aid patches where we learn facts in isolation. A human can reason, and when exposed to novel situations figure out a path forward. You don't need to tell a human how many rs are in "strawberry"; as long as they know what the letter r is they can count it in any word you choose to give them. As proven time and time again, LLMs can't do this. The embarrassing failure of Claude to figure out how to play Pokemon a year or so ago is a good example. You could hand a five year old human a Gameboy with Pokemon in it, and he could figure out how to move around and do the basics. He wouldn't be very good, but he would figure it out as he goes. Claude couldn't figure out to stop going in and out of a building. LLMs, usefulness aside, have repeatedly shown themselves to have zero intelligence.
- _fizz_buzz_ 1y ago> Basically an LLM translating across languages will "light up" (to use a rough fMRI equivalent) for the same concepts (e.g. bigness) across languages. That doesn't seem surprising at all. My understanding is that transformers where invented exactly for the application of translations. So, concepts must be grouped together in different languages. That was originally the whole point and then turned out to be very useful for broader AI applications.
- Hendrikto 1y agoThat is just how embeddings work. It does not confirm nor deny whether LLMs have a world model.