8 ms·

LLMs aren't world models

371 points by ingve 1y ago

t0md4n 1y ago
https://arxiv.org/abs/2501.17186 https://arxiv.org/abs/2501.17186
- yosefk 1y ago
  This is interesting. The "professional level" rating of <1800 isn't, but still. However: "A significant Elo rating jump occurs when the model’s Legal Move accuracy reaches 99.8%. This increase is due to the reduction in errors after the model learns to generate legal moves, reinforcing that continuous error correction and learning the correct moves significantly improve ELO" You should be able to reach the move legality of around 100% with few resources spent on it. Failing to do so means that it has not learned a model of what chess is, at some basic level. There is virtually no challenge in making legal moves.
  - lostmsu 1y ago
    > r4rk1 pp6 8 4p2Q 3n4 4N3 qP5P 2KRB3 w — — 3 27 Can you say 100% you can generate a good next move (example from the paper) without using tools, and will never accidentally make a mistake and give an illegal move?
  - rpdillon 1y ago
    > Failing to do so means that it has not learned a model of what chess is, at some basic level. I'm not sure about this. Among a standard amateur set of chess players, how often when they lack any kind of guidance from a computer do they attempt to make a move that is illegal? I played chess for years throughout elementary, middle and high school, and I would easily say that even after hundreds of hours of playing, I might make two mistakes out of a thousand moves where the move was actually illegal, often because I had missed that moving that piece would continue to leave me in check due to a discovered check that I had missed. It's hard to conclude from that experience that players that are amateurs lack even a basic model of chess.
libraryofbabel 1y ago
This essay could probably benefit from some engagement with the literature on “interpretability” in LLMs, including the empirical results about how knowledge (like addition) is represented inside the neural network. To be blunt, I’m not sure being smart and reasoning from first principles after asking the LLM a lot of questions and cherry picking what it gets wrong gets to any novel insights at this point. And it already feels a little out date, with LLMs getting gold on the mathematical Olympiad they clearly have a pretty good world model of mathematics. I don’t think cherry-picking a failure to prove 2 + 2 = 4 in the particular specific way the writer wanted to see disproves that at all. LLMs have imperfect world models, sure. (So do humans.) That’s because they are trained to be generalists and because their internal representations of things are massively compressed single they don’t have enough weights to encode everything. I don’t think this means there are some natural limits to what they can do.
- yosefk 1y ago
  Your being blunt is actually very kind, if you're describing what I'm doing as "being smart and reasoning from first principles"; and I agree that I am not saying something very novel, at most it's slightly contrarian given the current sentiment. My goal is not to cherry-pick failures for its own sake as much as to try to explain why I get pretty bad output from LLMs much of the time, which I do. They are also very useful to me at times. Let's see how my predictions hold up; I have made enough to look very wrong if they don't. Regarding "failure disproving success": it can't, but it can disprove a theory of how this success is achieved. And, I have much better examples than the 2+2=4, which I am citing as something that sorta works these says
  - libraryofbabel 1y ago
    I mean yeah, it’s a good essay in that it made me think and try to articulate the gaps, and I’m always looking to read things that push back on AI hype. I usually just skip over the hype blogging. I think my biggest complaint is that the essay points out flaws in LLM’s world models (totally valid, they do confidently get things wrong and hallucinate in ways that are different, and often more frustrating, from how humans get things wrong) but then it jumps to claiming that there is some fundamental limitation about LLMs that prevents them from forming workable world models. In particular, it strays a bit towards the “they’re just stochastic parrots” critique, e.g. “that just shows the LLM knows to put the words explaining it after the words asking the question.” That just doesn’t seem to hold up in the face of e.g. LLMs getting gold on the Mathematical Olympiad, which features novel questions. If that isn’t a world model of mathematics - being able to apply learned techniques to challenging new questions - then I don’t know what is. A lot of that success is from reinforcement learning techniques where the LLM is made to solve tons of math problems after the pre-training “read everything” step, which then gives it a chance to update its weights. LLMs aren’t just trained from reading a lot of text anymore. It’s very similar to how the alpha zero chess engine was trained, in fact. I do think there’s a lot that the essay gets right. If I was to recast it, I’d put it something like this: * LLMs have imperfect models of the world which is conditioned by how they’re trained on next token prediction. * We’ve shown we can drastically improve those world models for particular tasks by reinforcement learning. you kind of allude to this already by talking about how they’ve been “flogged” to be good at math. * I would claim that there’s no particular reason these RL techniques aren’t extensible in principle to beat all sorts of benchmarks that might look unrealistic now. (Two years ago it would have been an extreme optimist position to say an LLM could get gold on the mathematical Olympiad, and most LLM skeptics would probably have said it could never happen.) * Of course it’s very expensive, so most world models LLMs have won’t get the RL treatment and so will be full of gaps, especially for things that aren’t amenable to RL. It’s good to beware of this. I think the biggest limitation LLMs actually have, the one that is the biggest barrier to AGI, is that they can’t learn on the job, during inference. This means that with a novel codebase they are never able to build a good model of it, because they can never update their weights. (If an LLM was given tons of RL training on that codebase, it could build a better world model, but that’s expensive and very challenging to set up.) This problem is hinted at in your essay, but the lack of on-the-job learning isn’t centered. But it’s the real elephant in the room with LLMs and the one the boosters don’t really have an answer to. Anyway thanks for writing this and responding!
GaggiX 1y ago
https://www.youtube.com/watch?v=LtG0ACIbmHw https://www.youtube.com/watch?v=LtG0ACIbmHw Sota LLMs do play legal moves in chess, I don't why the article seem to say otherwise.
- tickettotranai 1y ago
  Technically yes, but... it's moderately tricky to get an LLM to play good chess even though it can. https://dynomight.net/more-chess/ https://dynomight.net/more-chess/ This is significant in general because I personally would love to get these things to code-switch into "hackernews poster" or "writer for the Economist" or "academic philosopher", but I think the "chat" format makes it impossible. The inaccessibility of this makes me want to host my own LLM...
rishi_devan 1y ago
Haha. I enjoyed that Soviet-era joke at the end.
- svantana 1y ago
  Yes, I hadn't heard that before. It's similar in spirit to this norwegian folk tale about a deaf man guessing what someone is saying to him: https://en.wikipedia.org/wiki/%22Good_day,_fellow!%22_%22Axe_handle!%22 https://en.wikipedia.org/wiki/%22Good_day,_fellow!%22_%22Axe...
  - kgwgk 1y ago
    Another similar story: King Frederick, the great of Prussia had a very fine army, and none of the soldiers in it were finer than Giant Guards, who were all extremely tall men. It was difficult to find enough soldiers for these Guards, as there were not many men who were tall enough. Frederick had made it a rule that no soldiers who did not speak German could be admitted to the Giant Guards, and this made the work of the officers who had to find men for them even more difficult. When they had to choose between accepting or refusing a really tall man who knew no German, the officers used to accept him, and then teach him enough. German to be able to answer if the King questioned him. Frederick, sometimes, used to visit the men who were on guard around his castle at night to see that they were doing their job properly, and it was his habit to ask each new one that he saw three questions: “How old are you?” “How long have you been in my army?” and “Are you satisfied with your food and your conditions?” The offices of the Giant Guards therefore used to teach new soldiers who did not know German the answers to these three questions. One day, however, the King asked a new soldier the questions in a different order, he began with, “How long have you been in my army?” The young soldier immediately answered, “Twenty – two years, Your Majesty”. Frederick was very surprised. “How old are you then?”, he asked the soldier. “Six months, Your Majesty”, came the answer. At this Frederick became angry, “Am I a fool, or are you one?” he asked. “Both, Your Majesty”, the soldier answered politely. https://archive.org/details/advancedstoriesf0000hill https://archive.org/details/advancedstoriesf0000hill
deadbabe 1y ago
Don’t: use LLMs to play chess against you Do: use LLMs to talk shit to you while a real chess AI plays chess against you. The above applies to a lot of things besides chess, and illustrates a proper application of LLMs.
- Seb-C 1y ago
  Are you suggesting that we use an LLM as an interface between the AI and the player? Why would anyone choose to awkwardly play using natural language rather than a reliable, fast and intuitive UI?
  - crabmusket 1y ago
    No, I think they're suggesting the LLM should literally be "talking shit", e.g. in a chat window alongside the game UI, as if you're in a live chat with another player. As in, use the LLM for processing language, and the chess engine for playing chess. I think this is quite an amusing idea, as the LLM would see the moves the chess engine made and comment along the lines of "wow, I didn't see that one coming!" very Roger Sperry.
imenani 1y ago
As far as I can tell they don’t say which LLM they used which is kind of a shame as there is a huge range of capabilities even in newly released LLMs (e.g. reasoning vs not).
- yosefk 1y ago
  ChatGPT, Claude, Grok and Google AI Overviews, whatever powers the latter, were all used in one or more of these examples, in various configurations. I think they can perform differently, and I often try more than one when the 1st try doesn't work great. I don't think there's any fundamental difference in the principle of their operation, and I think there never will be - there will be another major breakthrough
  - red75prime 1y ago
    My hypothesis is that a model fails to switch into a deep thinking mode (if it has it) and blurts whatever it got from all the internet data during autoregressive training. I tested it with alpha-blending example. Gemini 2.5 flash - fails, Gemini 2.5 pro - succeeds. How presence/absence of a world model, er, blends into all this? I guess "having a consistent world model at all times" is an incorrect description of humans, too. We seem to have it because we have mechanisms to notice errors, correct errors, remember the results, and use the results when similar situations arise, while slowly updating intuitions about the world to incorporate changes. The current models lack "remember/use/update" parts.
  - imenani 1y ago
    Each of these models has a thinking/reasoning variant and a default non-thinking variant. I would expect the reasoning variants (o3 or “GPT5 Thinking”, Gemini DeepThink, Claude with Extended Thinking, etc) to do better at this. I think there is also some chance that in their reasoning traces they may display something you might see as closer to world modelling. In particular, you might find them explicitly tracking positions of pieces and checking validity.
  - red75prime 1y ago
    > I don't think there's any fundamental difference in the principle of their operation Yeah, they seem to be a subject to the universal approximation theorem (it needs to be checked more thoroughly, but I think we can build a transformer that is equivalent to any given fully-connected multilayered network). That is at a certain size they can do anything a human can do at a certain point in their life (that is with no additional training) regardless of whether humans have world models and what those model are on the neuronal level. But there are additional nuances that are related to their architectures and training regimes. And practical questions of the required size.
famouswaffles 1y ago
Yes LLMs can play chess and yes they can model it fine https://arxiv.org/pdf/2403.15498v2 https://arxiv.org/pdf/2403.15498v2
lordnacho 1y ago
Here's what LLMs remind me of. When I went to uni, we had tutorials several times a week. Two students, one professor, going over whatever was being studied that week. The professor would ask insightful questions, and the students would try to answer. Sometimes, I would answer a question correctly without actually understanding what I was saying. I would be spewing out something that I had read somewhere in the huge pile of books, and it would be a sentence, with certain special words in it, that the professor would accept as an answer. But I would sometimes have this weird feeling of "hmm I actually don't get it" regardless. This is kinda what the tutorial is for, though. With a bit more prodding, the prof will ask something that you genuinely cannot produce a suitable word salad for, and you would be found out. In math-type tutorials it would be things like realizing some equation was useful for finding an answer without having a clue about what the equation actually represented. In economics tutorials it would be spewing out words about inflation or growth or some particular author but then having nothing to back up the intuition. This is what I suspect LLMs do. They can often be very useful to someone who actually has the models in their minds, but not the data to hand. You may have forgotten the supporting evidence for some position, or you might have missed some piece of the argument due to imperfect memory. In these cases, LLM is fantastic as it just glues together plausible related words for you to examine. The wheels come off when you're not an expert. Everything it says will sound plausible. When you challenge it, it just apologizes and pretends to correct itself.
- roywiggins 1y ago
  > When you challenge it, it just apologizes and pretends to correct itself. Even when it was right the first time!
- nhaehnle 1y ago
  Good on you for having the meta-cognition to recognize it. I've graded many exams in my university days (and set some myself), and it's exceedingly obvious that that's what many students are doing. I do wonder though how often they manage to fly under the radar. I'm sure it happens, as you described. (This is also the reason why I strongly believe that in exams where students write free-form answers, points should be subtracted for incorrect statements even if a correct solution is somewhere in the word salad.)
ej88 1y ago
This article is interesting but pretty shallow. 0(?): there’s no provided definition of what a ‘world model’ is. Is it playing chess? Is it remembering facts like how computers use math to blend Colors? If so, then ChatGPT: https://chatgpt.com/s/t_6898fe6178b88191a138fba8824c1a2c https://chatgpt.com/s/t_6898fe6178b88191a138fba8824c1a2c has a world model right? 1. The author seems to conflate context windows with failing to model the world in the chess example. I challenge them to ask a SOTA model with an image of a chess board or notation and ask it about the position. It might not give you GM level analysis but it definitely has a model of what’s going on. 2. Without explaining which LLM they used or sharing the chats these examples are just not valuable. The larger and better the model, the better its internal representation of the world. You can try it yourself. Come up with some question involving interacting with the world and / or physics and ask GPT-5 Thinking. It’s got a pretty good understanding of how things work! https://chatgpt.com/s/t_689903b03e6c8191b7ce1b85b1698358 https://chatgpt.com/s/t_689903b03e6c8191b7ce1b85b1698358
- yosefk 1y ago
  A "world model" depends on the context which defines which world the problem is in. For chess, which moves are legal and needing to know where the pieces are to make legal moves are parts of the world model. For alpha blending, it being a mathematical operation and the visibility of a background given the transparency of the foreground are parts of the world model. The examples are from all the major commercial American LLMs as listed in a sister comment. You seem to conflate context windows with tracking chess pieces. The context windows are more than large enough to remember 10 moves. The model should either track the pieces, or mention that it would be playing blindfold chess absent a board to look at and it isn't good at this, so could you please list the position after every move to make it fair, or it doesn't know what it's doing; it's demonstrably the latter.
  - tmnvdb 1y ago
    If you train an LLM on chess, it will learn that too. You don't need to explain the rules, just feed it chess games, and it will stop making illegal moves at some point. It is a clear example of an inferred world model from training. https://arxiv.org/abs/2501.17186 https://arxiv.org/abs/2501.17186 PS "Major commercial American LLM" is not very meaningful, you could be using GPT4o with that description.
jonplackett 1y ago
I just tried a few things that are simple and a world model would probably get right. Eg Question to GPT5: I am looking straight on to some objects. Looking parallel to the ground. In front of me I have a milk bottle, to the right of that is a Coca-Cola bottle. To the right of that is a glass of water. And to the right of that there’s a cherry. Behind the cherry there’s a cactus and to the left of that there’s a peanut. Everything is spaced evenly. Can I see the peanut? Answer (after choosing thinking mode) No. The cactus is directly behind the cherry (front row order: milk, Coke, water, cherry). “To the left of that” puts the peanut behind the glass of water. Since you’re looking straight on, the glass sits in front and occludes the peanut. It doesn’t consider transparency until you mention it, then apologises and says it didn’t think of transparency
- RugnirViking 1y ago
  this seems like a strange riddle. In my mind I was thinking that regardless of the glass, all of the objects can be seen (due to perspective, and also the fact you mentioned the locations, meaning you're aware of them). It seems to me it would only actually work in an orthographic perspective, which is not how our reality works
  - jonplackett 1y ago
    You can tell from the response it does understand the riddle just fine, it just gets it wrong.
    - rpdillon 1y ago
      Have you asked five adults this riddle? I suspect at least two of them would get it wrong or have some uncertainty about whether or not the peanut was visible.
      - xg15 1y ago
        This. Was also thinking "yes" first because of the glass of water, transparency, etc, but then got unsure: The objects might be spaced so widely that the milk or coke bottle would obscure the view due to perspective - or the peanut would simply end up outside the viewer's field of vision. Shows that even if you have a world model, it might not be the right one.
Razengan 1y ago
A slight tangent: I think/wonder if the one place where AIs could be really useful, might be in translating alien languages :) As in, an alien could teach one of our AIs their language faster than an alien could teach an human, and vice versa.. ..though the potential for catastrophic disasters is also great there lol
keeda 1y ago
That whole bit about color blending and transparency and LLMs "not knowing colors" is hard to believe. I am literally using LLMs every day to write image-processing and computer vision code using OpenCV. It seamlessly reasons across a range of concepts like color spaces, resolution, compression artifacts, filtering, segmentation and human perception. I mean, removing the alpha from a PNG image was a preprocessing step it wrote by itself as part of a larger task I had given it, so it certainly understands transparency. I even often describe the results e.g. "this fails when in X manner when the image has grainy regions" and it figures out what is going on, and adapts the code accordingly. (It works with uploading actual images too, but those consume a lot of tokens!) And all this in a rather niche domain that seems relatively less explored. The images I'm working with are rather small and low-resolution, which most literature does not seem to contemplate much. It uses standard techniques well known in the art, but it adapts and combines them well to suit my particular requirements. So they seem to handle "novel" pretty well too. If it can reason about images and vision and write working code for niche problems I throw at it, whether it "knows" colors in the human sense is a purely philosophical question.
- geraneum 1y ago
  > it wrote by itself as part of a larger task I had given it, so it certainly understands transparency Or it’s a common step or a known pattern or combination of steps that is prevalent in its training data for certain input. I’m guessing you don’t know what’s exactly in the training sets. I don’t know either. They don’t tell ;) > but it adapts and combines them well to suit my particular requirements. So they seem to handle "novel" pretty well too. We tend to overestimate the novelty of our own work and our methods and at the same time, underestimate the vastness of the data and information available online for machines to train on. LLMs are very sophisticated pattern recognizers. It doesn’t mean what you are doing specifically is done in this exact way before, rather the patterns adapted and the approach may not be one of their kind. > is a purely philosophical question It is indeed. A question we need to ask ourselves.
  - Uehreka 1y ago
    > We tend to overestimate the novelty of our own work and our methods and at the same time, underestimate the vastness of the data and information available online for machines to train on. LLMs are very sophisticated pattern recognizers. If LLMs are stochastic parrots, but also we’re just stochastic parrots, then what does it matter? That would mean that LLMs are in fact useful for many things (which is what I care about far more than any abstract discussion of free will).
skeledrew 1y ago
Agree in general with most of the points, except > but because I know you and I get by with less. Actually we got far more data and training than any LLM. We've been gathering and processing sensory data every second at least since birth (more processing than gathering when asleep), and are only really considered fully intelligent in our late teens to mid-20s.
- helloplanets 1y ago
  Don't forget the millions of years of pre-training! ;)
o_nate 1y ago
What with this and your previous post about why sometimes incompetent management leads to better outcomes, you are quickly becoming one of my favorite tech bloggers. Perhaps I enjoyed the piece so much because your conclusions basically track mine. (I'm a software developer who has dabbled with LLMs, and has some hand-wavey background on how they work, but otherwise can claim no special knowledge.) Also your writing style really pops. No one would accuse your post of having been generated by an LLM.
- yosefk 1y ago
  thank you for your kind words!
neuroelectron 1y ago
Not yet
ameliaquining 1y ago
One thing I appreciated about this post, unlike a lot of AI-skeptic posts, is that it actually makes a concrete falsifiable prediction; specifically, "LLMs will never manage to deal with large code bases 'autonomously'". So in the future we can look back and see whether it was right. For my part, I'd give 80% confidence that LLMs will be able to do this within two years, without fundamental architectural changes.
- moduspol 1y ago
  "Deal with" and "autonomously" are doing a lot of heavy lifting there. Cursor already does a pretty good job indexing all the files in a code base in a way that lets it ask questions and get answers pretty quickly. It's just a matter of where you set the goalposts.
  - ameliaquining 1y ago
    True, there'd be a need to operationalize these things a bit more than is done in the post to have a good advance prediction.
  - jononor 1y ago
    "LLM" as well, because coding agents are already more than just an LLM. There is very useful context management around it, and tool calling, and ability to run tests/programs, etc. Though they are LLM-based systems, they are not LLMs.
    - smnrchrds 1y ago
      Indeed. If the LLM calls a chess engine tool behind the scenes, it would be able to play excellent chess as well.
      - cavisne 1y ago
        The author would still be wrong in the tool-calling scenario. There is already perfect (or at least superhuman) chess engines. There is no perfect "coding engine". LLM's + tools being able to reliably work on large codebases would be a new thing.
bithive123 1y ago
Language models aren't world models for the same reason languages aren't world models. Symbols, by definition, only represent a thing. They are not the same as the thing. The map is not the territory, the description is not the described, you can't get wet in the word "water". They only have meaning to sentient beings, and that meaning is heavily subjective and contextual. But there appear to be some who think that we can grasp truth through mechanical symbol manipulation. Perhaps we just need to add a few million more symbols, they think. If we accept the incompleteness theorem, then there are true propositions that even a super-intelligent AGI would not be able to express, because all it can do is output a series of placeholders. Not to mention the obvious fallacy of knowing super-intelligence when we see it. Can you write a test suite for it?
- habitue 1y ago
  > Symbols, by definition, only represent a thing. This is missing the lesson of the Yoneda Lemma: symbols are uniquely identified by their relationships with other symbols. If those relationships are represented in text, then in principle they can be inferred and navigated by an LLM. Some relationships are not represented well in text: tacit knowledge like how hard to twist a bottle cap to get it to come off, etc. We aren't capturing those relationships between all your individual muscles and your brain well in language, so an LLM will miss them or have very approximate versions of them, but... that's always been the problem with tacit knowledge: it's the exact kind of knowledge that's hard to communicate!
  - nomel 1y ago
    I don’t think it’s a communication problem as much as there is no possible relation between a word and a (literal) physical experiences. They’re, quite literally, on different planes of existence.
    - drdeca 1y ago
      When I have a physical experience, sometimes it results in me saying a word. Now, maybe there are other possible experiences that would result in me behaving identically, such that from my behavior (including what words I say) it is impossible to distinguish between different potential experiences I could have had. But, “caused me to say” is a relation, is it not? Unless you want to say that it wasn’t the experience that caused me to do something, but some physical thing that went along with the experience, either causing or co-occurring with the experience, and also causing me to say the word I said. But, that would still be a relation, I think.
frankfrank13 1y ago
Great quote at the end that I think I resonate a lot with: > Feeding these algorithms gobs of data is another example of how an approach that must be fundamentally incorrect at least in some sense, as evidenced by how data-hungry it is, can be taken very far by engineering efforts — as long as something is useful enough to fund such efforts and isn’t outcompeted by a new idea, it can persist.
1970-01-01 1y ago
I'm surprised the models haven't been enshittified by capitalism. I think in a few years we're going to see lightning-fast LLMs generating better output compared to what we're seeing today. But it won't be 1000x better, it will be 10x better, 10x faster, and completely enshittified with ads and clickbait links. Enjoy ChatGPT while it lasts.
UltraSane 1y ago
I wonder how the nature of the language used to train an LLM affects its model of the world. Would a language designed for the maximum possible information content and clarity like Ithkuil make an LLMs world model more accurate?
DennisP 1y ago
Maybe pure language models aren't world models, but Genie 3 for example seems to be a pretty good world model: https://deepmind.google/discover/blog/genie-3-a-new-frontier-for-world-models/ https://deepmind.google/discover/blog/genie-3-a-new-frontier... We also have multimodal AIs that can do both language and video. Genie 3 made multimodal with language might be pretty impressive. Focusing only on what pure language models can do is a bit of a straw man at this point.
ordu 1y ago
> LLMs are not by themselves sufficient as a path to general machine intelligence; in some sense they are a distraction because of how far you can take them despite the approach being fundamentally incorrect. I don't believe that it is a fundamentally incorrect approach. I believe, that human mind does something like that all the time, the difference is our minds have some additional processes that can, for example, filter hallucinations. Kids at a specific age range are afraid of their imagination. Their imagination can place a monster into any dark place where nothing can be seen. Adult mind can do the same easily, but the difference is kids have difficulties distinguishing imagination and perception, while adult generally manage. I believe, the ability of human mind to see difference between imagination/hallucinations from one hand and perception and memory from the other is not a fundamental thing stemming from the architecture of brains but a learned skill. Moreover people can be tricked to acquire false memory[1]. If LLM fell to tricks of Elizabet Loftus, we'd say LLM hallucinated. What LLMs need is to learn some tricks to detect hallucinations. Probably they will not get 100% reliable detector, but to get to the level of humans they don't need 100% reliability.
- shkkmo 1y ago
  > If LLM fell to tricks of Elizabet Loftus, we'd say LLM hallucinated. She's strongly oversold how and when false memories can be created. She testified in defense of Ghislaine Maxwell at her 2021 trial that financial incentives can create false memories and only later admitted that there were no studies to back this up when directly questioned. She's spent a career over-generalizing data about implanting false minor memories to make money discrediting victims' traumatic memories and defend abusers. You conflate "hallucination" with "imagination" but the former has much more in common with lieing than it does with imagining.
  - taneq 1y ago
    > She testified in defense of Ghislaine Maxwell at her 2021 trial that financial incentives can create false memories and only later admitted that there were no studies to back this up when directly questioned. Did she have financial incentives? Was this a live demonstration? :P
- 1y ago
globular-toast 1y ago
I agree with the article. I will be very surprised if LLMs end up being "it". I say this as a language geek who has always been amazed how language drives our thinking. However, I think language exists between brains, not inside them. There's something else in us and LLMs aren't it.
revskill 1y ago
LLM is not AI, it's dumbass, too stupid to NOT assume and hallucinate.
- mft_ 1y ago
  Lots of people assume, confabulate, misremember, and lie every day.
  - revskill 1y ago
    They are not intelligent.
    - mft_ 1y ago
      LOL, you really think that intelligence (however you want to define or measure the concept) is a guarantee that people won't make mistakes, misremember, make stuff up, or lie?
antirez 1y ago
The post is based on a misconception. If you read the blog post linked at the end of this message, you'll see how a very small GPT-2 alike transformer (Karpathy nano-gpt trained to a very small size) after seeing just PGN games and nothing more develops an 8x8 internal representation with which chess piece is where. This representation can be extracted by linear probing (and can be even altered by using the probe in reverse). LLMs are decent but not very good chess players for other reasons, not because they don't have a world model of the chess board. https://www.lesswrong.com/posts/yzGDwpRBx6TEcdeA5/a-chess-gpt-linear-emergent-world-representation https://www.lesswrong.com/posts/yzGDwpRBx6TEcdeA5/a-chess-gp...
- yosefk 1y ago
  The post or rather the part you refer to is based on a simple experiment which I encourage you to repeat. (It is way likelier to reproduce in the short to medium run than the others.) From your link: "...The first was gpt-3.5-turbo-instruct's ability to play chess at 1800 Elo" These things don't play at 1800 ELO, though maybe someone measured this ELO without cheating but rather relying on some artifacts of how an engine told to play at a low rating does against an LLM (engines are weird when you ask them to play badly, as a rule); a good start to a decent measurement would be to try it on chess 960. These things do lose track of the pieces in 10 moves. (As do I absent a board to look at, but I understand enough to say "I can't play blindfold chess, let's set things up so I can look at the current position somehow")
  - famouswaffles 1y ago
    >These things don't play at 1800 ELO Why are you saying 'these things'?. That statement is about a specific model which did play at that level and did not lose track of the pieces. There's no cheating or weirdness. https://github.com/adamkarvonen/chess_gpt_eval https://github.com/adamkarvonen/chess_gpt_eval
- thecupisblue 1y ago
  Ironically, that lesswrong article is more wrong than right. First, chess is perfect for such modeling. The game is basically a tree of legal moves. The "world model" representation is already encoded in the dataset itself and at a certain scale the chance of making an illegal move is minimal, as the dataset itself includes an insane amount of legal moves compared to illegal moves, let alone when you are training it on a chess dataset like PGN one Second, the probing is quite... a subjective thing. We are cherry-picking activations across an arbitrary amount of dimensions, on a model specifically trained for chess, taking these arbitrary representations and displaying it on 2D graph. Well yeah, with enough dimensions and cherry-picking, we can also show how "all zebras are elephants, because all elephants are horses and look their weights overlap in so many dimensions - large four-legged animals you see on safari!" - especially if we cherry-pick it. Especially if we tune a dataset on it. This shows nothing other than "training LLMs on a constrained move dataset makes LLM great at predicting next move in that dataset".
FailMore 1y ago
It seems clear to me that LLMs are a useful sort of dumb smart activity. They can take some pretty useful stabs in the dark, and therefore do much better in an environment which can give them feedback (coding) or where there is no objective correct answer (write a poem). It opens the door for some novel type of computational tasks, and the more feedback you can provide within the architecture of your application, the more useful the LLM will probably be. I think the hype of their genuine intelligence is overblown, but doesn’t mean they are not useful.
Peteragain 1y ago
This article appeared on HN a while ago. https://dynomight.net/more-chess/ https://dynomight.net/more-chess/ It basically is in agreement with this article and provides a few more trials and explanations.
devxpy 1y ago
I don’t think you even need such complex tests as chess to see it doesn’t have a world model - just ask it to do any 5+ digit multiplication
machiaweliczny 1y ago
I think they know this but don’t have causality built-in. In the sense they aren’t incentivised to understand holistically. Kids around 4 years old are spamming with “why? why? why?” questions and I think this is some process we need yet to reproduce. (BTW I suspect they ask this as manifestation of what is going in their brain and not real curiosity as they ask same question multiple times)
- machiaweliczny 1y ago
  I think it’s mostly because they are incentivised to answer verbatim as medicine students and not with their own understanding. RL methods change that.
glenstein 1y ago
This is the best and clearest explanation I have yet seen that describe a tricky thing, namely that LLMs, which are synonymous with "AI" for so many people, are just one variation of many possible types of machine intelligence. Which I find important because, well, hallucinating facts is what you would expect from an LLM, but isn't necessarily inherent issue with machine intelligence writ large if it's trained from the ground up on different principles, or modelling something else. We use LLMs as a stand in for tutors because being really good at language incidentally makes them able to explain math or history as a side effect. Importantly it doesn't show that hallucinating is a baked in problem for AI writ large. Presumably different models will have different kinds of systemic errors based on their respective designs.
m4nu3l 1y ago
Developing a model of the real world, or even just learning only a subset of self-consistent information, could be detrimental to the task of predicting the next token in the average text, given that most of the written information on many subjects could be contradictory and somehow wrong. I don't know how they are doing RL on top of that, how they are using synthetic data or filtering them. But it's clear that even with GPT-5 they haven't solved the problem, as the presentation demonstrated with the very first prompt (I'm talking about the wrong explanation for lift produced by a wing).
maebert 1y ago
It might be worth noting that humans also struggle with keeping up a coherent world model over time. Luckily, we don’t have to; we externalize a lot of our representations. When shopping together with a friend we might put our stuff on one side of the shopping cart and our friends’ on the other. There’s a reason we don’t just play chess in our heads but use a chess board. We use notebooks to write things down, etc. Some reasoning model can do similar things (keep a persistent notebook that gets fed back into the context window on every pass), but I expect that we need a few more dirty representational ist tricks to get there. In other words, I don’t think it’s an LLMs job to have a world model, but an LLM is just one part of an AI system.
3willows 1y ago
I've been looking for a way to play chess/go with a generalist LLM. It wouldn't matter if the moves are bad (I like winning), but being able to chat on unrelated topics while playing the game would take the experience to the next level.
quantumcotton 1y ago
I suppose teaching mine gsis and point clouds and connecting it to the world gsis map is a bad idea?

371 points by ingve 1y ago

t0md4n 1y ago

https://arxiv.org/abs/2501.17186 https://arxiv.org/abs/2501.17186

yosefk 1y ago
This is interesting. The "professional level" rating of <1800 isn't, but still. However: "A significant Elo rating jump occurs when the model’s Legal Move accuracy reaches 99.8%. This increase is due to the reduction in errors after the model learns to generate legal moves, reinforcing that continuous error correction and learning the correct moves significantly improve ELO" You should be able to reach the move legality of around 100% with few resources spent on it. Failing to do so means that it has not learned a model of what chess is, at some basic level. There is virtually no challenge in making legal moves.
- lostmsu 1y ago
  > r4rk1 pp6 8 4p2Q 3n4 4N3 qP5P 2KRB3 w — — 3 27 Can you say 100% you can generate a good next move (example from the paper) without using tools, and will never accidentally make a mistake and give an illegal move?
- rpdillon 1y ago
  > Failing to do so means that it has not learned a model of what chess is, at some basic level. I'm not sure about this. Among a standard amateur set of chess players, how often when they lack any kind of guidance from a computer do they attempt to make a move that is illegal? I played chess for years throughout elementary, middle and high school, and I would easily say that even after hundreds of hours of playing, I might make two mistakes out of a thousand moves where the move was actually illegal, often because I had missed that moving that piece would continue to leave me in check due to a discovered check that I had missed. It's hard to conclude from that experience that players that are amateurs lack even a basic model of chess.

libraryofbabel 1y ago

This essay could probably benefit from some engagement with the literature on “interpretability” in LLMs, including the empirical results about how knowledge (like addition) is represented inside the neural network. To be blunt, I’m not sure being smart and reasoning from first principles after asking the LLM a lot of questions and cherry picking what it gets wrong gets to any novel insights at this point. And it already feels a little out date, with LLMs getting gold on the mathematical Olympiad they clearly have a pretty good world model of mathematics. I don’t think cherry-picking a failure to prove 2 + 2 = 4 in the particular specific way the writer wanted to see disproves that at all. LLMs have imperfect world models, sure. (So do humans.) That’s because they are trained to be generalists and because their internal representations of things are massively compressed single they don’t have enough weights to encode everything. I don’t think this means there are some natural limits to what they can do.

yosefk 1y ago
Your being blunt is actually very kind, if you're describing what I'm doing as "being smart and reasoning from first principles"; and I agree that I am not saying something very novel, at most it's slightly contrarian given the current sentiment. My goal is not to cherry-pick failures for its own sake as much as to try to explain why I get pretty bad output from LLMs much of the time, which I do. They are also very useful to me at times. Let's see how my predictions hold up; I have made enough to look very wrong if they don't. Regarding "failure disproving success": it can't, but it can disprove a theory of how this success is achieved. And, I have much better examples than the 2+2=4, which I am citing as something that sorta works these says
- libraryofbabel 1y ago
  I mean yeah, it’s a good essay in that it made me think and try to articulate the gaps, and I’m always looking to read things that push back on AI hype. I usually just skip over the hype blogging. I think my biggest complaint is that the essay points out flaws in LLM’s world models (totally valid, they do confidently get things wrong and hallucinate in ways that are different, and often more frustrating, from how humans get things wrong) but then it jumps to claiming that there is some fundamental limitation about LLMs that prevents them from forming workable world models. In particular, it strays a bit towards the “they’re just stochastic parrots” critique, e.g. “that just shows the LLM knows to put the words explaining it after the words asking the question.” That just doesn’t seem to hold up in the face of e.g. LLMs getting gold on the Mathematical Olympiad, which features novel questions. If that isn’t a world model of mathematics - being able to apply learned techniques to challenging new questions - then I don’t know what is. A lot of that success is from reinforcement learning techniques where the LLM is made to solve tons of math problems after the pre-training “read everything” step, which then gives it a chance to update its weights. LLMs aren’t just trained from reading a lot of text anymore. It’s very similar to how the alpha zero chess engine was trained, in fact. I do think there’s a lot that the essay gets right. If I was to recast it, I’d put it something like this: * LLMs have imperfect models of the world which is conditioned by how they’re trained on next token prediction. * We’ve shown we can drastically improve those world models for particular tasks by reinforcement learning. you kind of allude to this already by talking about how they’ve been “flogged” to be good at math. * I would claim that there’s no particular reason these RL techniques aren’t extensible in principle to beat all sorts of benchmarks that might look unrealistic now. (Two years ago it would have been an extreme optimist position to say an LLM could get gold on the mathematical Olympiad, and most LLM skeptics would probably have said it could never happen.) * Of course it’s very expensive, so most world models LLMs have won’t get the RL treatment and so will be full of gaps, especially for things that aren’t amenable to RL. It’s good to beware of this. I think the biggest limitation LLMs actually have, the one that is the biggest barrier to AGI, is that they can’t learn on the job, during inference. This means that with a novel codebase they are never able to build a good model of it, because they can never update their weights. (If an LLM was given tons of RL training on that codebase, it could build a better world model, but that’s expensive and very challenging to set up.) This problem is hinted at in your essay, but the lack of on-the-job learning isn’t centered. But it’s the real elephant in the room with LLMs and the one the boosters don’t really have an answer to. Anyway thanks for writing this and responding!

GaggiX 1y ago

https://www.youtube.com/watch?v=LtG0ACIbmHw https://www.youtube.com/watch?v=LtG0ACIbmHw Sota LLMs do play legal moves in chess, I don't why the article seem to say otherwise.

tickettotranai 1y ago
Technically yes, but... it's moderately tricky to get an LLM to play good chess even though it can. https://dynomight.net/more-chess/ https://dynomight.net/more-chess/ This is significant in general because I personally would love to get these things to code-switch into "hackernews poster" or "writer for the Economist" or "academic philosopher", but I think the "chat" format makes it impossible. The inaccessibility of this makes me want to host my own LLM...

rishi_devan 1y ago

Haha. I enjoyed that Soviet-era joke at the end.

svantana 1y ago
Yes, I hadn't heard that before. It's similar in spirit to this norwegian folk tale about a deaf man guessing what someone is saying to him: https://en.wikipedia.org/wiki/%22Good_day,_fellow!%22_%22Axe_handle!%22 https://en.wikipedia.org/wiki/%22Good_day,_fellow!%22_%22Axe...
- kgwgk 1y ago
  Another similar story: King Frederick, the great of Prussia had a very fine army, and none of the soldiers in it were finer than Giant Guards, who were all extremely tall men. It was difficult to find enough soldiers for these Guards, as there were not many men who were tall enough. Frederick had made it a rule that no soldiers who did not speak German could be admitted to the Giant Guards, and this made the work of the officers who had to find men for them even more difficult. When they had to choose between accepting or refusing a really tall man who knew no German, the officers used to accept him, and then teach him enough. German to be able to answer if the King questioned him. Frederick, sometimes, used to visit the men who were on guard around his castle at night to see that they were doing their job properly, and it was his habit to ask each new one that he saw three questions: “How old are you?” “How long have you been in my army?” and “Are you satisfied with your food and your conditions?” The offices of the Giant Guards therefore used to teach new soldiers who did not know German the answers to these three questions. One day, however, the King asked a new soldier the questions in a different order, he began with, “How long have you been in my army?” The young soldier immediately answered, “Twenty – two years, Your Majesty”. Frederick was very surprised. “How old are you then?”, he asked the soldier. “Six months, Your Majesty”, came the answer. At this Frederick became angry, “Am I a fool, or are you one?” he asked. “Both, Your Majesty”, the soldier answered politely. https://archive.org/details/advancedstoriesf0000hill https://archive.org/details/advancedstoriesf0000hill

deadbabe 1y ago

Don’t: use LLMs to play chess against you Do: use LLMs to talk shit to you while a real chess AI plays chess against you. The above applies to a lot of things besides chess, and illustrates a proper application of LLMs.

Seb-C 1y ago
Are you suggesting that we use an LLM as an interface between the AI and the player? Why would anyone choose to awkwardly play using natural language rather than a reliable, fast and intuitive UI?
- crabmusket 1y ago
  No, I think they're suggesting the LLM should literally be "talking shit", e.g. in a chat window alongside the game UI, as if you're in a live chat with another player. As in, use the LLM for processing language, and the chess engine for playing chess. I think this is quite an amusing idea, as the LLM would see the moves the chess engine made and comment along the lines of "wow, I didn't see that one coming!" very Roger Sperry.

imenani 1y ago

As far as I can tell they don’t say which LLM they used which is kind of a shame as there is a huge range of capabilities even in newly released LLMs (e.g. reasoning vs not).

yosefk 1y ago
ChatGPT, Claude, Grok and Google AI Overviews, whatever powers the latter, were all used in one or more of these examples, in various configurations. I think they can perform differently, and I often try more than one when the 1st try doesn't work great. I don't think there's any fundamental difference in the principle of their operation, and I think there never will be - there will be another major breakthrough
- red75prime 1y ago
  My hypothesis is that a model fails to switch into a deep thinking mode (if it has it) and blurts whatever it got from all the internet data during autoregressive training. I tested it with alpha-blending example. Gemini 2.5 flash - fails, Gemini 2.5 pro - succeeds. How presence/absence of a world model, er, blends into all this? I guess "having a consistent world model at all times" is an incorrect description of humans, too. We seem to have it because we have mechanisms to notice errors, correct errors, remember the results, and use the results when similar situations arise, while slowly updating intuitions about the world to incorporate changes. The current models lack "remember/use/update" parts.
- imenani 1y ago
  Each of these models has a thinking/reasoning variant and a default non-thinking variant. I would expect the reasoning variants (o3 or “GPT5 Thinking”, Gemini DeepThink, Claude with Extended Thinking, etc) to do better at this. I think there is also some chance that in their reasoning traces they may display something you might see as closer to world modelling. In particular, you might find them explicitly tracking positions of pieces and checking validity.
- red75prime 1y ago
  > I don't think there's any fundamental difference in the principle of their operation Yeah, they seem to be a subject to the universal approximation theorem (it needs to be checked more thoroughly, but I think we can build a transformer that is equivalent to any given fully-connected multilayered network). That is at a certain size they can do anything a human can do at a certain point in their life (that is with no additional training) regardless of whether humans have world models and what those model are on the neuronal level. But there are additional nuances that are related to their architectures and training regimes. And practical questions of the required size.

famouswaffles 1y ago

Yes LLMs can play chess and yes they can model it fine https://arxiv.org/pdf/2403.15498v2 https://arxiv.org/pdf/2403.15498v2

lordnacho 1y ago

Here's what LLMs remind me of. When I went to uni, we had tutorials several times a week. Two students, one professor, going over whatever was being studied that week. The professor would ask insightful questions, and the students would try to answer. Sometimes, I would answer a question correctly without actually understanding what I was saying. I would be spewing out something that I had read somewhere in the huge pile of books, and it would be a sentence, with certain special words in it, that the professor would accept as an answer. But I would sometimes have this weird feeling of "hmm I actually don't get it" regardless. This is kinda what the tutorial is for, though. With a bit more prodding, the prof will ask something that you genuinely cannot produce a suitable word salad for, and you would be found out. In math-type tutorials it would be things like realizing some equation was useful for finding an answer without having a clue about what the equation actually represented. In economics tutorials it would be spewing out words about inflation or growth or some particular author but then having nothing to back up the intuition. This is what I suspect LLMs do. They can often be very useful to someone who actually has the models in their minds, but not the data to hand. You may have forgotten the supporting evidence for some position, or you might have missed some piece of the argument due to imperfect memory. In these cases, LLM is fantastic as it just glues together plausible related words for you to examine. The wheels come off when you're not an expert. Everything it says will sound plausible. When you challenge it, it just apologizes and pretends to correct itself.

roywiggins 1y ago
> When you challenge it, it just apologizes and pretends to correct itself. Even when it was right the first time!
nhaehnle 1y ago
Good on you for having the meta-cognition to recognize it. I've graded many exams in my university days (and set some myself), and it's exceedingly obvious that that's what many students are doing. I do wonder though how often they manage to fly under the radar. I'm sure it happens, as you described. (This is also the reason why I strongly believe that in exams where students write free-form answers, points should be subtracted for incorrect statements even if a correct solution is somewhere in the word salad.)

ej88 1y ago

This article is interesting but pretty shallow. 0(?): there’s no provided definition of what a ‘world model’ is. Is it playing chess? Is it remembering facts like how computers use math to blend Colors? If so, then ChatGPT: https://chatgpt.com/s/t_6898fe6178b88191a138fba8824c1a2c https://chatgpt.com/s/t_6898fe6178b88191a138fba8824c1a2c has a world model right? 1. The author seems to conflate context windows with failing to model the world in the chess example. I challenge them to ask a SOTA model with an image of a chess board or notation and ask it about the position. It might not give you GM level analysis but it definitely has a model of what’s going on. 2. Without explaining which LLM they used or sharing the chats these examples are just not valuable. The larger and better the model, the better its internal representation of the world. You can try it yourself. Come up with some question involving interacting with the world and / or physics and ask GPT-5 Thinking. It’s got a pretty good understanding of how things work! https://chatgpt.com/s/t_689903b03e6c8191b7ce1b85b1698358 https://chatgpt.com/s/t_689903b03e6c8191b7ce1b85b1698358

yosefk 1y ago
A "world model" depends on the context which defines which world the problem is in. For chess, which moves are legal and needing to know where the pieces are to make legal moves are parts of the world model. For alpha blending, it being a mathematical operation and the visibility of a background given the transparency of the foreground are parts of the world model. The examples are from all the major commercial American LLMs as listed in a sister comment. You seem to conflate context windows with tracking chess pieces. The context windows are more than large enough to remember 10 moves. The model should either track the pieces, or mention that it would be playing blindfold chess absent a board to look at and it isn't good at this, so could you please list the position after every move to make it fair, or it doesn't know what it's doing; it's demonstrably the latter.
- tmnvdb 1y ago
  If you train an LLM on chess, it will learn that too. You don't need to explain the rules, just feed it chess games, and it will stop making illegal moves at some point. It is a clear example of an inferred world model from training. https://arxiv.org/abs/2501.17186 https://arxiv.org/abs/2501.17186 PS "Major commercial American LLM" is not very meaningful, you could be using GPT4o with that description.

jonplackett 1y ago

I just tried a few things that are simple and a world model would probably get right. Eg Question to GPT5: I am looking straight on to some objects. Looking parallel to the ground. In front of me I have a milk bottle, to the right of that is a Coca-Cola bottle. To the right of that is a glass of water. And to the right of that there’s a cherry. Behind the cherry there’s a cactus and to the left of that there’s a peanut. Everything is spaced evenly. Can I see the peanut? Answer (after choosing thinking mode) No. The cactus is directly behind the cherry (front row order: milk, Coke, water, cherry). “To the left of that” puts the peanut behind the glass of water. Since you’re looking straight on, the glass sits in front and occludes the peanut. It doesn’t consider transparency until you mention it, then apologises and says it didn’t think of transparency

RugnirViking 1y ago
this seems like a strange riddle. In my mind I was thinking that regardless of the glass, all of the objects can be seen (due to perspective, and also the fact you mentioned the locations, meaning you're aware of them). It seems to me it would only actually work in an orthographic perspective, which is not how our reality works
- jonplackett 1y ago
  You can tell from the response it does understand the riddle just fine, it just gets it wrong.
  - rpdillon 1y ago
    Have you asked five adults this riddle? I suspect at least two of them would get it wrong or have some uncertainty about whether or not the peanut was visible.
    - xg15 1y ago
      This. Was also thinking "yes" first because of the glass of water, transparency, etc, but then got unsure: The objects might be spaced so widely that the milk or coke bottle would obscure the view due to perspective - or the peanut would simply end up outside the viewer's field of vision. Shows that even if you have a world model, it might not be the right one.

Razengan 1y ago

A slight tangent: I think/wonder if the one place where AIs could be really useful, might be in translating alien languages :) As in, an alien could teach one of our AIs their language faster than an alien could teach an human, and vice versa.. ..though the potential for catastrophic disasters is also great there lol

keeda 1y ago

That whole bit about color blending and transparency and LLMs "not knowing colors" is hard to believe. I am literally using LLMs every day to write image-processing and computer vision code using OpenCV. It seamlessly reasons across a range of concepts like color spaces, resolution, compression artifacts, filtering, segmentation and human perception. I mean, removing the alpha from a PNG image was a preprocessing step it wrote by itself as part of a larger task I had given it, so it certainly understands transparency. I even often describe the results e.g. "this fails when in X manner when the image has grainy regions" and it figures out what is going on, and adapts the code accordingly. (It works with uploading actual images too, but those consume a lot of tokens!) And all this in a rather niche domain that seems relatively less explored. The images I'm working with are rather small and low-resolution, which most literature does not seem to contemplate much. It uses standard techniques well known in the art, but it adapts and combines them well to suit my particular requirements. So they seem to handle "novel" pretty well too. If it can reason about images and vision and write working code for niche problems I throw at it, whether it "knows" colors in the human sense is a purely philosophical question.

geraneum 1y ago
> it wrote by itself as part of a larger task I had given it, so it certainly understands transparency Or it’s a common step or a known pattern or combination of steps that is prevalent in its training data for certain input. I’m guessing you don’t know what’s exactly in the training sets. I don’t know either. They don’t tell ;) > but it adapts and combines them well to suit my particular requirements. So they seem to handle "novel" pretty well too. We tend to overestimate the novelty of our own work and our methods and at the same time, underestimate the vastness of the data and information available online for machines to train on. LLMs are very sophisticated pattern recognizers. It doesn’t mean what you are doing specifically is done in this exact way before, rather the patterns adapted and the approach may not be one of their kind. > is a purely philosophical question It is indeed. A question we need to ask ourselves.
- Uehreka 1y ago
  > We tend to overestimate the novelty of our own work and our methods and at the same time, underestimate the vastness of the data and information available online for machines to train on. LLMs are very sophisticated pattern recognizers. If LLMs are stochastic parrots, but also we’re just stochastic parrots, then what does it matter? That would mean that LLMs are in fact useful for many things (which is what I care about far more than any abstract discussion of free will).

skeledrew 1y ago

Agree in general with most of the points, except > but because I know you and I get by with less. Actually we got far more data and training than any LLM. We've been gathering and processing sensory data every second at least since birth (more processing than gathering when asleep), and are only really considered fully intelligent in our late teens to mid-20s.

helloplanets 1y ago
Don't forget the millions of years of pre-training! ;)

o_nate 1y ago

What with this and your previous post about why sometimes incompetent management leads to better outcomes, you are quickly becoming one of my favorite tech bloggers. Perhaps I enjoyed the piece so much because your conclusions basically track mine. (I'm a software developer who has dabbled with LLMs, and has some hand-wavey background on how they work, but otherwise can claim no special knowledge.) Also your writing style really pops. No one would accuse your post of having been generated by an LLM.

yosefk 1y ago
thank you for your kind words!

neuroelectron 1y ago

Not yet

ameliaquining 1y ago

One thing I appreciated about this post, unlike a lot of AI-skeptic posts, is that it actually makes a concrete falsifiable prediction; specifically, "LLMs will never manage to deal with large code bases 'autonomously'". So in the future we can look back and see whether it was right. For my part, I'd give 80% confidence that LLMs will be able to do this within two years, without fundamental architectural changes.

moduspol 1y ago
"Deal with" and "autonomously" are doing a lot of heavy lifting there. Cursor already does a pretty good job indexing all the files in a code base in a way that lets it ask questions and get answers pretty quickly. It's just a matter of where you set the goalposts.
- ameliaquining 1y ago
  True, there'd be a need to operationalize these things a bit more than is done in the post to have a good advance prediction.
- jononor 1y ago
  "LLM" as well, because coding agents are already more than just an LLM. There is very useful context management around it, and tool calling, and ability to run tests/programs, etc. Though they are LLM-based systems, they are not LLMs.
  - smnrchrds 1y ago
    Indeed. If the LLM calls a chess engine tool behind the scenes, it would be able to play excellent chess as well.
    - cavisne 1y ago
      The author would still be wrong in the tool-calling scenario. There is already perfect (or at least superhuman) chess engines. There is no perfect "coding engine". LLM's + tools being able to reliably work on large codebases would be a new thing.

bithive123 1y ago

Language models aren't world models for the same reason languages aren't world models. Symbols, by definition, only represent a thing. They are not the same as the thing. The map is not the territory, the description is not the described, you can't get wet in the word "water". They only have meaning to sentient beings, and that meaning is heavily subjective and contextual. But there appear to be some who think that we can grasp truth through mechanical symbol manipulation. Perhaps we just need to add a few million more symbols, they think. If we accept the incompleteness theorem, then there are true propositions that even a super-intelligent AGI would not be able to express, because all it can do is output a series of placeholders. Not to mention the obvious fallacy of knowing super-intelligence when we see it. Can you write a test suite for it?

habitue 1y ago
> Symbols, by definition, only represent a thing. This is missing the lesson of the Yoneda Lemma: symbols are uniquely identified by their relationships with other symbols. If those relationships are represented in text, then in principle they can be inferred and navigated by an LLM. Some relationships are not represented well in text: tacit knowledge like how hard to twist a bottle cap to get it to come off, etc. We aren't capturing those relationships between all your individual muscles and your brain well in language, so an LLM will miss them or have very approximate versions of them, but... that's always been the problem with tacit knowledge: it's the exact kind of knowledge that's hard to communicate!
- nomel 1y ago
  I don’t think it’s a communication problem as much as there is no possible relation between a word and a (literal) physical experiences. They’re, quite literally, on different planes of existence.
  - drdeca 1y ago
    When I have a physical experience, sometimes it results in me saying a word. Now, maybe there are other possible experiences that would result in me behaving identically, such that from my behavior (including what words I say) it is impossible to distinguish between different potential experiences I could have had. But, “caused me to say” is a relation, is it not? Unless you want to say that it wasn’t the experience that caused me to do something, but some physical thing that went along with the experience, either causing or co-occurring with the experience, and also causing me to say the word I said. But, that would still be a relation, I think.

frankfrank13 1y ago

Great quote at the end that I think I resonate a lot with: > Feeding these algorithms gobs of data is another example of how an approach that must be fundamentally incorrect at least in some sense, as evidenced by how data-hungry it is, can be taken very far by engineering efforts — as long as something is useful enough to fund such efforts and isn’t outcompeted by a new idea, it can persist.

1970-01-01 1y ago

I'm surprised the models haven't been enshittified by capitalism. I think in a few years we're going to see lightning-fast LLMs generating better output compared to what we're seeing today. But it won't be 1000x better, it will be 10x better, 10x faster, and completely enshittified with ads and clickbait links. Enjoy ChatGPT while it lasts.

UltraSane 1y ago

I wonder how the nature of the language used to train an LLM affects its model of the world. Would a language designed for the maximum possible information content and clarity like Ithkuil make an LLMs world model more accurate?

DennisP 1y ago

Maybe pure language models aren't world models, but Genie 3 for example seems to be a pretty good world model: https://deepmind.google/discover/blog/genie-3-a-new-frontier-for-world-models/ https://deepmind.google/discover/blog/genie-3-a-new-frontier... We also have multimodal AIs that can do both language and video. Genie 3 made multimodal with language might be pretty impressive. Focusing only on what pure language models can do is a bit of a straw man at this point.

ordu 1y ago

> LLMs are not by themselves sufficient as a path to general machine intelligence; in some sense they are a distraction because of how far you can take them despite the approach being fundamentally incorrect. I don't believe that it is a fundamentally incorrect approach. I believe, that human mind does something like that all the time, the difference is our minds have some additional processes that can, for example, filter hallucinations. Kids at a specific age range are afraid of their imagination. Their imagination can place a monster into any dark place where nothing can be seen. Adult mind can do the same easily, but the difference is kids have difficulties distinguishing imagination and perception, while adult generally manage. I believe, the ability of human mind to see difference between imagination/hallucinations from one hand and perception and memory from the other is not a fundamental thing stemming from the architecture of brains but a learned skill. Moreover people can be tricked to acquire false memory[1]. If LLM fell to tricks of Elizabet Loftus, we'd say LLM hallucinated. What LLMs need is to learn some tricks to detect hallucinations. Probably they will not get 100% reliable detector, but to get to the level of humans they don't need 100% reliability.

shkkmo 1y ago
> If LLM fell to tricks of Elizabet Loftus, we'd say LLM hallucinated. She's strongly oversold how and when false memories can be created. She testified in defense of Ghislaine Maxwell at her 2021 trial that financial incentives can create false memories and only later admitted that there were no studies to back this up when directly questioned. She's spent a career over-generalizing data about implanting false minor memories to make money discrediting victims' traumatic memories and defend abusers. You conflate "hallucination" with "imagination" but the former has much more in common with lieing than it does with imagining.
- taneq 1y ago
  > She testified in defense of Ghislaine Maxwell at her 2021 trial that financial incentives can create false memories and only later admitted that there were no studies to back this up when directly questioned. Did she have financial incentives? Was this a live demonstration? :P
1y ago

globular-toast 1y ago

I agree with the article. I will be very surprised if LLMs end up being "it". I say this as a language geek who has always been amazed how language drives our thinking. However, I think language exists between brains, not inside them. There's something else in us and LLMs aren't it.

revskill 1y ago

LLM is not AI, it's dumbass, too stupid to NOT assume and hallucinate.

mft_ 1y ago
Lots of people assume, confabulate, misremember, and lie every day.
- revskill 1y ago
  They are not intelligent.
  - mft_ 1y ago
    LOL, you really think that intelligence (however you want to define or measure the concept) is a guarantee that people won't make mistakes, misremember, make stuff up, or lie?

antirez 1y ago

The post is based on a misconception. If you read the blog post linked at the end of this message, you'll see how a very small GPT-2 alike transformer (Karpathy nano-gpt trained to a very small size) after seeing just PGN games and nothing more develops an 8x8 internal representation with which chess piece is where. This representation can be extracted by linear probing (and can be even altered by using the probe in reverse). LLMs are decent but not very good chess players for other reasons, not because they don't have a world model of the chess board. https://www.lesswrong.com/posts/yzGDwpRBx6TEcdeA5/a-chess-gpt-linear-emergent-world-representation https://www.lesswrong.com/posts/yzGDwpRBx6TEcdeA5/a-chess-gp...

yosefk 1y ago
The post or rather the part you refer to is based on a simple experiment which I encourage you to repeat. (It is way likelier to reproduce in the short to medium run than the others.) From your link: "...The first was gpt-3.5-turbo-instruct's ability to play chess at 1800 Elo" These things don't play at 1800 ELO, though maybe someone measured this ELO without cheating but rather relying on some artifacts of how an engine told to play at a low rating does against an LLM (engines are weird when you ask them to play badly, as a rule); a good start to a decent measurement would be to try it on chess 960. These things do lose track of the pieces in 10 moves. (As do I absent a board to look at, but I understand enough to say "I can't play blindfold chess, let's set things up so I can look at the current position somehow")
- famouswaffles 1y ago
  >These things don't play at 1800 ELO Why are you saying 'these things'?. That statement is about a specific model which did play at that level and did not lose track of the pieces. There's no cheating or weirdness. https://github.com/adamkarvonen/chess_gpt_eval https://github.com/adamkarvonen/chess_gpt_eval
thecupisblue 1y ago
Ironically, that lesswrong article is more wrong than right. First, chess is perfect for such modeling. The game is basically a tree of legal moves. The "world model" representation is already encoded in the dataset itself and at a certain scale the chance of making an illegal move is minimal, as the dataset itself includes an insane amount of legal moves compared to illegal moves, let alone when you are training it on a chess dataset like PGN one Second, the probing is quite... a subjective thing. We are cherry-picking activations across an arbitrary amount of dimensions, on a model specifically trained for chess, taking these arbitrary representations and displaying it on 2D graph. Well yeah, with enough dimensions and cherry-picking, we can also show how "all zebras are elephants, because all elephants are horses and look their weights overlap in so many dimensions - large four-legged animals you see on safari!" - especially if we cherry-pick it. Especially if we tune a dataset on it. This shows nothing other than "training LLMs on a constrained move dataset makes LLM great at predicting next move in that dataset".

FailMore 1y ago

It seems clear to me that LLMs are a useful sort of dumb smart activity. They can take some pretty useful stabs in the dark, and therefore do much better in an environment which can give them feedback (coding) or where there is no objective correct answer (write a poem). It opens the door for some novel type of computational tasks, and the more feedback you can provide within the architecture of your application, the more useful the LLM will probably be. I think the hype of their genuine intelligence is overblown, but doesn’t mean they are not useful.

Peteragain 1y ago

This article appeared on HN a while ago. https://dynomight.net/more-chess/ https://dynomight.net/more-chess/ It basically is in agreement with this article and provides a few more trials and explanations.

devxpy 1y ago

I don’t think you even need such complex tests as chess to see it doesn’t have a world model - just ask it to do any 5+ digit multiplication

machiaweliczny 1y ago

I think they know this but don’t have causality built-in. In the sense they aren’t incentivised to understand holistically. Kids around 4 years old are spamming with “why? why? why?” questions and I think this is some process we need yet to reproduce. (BTW I suspect they ask this as manifestation of what is going in their brain and not real curiosity as they ask same question multiple times)

machiaweliczny 1y ago
I think it’s mostly because they are incentivised to answer verbatim as medicine students and not with their own understanding. RL methods change that.

glenstein 1y ago

This is the best and clearest explanation I have yet seen that describe a tricky thing, namely that LLMs, which are synonymous with "AI" for so many people, are just one variation of many possible types of machine intelligence. Which I find important because, well, hallucinating facts is what you would expect from an LLM, but isn't necessarily inherent issue with machine intelligence writ large if it's trained from the ground up on different principles, or modelling something else. We use LLMs as a stand in for tutors because being really good at language incidentally makes them able to explain math or history as a side effect. Importantly it doesn't show that hallucinating is a baked in problem for AI writ large. Presumably different models will have different kinds of systemic errors based on their respective designs.

m4nu3l 1y ago

Developing a model of the real world, or even just learning only a subset of self-consistent information, could be detrimental to the task of predicting the next token in the average text, given that most of the written information on many subjects could be contradictory and somehow wrong. I don't know how they are doing RL on top of that, how they are using synthetic data or filtering them. But it's clear that even with GPT-5 they haven't solved the problem, as the presentation demonstrated with the very first prompt (I'm talking about the wrong explanation for lift produced by a wing).

maebert 1y ago

It might be worth noting that humans also struggle with keeping up a coherent world model over time. Luckily, we don’t have to; we externalize a lot of our representations. When shopping together with a friend we might put our stuff on one side of the shopping cart and our friends’ on the other. There’s a reason we don’t just play chess in our heads but use a chess board. We use notebooks to write things down, etc. Some reasoning model can do similar things (keep a persistent notebook that gets fed back into the context window on every pass), but I expect that we need a few more dirty representational ist tricks to get there. In other words, I don’t think it’s an LLMs job to have a world model, but an LLM is just one part of an AI system.

3willows 1y ago

I've been looking for a way to play chess/go with a generalist LLM. It wouldn't matter if the moves are bad (I like winning), but being able to chat on unrelated topics while playing the game would take the experience to the next level.

quantumcotton 1y ago

I suppose teaching mine gsis and point clouds and connecting it to the world gsis map is a bad idea?