7 ms·
Do Large Language Models learn world models or just surface statistics? (2023)
- pvg 2y agoBig thread at the time https://news.ycombinator.com/item?id=34474043 https://news.ycombinator.com/item?id=34474043
- randcraw 2y agoThanks. Now, after almost two years of incomparably explosive growth in LLMs since that paper, it's remarkable to realize that we still don't know if Scarecrow has a brain. Or if he'll forever remain just a song and dance man.
- deleted 2y ago[deleted]
- dang 2y agoThanks! Macroexpanded: Do Large Language Models learn world models or just surface statistics? - https://news.ycombinator.com/item?id=34474043 https://news.ycombinator.com/item?id=34474043 - Jan 2023 (174 comments)
- mjburgess 2y agoThis is irrelevant, and it's very frustrating that computer scientists think it is relevant. If you give a universal function approximator the task of approximating an abstract function, you will get an approximation. Eg., def circle(radius): ... return points() aprox_cricle = neuralnetwork(sample(circle())) if is_model_of(samples(aprox_circle), circle)): print("OF COURSE!") This is irrelevant: games, rules, shapes, etc. are all abstract. So any model of samples of these is a model of them. The "world model" in question is a model of the world. Here "data" is not computer science data, ie., numbers its measurements of the world, ie., the state of a measuring device causally induced by the target of measurement. Here there is no "world" in the data, you have to make strong causal assumptions about what properties of the target cause the measures. This is not in the data. There is no "world model" in measurement data. Hence the entirety of experimental science. No result based on one mathematical function succeeding in approximating another is relevant whether measurement data "contains" a theory of the world which generates it: it does not. And of course if your data is abstract, and hence constitutes the target of modelling (only applies to pure math), then there is no gap -- a model of "measures" (ie., the points on a circle) is the target. No model of actual measurement data, ie., no model in the whole family we call "machine learning", is a model of its generating process. It contains no "world model". Photographs of the night sky are compatible with all theories of the solar system in human history (including, eg., stars are angels). There is no summary of these photographs which gives information about the world over and above just summarising patterns in the night sky. The sense in which any model of measurement data is "surface statistics" is the same. Consider plato's cave: pots, swords, etc. on the outside project shadows inside. Modelling the measurement data is taking cardboard and cutting it out so it matches the shadows. Modelling the world means creating clay pots to match the ones passing by. The latter is science: you build models of the world and compare them to data, using the data to decide between them. The former is engineering (, pseudoscience): you take models of measures and reply these models to "predict" the next shadow. If you claim the latter is just a "surface shortcut" you're an engineer. If you claim its a world model you're a pseudoscientist.
- deleted 2y ago[deleted]
- sebzim4500 2y agoI don't understand your objection at all. In the example, the 'world' is the grid state. Obviously that's much simpler than the real world but the point is to show that even when the model is not directly trained to input/output this world state it is still learned as a side effect of prediction the next token.
- mjburgess 2y agoThere is no world. The grid state is not a world, there is no causal relationship between the grid state and the board. No one in this debate denies that NNs approximate functions. Since a game is just a discrete function, no one denies an NN can approximate it. Showing this is entirely irrelevant and shows a profound misunderstanding of what's at issue. The whole debate is about whether surface patterns in measurement data can be reversed by NNs to describe their generating process, ie., the world. If the "data" isnt actual measurements of the world, no one arguing about it. If there is no gap between the generating algorithm and the samples, eg., between a "circle" and "the points on a circle" -- then there is no "world model" to learn. The world is the data. To learn "the points on a cirlce" is to learn the cirlce. By taking cases where "the world" and "the data" are the same object (in the limit of all samples), you're just showing that NNs model data. That's already obvious, no ones arguing about it. That a NN can approximate a discrete function does not mean it can do science. The whole issue is that the cause of pixel distributions is not in those distributions. A model of pixel patterns is just a model of pixel patterns, not of the objects which cause those patterns. A TV is not made out of pixels. The "debate" insofar as there is one, is just some researchers being profoundly confused about what measurement data is: measurements are not their targets, and so no model of data is a model of the target. A model of data is just "surface statistics" in the sense that these statistics describe measurements, not what caused them.
- bubblyworld 2y ago> There is no summary of these photographs which gives information about the world over and above just summarising patterns in the night sky. You're stating this as fact but it seems to be the very hypothesis the authors (and related papers) are exploring. To my mind, the OthelloGPT papers are plainly evidence against what you've written - summarising patterns in the sky really does seem to give you information about the world above and beyond the patterns themselves. (to a scientist this is obvious, no? the precession of mercury, a pattern observable in these photographs, was famously not compatible with known theories until fairly recently) > Modelling the measurement data is taking cardboard and cutting it out so it matches the shadows. Modelling the world means creating clay pots to match the ones passing by. I think these are matters of degree. The former is simply a worse model than the latter of the "reality" in this case. Note that our human impressions of what a pot "is" are shadows too, on a higher-dimensional stage, and from a deeper viewpoint any pot we build to "match" reality will likely be just as flawed. Turtles all the way down.
- foobarqux 2y agoLots of problems with this paper including the fact that, even if you accept their claim that internal board state is equivalent to world model, they don't appear to do the obvious thing which is display the reconstructed "internal" board state. More fundamentally though, reifying the internal board as a "world model" is absurd: otherwise a (trivial) autoencoder would also be building a "world model".
- sebzim4500 2y ago>More fundamentally though, reifying the internal board as a "world model" is absurd: otherwise a (trivial) autoencoder would also be building a "world model". The point is that they aren't directly training the model to output the grid state, like you would an autoencoder. It's trained to predict the next action and learning the state of the 'world' happens incidentally. It's like how LLMs learn to build world models without directly being trained to do so, just in order to predict the next token.
- optimalsolver 2y ago>It's like how LLMs learn to build world models without directly being trained to do so, just in order to predict the next token That's the whole point under contention, but you're stating it as fact.
- foobarqux 2y agoBy the same reasoning if you train a neural net to output next action from the output of the autoencoder then the whole system also has a "world model", but if you accept that definition of "world model" then it is extremely weak and not the intelligence-like capability that is being implied. And as I said in my original comment they are probably not even able to extract the board state very well, otherwise they would depict some kind of direct representation of the state, not all of the other figures of board move causality etc. Note also that the board state is not directly encoded in the neural network: they train another neural network to find weights to approximate the board state if given the internal weights of the Othello network. It's a bit of fishing for the answer you want.
- burnt-resistor 2y agoI think they learn how to become salespeople, politicians, lawyers, and résumé consultants with fanciful language lacking in facts, truth, and honesty.
- vonneumannstan 2y ago[flagged]
- 01HNNWZ0MV43FF 2y agoIf we can put salespeople out of work it will be a great boon to humankind
- natpalmer1776 2y agoI suddenly have a vision of an AI driven sales pipeline that uses millions of invasive datapoints about you to create the most convincing sales pitch mathematically possible.
- javaunsafe2019 2y agoIdk from when even id this article? Got me LLMs currently are broke and the majority is already aware of this. Copilot fails the cleanly refactor complex Java methods in a way that I’m better of writing that stuff by my own as I have to understand it anyways. And the news that they don’t scale as predicted is too bad compared to how weak they currently perform…
- lxgr 2y agoWhy does an LLM have to be better than you to be useful to you? Personally, I use them for the things they can do, and for the things they can't, I just don't, exactly as I would for any other tool. People assuming they can do more than they are actually capable of is a problem (compounded by our tendency to attribute intelligence to entities with eloquent language, which might be more of a surface level thing than we used to believe), but that's literally been one for as long as we had proverbial hammers and nails.
- lou1306 2y ago> Why does an LLM have to be better than you to be useful to you? If ((time to craft the prompt) + (time required to fix LLM output)) ~ (time to achieve the task on my own) it's not hard to see that working on my own is a very attractive proposition. It drives down complexity, does not require me to acquire new skills (i.e., prompt engineering), does not require me to provide data to a third party nor to set up an expensive rig to run a model locally, etc.
- lxgr 2y agoThen they might indeed not be the right tool for what you're trying to do. I'm just a little bit tired of sweeping generalizations like "LLMs are completely broken". You can easily use them as a tool part of a process that then ends up being broken (because it's the wrong tool!), yet that doesn't disqualify them for all tool use.
- lou1306 2y ago
- dboreham 2y agoIt turns out our word for "surface statistics" is "world model".
- marcosdumay 2y agoWell, for some sufficiently platonic definition of "world".
- mdp2021 2y agoIn a way the opposite, I'd say: the archetypes in Plato are the most stable reality and are akin to the logos that the past and future tradition hunted - knowing it is to know how things are (how things work), hence knowledge of the state of things, hence a faithful world model. To utter conformist statements spawned from surface statistics would be "doxa" - repeating "opinions".
- marcosdumay 2y agoIt has a profound and extensive knowledge about something. But that "something" is how words follow each other on popular media. LLMs are very firmly stuck inside the Cave Allegory.
- mdp2021 2y agoIf you mean that just like the experiencer in the cave, seeing shadows instead of things (really, things instead of Ideas), the machine sees words instead of things, that would be in a way very right. But we could argue it could not be impossible to create an ontology (a very descriptive ontology - "this is said to be that, and that, and that...") from language alone. Hence the question whether the ontology is there. (Actually, the question at this stage remains: "How do they work - in sufficient detail? Why the appearance of some understanding?")
- marcosdumay 2y agoYeah, what I'm saying is that something very similar to an ontology is there. (It's incomplete but extensive, not coherent, and it's deeper in details than anything anybody ever created.) It's just that it's a kind of a useless ontology, because the reality it's describing is language. Well, only "kind of useless" because it should be very useful to parse, synthesize and transform language. But it doesn't have the kind of "knowledge" that most people expect an intelligence to have. Also, its world isn't only composed of words. All of them got a very strong "Am I fooling somebody?" signal during training.
- jebarker 2y ago> Do they merely memorize training data and reread it out loud, or are they picking up the rules of English grammar and the syntax of C language? This is a false dichotomy. Functionally the reality is in the middle. They "memorize" training data in the sense that the loss curve is fit to these points but at test time they are asked to interpolate (and extrapolate) to new points. How well they generalize depends on how well an interpolation between training points works. If it reliably works then you could say that interpolation is a good approximation of some grammar rule, say. It's all about the data.
- mjburgess 2y agoThis only applies to intra-distribution "generalisation", which is not the meaning of the term we've come to associate with science. Here generalisation means across all environments (ie., something generalises if its valid and reliable where valid = measures property, and reliable = under causal permutation to the environment). Since an LLM does not change in response to the change in meaning of terms (eg., consider the change to "the war in ukraine" over the last 10 years) -- it isn't reliable in the scientific sense. Explaining why it isnt valid would take much longer, but its not valid either. In any case: the notion of 'generalisation' used in ML just means we assume there is a single stationary distribution of words, and we want to randomly sample from that distribution without bias to oversampling from points identical to the data. Not least that this assumption is false (there is no stationary distribution), it is also irrelevant to generalisation in traditional sense. Since whether we are biased towards the data or not isn't what we're interested in. We want output to be valid (the system to use words to mean what they mean) and to be reliable (to do so across all environments in which they mean something). This does not follow from, nor is it even related to, this ML sense of generalisation. Indeed, if LLMs generalised in this sense, they would be very bad at usefully generalising -- since the assumptions here are false.
- jebarker 2y agoI don't really follow what you're saying here. I understand that the use of language in the real-world world is not sampled from a stationary distribution, but it also seems plausible that you could relax that assumption in an LLM, e.g. conditioning the distribution on time, and then intra-distribution generalization would still make sense to study how well the LLM works for held-out test samples. Intra-distribution generalization seems like the only rigorously defined kind of generalization we have. Can you provide any references that describe this other kind of generalization? I'd love to learn more.
- maximus93 2y agoHonestly, I think it’s somewhere in between. LLMs are great at spotting patterns in data and using that to make predictions, so you could say they build a sort of "world model" for the data they see. But it’s not the same as truly understanding or reasoning about the world, it’s more like theyre really good at connecting the dots we give them. They dont do science or causality theyre just working with the shadows on the wall, not the actual objects casting them. So yeah, they’re impressive, but let’s not overhype what they’re doing. It’s pattern matching at scale, not magic. Correct me if I am wrong.
- not2b 2y agoThey are learning a grammar, finding structure in the text. In the case of Othello, the rules for what moves are valid are quite simple, and can be represented in a very small model. The slogan is "a minute to learn, a lifetime to master". So "what is a legal move" is a much simpler problem than "what is a winning strategy". It's similar to asking a model to only produce outputs corresponding to a regular expression, given a very large number of inputs that match that regular expression. The RE is the most compact representation that matches them all and it can figure this out. But we aren't building a "world model", we're building a model of the training data. In artificial problems with simple rules, the model might be essentially perfect, never producing an invalid Othello move, because the problem is so limited. I'd be cautious about generalizing from this work to a more open-ended situation.
- famouswaffles 2y agoI don't think the point is that Othello-GPT has somehow modellled the real world training on only games but that tasking it to predict the next move forces it to model its data in a deep way. There's nothing special about Othello games vs internet text except that the latter will force it to model much more things.
- tyronehed 2y ago[dead]
- setopt 2y agoI’m reminded of the Holographic Principle in physics: https://en.m.wikipedia.org/wiki/Holographic_principle https://en.m.wikipedia.org/wiki/Holographic_principle Sometimes a sufficiently good model of a surface is completely identical to a model of the volume.