2 ms·

Technically, under a compressive regime in the limit, LLMs necessarily need to encode an implicit world model, as the text input into is sampled from said world

Technically, under a compressive regime in the limit, LLMs necessarily need to encode an implicit world model, as the text input into is sampled from said world model. Variety, amount of training samples, capacity of the model, etc come into play, but a lot of tie ins to info theory for this, as well as things like the MDL (achieved in the limit by an L2 decay over the weights, for example). Like, in a lossy compression environment, conceptual "understanding" is all but required. These are the very basics of information theory for large nonlinear algorithms, I'm not sure why it's not considered the basics! There are a lot more interesting things beyond this to talk about w.r.t. neural network capacity, certain information theoretic guarantees about the process and (within some rough perspective/metric) capacity of neural network training, as well as some of those dynamics as well! Some days it feels like in a world where calculus is available to us, we spend all our spare time debating exclusively whether Riemannian sums are real or not, and ignoring everything else beyond that. It to me is a very bizzare form of bike-shedding! Come on, y'all, we got much more interesting (and important, I might add! ;P) things to talk about!

Jensson 3y ago
> Technically, under a compressive regime in the limit, LLMs necessarily need to encode an implicit world model, as the text input into is sampled from said world model. This is backwards reasoning, we all know it would be better with a good world model but that doesn't mean that it actually created a good world model. Understanding how to go from here to actually making it get a good world model requires these discussions, it isn't a waste of time unless you think that further progress is impossible.
- tbalsam 3y ago
  The reasoning in the post is not backwards reasoning, it is literally the information theory of compression, described in human-friendly terms! But I must add the caveat that the original post is really only saying that it's rather trivial I think that LLMs are learning _some_ world model, not that they learn it very well. But that this must happen in the limit! :D As far as the quality of the world model goes, I think is important. What I'm focusing on in that post at least , I thinks, is that people debate whether or not LLMs even encode a world model, which is such a silly thing to me. Like you brought up, I agree that discussions about the quality of said world model are interesting, and how far whatever [trained XYZ model is] in that rotation towards learning said world model is. I agree that there is a lot of very interesting discussion there! One rough proxy is to measure the different types of variance within the noise of the model, the more high-level and believable it is, the higher-level the representations in the model. But, of course, this is just a rough human friendly proxy. Actually measuring this, maybe is quite difficult! D'''':

Jensson 3y ago

> Technically, under a compressive regime in the limit, LLMs necessarily need to encode an implicit world model, as the text input into is sampled from said world model. This is backwards reasoning, we all know it would be better with a good world model but that doesn't mean that it actually created a good world model. Understanding how to go from here to actually making it get a good world model requires these discussions, it isn't a waste of time unless you think that further progress is impossible.

tbalsam 3y ago
The reasoning in the post is not backwards reasoning, it is literally the information theory of compression, described in human-friendly terms! But I must add the caveat that the original post is really only saying that it's rather trivial I think that LLMs are learning _some_ world model, not that they learn it very well. But that this must happen in the limit! :D As far as the quality of the world model goes, I think is important. What I'm focusing on in that post at least , I thinks, is that people debate whether or not LLMs even encode a world model, which is such a silly thing to me. Like you brought up, I agree that discussions about the quality of said world model are interesting, and how far whatever [trained XYZ model is] in that rotation towards learning said world model is. I agree that there is a lot of very interesting discussion there! One rough proxy is to measure the different types of variance within the noise of the model, the more high-level and believable it is, the higher-level the representations in the model. But, of course, this is just a rough human friendly proxy. Actually measuring this, maybe is quite difficult! D'''':