6 ms·

>This is not a philosophical matter. We know that "statistical learning" which is nothing more than a "correlation maximisation objective" over non-phenomenolog

>This is not a philosophical matter. We know that "statistical learning" which is nothing more than a "correlation maximisation objective" over non-phenomenological, non-causal, non-physical data produces approximate associative models of those target domains -- that have little use beyond "replaying those associations". Why do you think the data LLMs are trained on are non-causal? Lets take causation as asymmetric correlation. That is, (A,B) present in the training data does not imply (B,A) presence. But of course human text is asymmetric in this manner and LLMs will pick up on this asymmetry. You might say that causation isn't merely about asymmetric correlation, but that of the former determining the latter. But this isn't something we observe from nature, it is an explanatory posit that humans have landed on in service to modelling the world. So causation is intrinsically explanatory, and explanation is intrinsically causal. The question is, does an LLM in the course of modelling asymmetric correlations, develop something analogous to an explanatory model. I think so, in the sense that a good statistical model will intrinsically capture explanatory relations. Cashing out explanation and explanatory model isn't easy. But as a first pass I can say that explanatory models capture intrinsic regularity of a target system such that the model has an analogical relationship with internal mechanisms in the target system. This means that certain transformations applied to the target system has a corresponding transformation in the model that identifies the same outcome. If we view phenomena in terms of mechanistic levels with the extrinsic observable properties as the top level and the internal mechanisms as lower levels, an explanatory model will model some lower mechanistic level and recover properties of the top level. But this is in the solution space of good models of statistical regularity of an external system. To maximally predict the next token in a sequence just requires a model of the process that generates that sequence.

mjburgess 4y ago
Well this is a good reply, but it's mistaken. The conditional probability: P(x[0]| x[-1], x[-2], x[-3] ...) is not the same as, P(x[0] | x[-1], x[-2], ... -> x[0]) Where `->` says we select only those cases where x[-1],... brought-about x[0]. To see why this is the case, suppose we do have a god's eye-view of all of spacetime. P(A|B) always selects for all instances where B follows A. P(A| B -> A) selects only those instances where B's following A was caused by A. Eg., P(ShoesWet | Raining) is very different from P(ShoesWet | Raining -> ShoesWet) in the former case the two events have, in general, nothing to do with each other. To select "Raining -> ShoesWet" even with a gods-eye-view we need more than statistics... since those events which count as "Rain -> ShoeWet" have to be selected on a non-statistical basis. For the athelete catching a ball, or the scientist designing the experiment, we're interested only in those causal cases. For sure P(A|B) is a (approximate, statistical) model of P(A| B->A) -- but it's a very restricted, limited model. The athlete needs to estimate P(ball-stops | catch -> ball-stops) NOT P(ball-stops | catch) which is just any case of the ball-stopping given any case of catching.
- YeGoblynQueenne 4y ago
  Wow, that's a nice way to put it. I haven't seen that P(A|B -> A) notation before. Where does it come from? But I think the OP is arguing, essentially, that P(A|B -> A) is only an interpretation of P(A|B) that we have chosen, among the many possible interpretations of P(A|B). Which I think evokes the problem of induction. How do we know that P(A| B -> A) when all we can observe ourselves is P(A|B)?
  - mjburgess 4y ago
    > when all we can observe ourselves is P(A|B)? No, we actually observe P(A | B -> A) where `B` is our body and `A` is some action we take on the world. Hume was WRONG. Very wrong. Statistical AI has the problem of induction; we have bodies, so we do not. ---- As for notation, I'm riffing of Judea Pearl's do notation. He'd say, P(A|do(B)) but his `do` operator is slightly more general Google: do-operator, causal analysis, judea pearl, etc.
    - YeGoblynQueenne 4y ago
      Ah, I thought it might be something to do with Judea Pearl. >> Hume was WRONG. Very wrong. Oh boy :) I can see what you're saying about having bodies, but bodies are very limited things and that's just making Hume's point. We can only know so much by experiencing it with our bodies. We've learned a lot more about the world, and its foundations, thanks to our ability to draw inferences without having to engage our bodies. For example, all of mathematics, including logic that studies inference, is "things we do without having to engage our bodies". And those very things have shown us the limits of our own abilities, or at least our ability to create formal systems that can describe the world in its entirety. They have shown us the limits of our ability for inductive inference (and in a very concrete manner - see Mark E. Gold's Language Identification in the Limit). Machine learning systems are more limited than ourselves, that's right. And that's because we have created them, and we are limited beings that cannot know the entirety of the world just by looking at it, or reasoning about it.
      - 4y ago
YeGoblynQueenne 4y ago
I very disagree but have an upvote for a well-argued comment. >> The question is, does an LLM in the course of modelling asymmetric correlations, develop something analogous to an explanatory model. I think so, in the sense that a good statistical model will intrinsically capture explanatory relations. A statistical model may "capture" explanatory relations, but can it use them? A data scientist showing a plot to a bunch of people is explaining something using a statistical model, so obviously the statistical model has some explanatory power. But it's the data scientist that is using the model as an explanation. I think the discussion is whether a statistical model can exist that doesn't just "capture" an explanation, but can also make use of that explanation like a human would, for example as background knowledge to build new explanations. That seems very far fetched: a statistical model that doesn't just model, but also introspects and has agency. Anyway I find it very hard to think of language models as explanatory models. They're predictive models, they are black boxes, they model language, but what do they explain? And to whom? The big debate is that (allegedly) "we don't understand language models" in the first place. We have a giant corpus of incomprehensible data; we train a giant black box model on it; now we have a giant incomprehensible model of the data. What did we explain? >> But this is in the solution space of good models of statistical regularity of an external system. To maximally predict the next token in a sequence just requires a model of the process that generates that sequence. Let's call that model M* for clarity. The search space of models, let's call it S. There are any number of models in S that can generate many of the same sequences as M* without being M*. The question is, and has always been, in machine learning, how do we find M* in S, without being distracted by M_1, M_2, M_3, ..., ... that are not M*. Given that we have a very limited way to test the capabilities of models, and that models are getting bigger and bigger (in machine learning anyway) which makes it harder and harder to get a good idea of what, exactly, they are modelling, how can we say which model we got a hold of?
- hackinthebochs 4y ago
  >A statistical model may "capture" explanatory relations, but can it use them? That's the beauty of autoregressive training, the model is rewarded for capturing and utilizing explanatory relations because they have an outsized effect on prediction. It's the difference between frequency counting while taking the past context as an opaque unit vs decomposing the past context and leveraging relevant tokens for generation while ignoring irrelevant ones. Self-attention does this by searching over all pairs of tokens in the context window for relevant associations. Induction heads[1] are a fully worked out example of this and help explain in-context learning in LLMs. >Anyway I find it very hard to think of language models as explanatory models. They're predictive models, they are black boxes, they model language, but what do they explain? And to whom? The model encodes explanatory relationships of phenomena in the world and it uses these relationships to successfully generalize its generation out-of-distribution. Basically, these models genuinely understand some things about the world. LLMs exhibit linguistic competence as it engages with subject matter to accurately respond to unseen variations in prompts of that subject matter. At least in some cases. I argue this point in some detail here[2]. >how can we say which model we got a hold of? More sophisticated tests, ideally that can isolate exactly what was in the training data in comparison to what was generated. I think the example of the wide variety of poetry these models generate should strongly raise one's credence that they capture a sufficiently accurate model of poetry. I go into detail on this example in the link I mentioned. Aside from that, various ways of testing in-context learning can do a lot of work here[3]. [1] https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html https://transformer-circuits.pub/2022/in-context-learning-an... [2] https://www.reddit.com/r/naturalism/comments/1236vzf/ https://www.reddit.com/r/naturalism/comments/1236vzf/ [3] https://twitter.com/leopoldasch/status/1638848881558704129 https://twitter.com/leopoldasch/status/1638848881558704129

mjburgess 4y ago

Well this is a good reply, but it's mistaken. The conditional probability: P(x[0]| x[-1], x[-2], x[-3] ...) is not the same as, P(x[0] | x[-1], x[-2], ... -> x[0]) Where `->` says we select only those cases where x[-1],... brought-about x[0]. To see why this is the case, suppose we do have a god's eye-view of all of spacetime. P(A|B) always selects for all instances where B follows A. P(A| B -> A) selects only those instances where B's following A was caused by A. Eg., P(ShoesWet | Raining) is very different from P(ShoesWet | Raining -> ShoesWet) in the former case the two events have, in general, nothing to do with each other. To select "Raining -> ShoesWet" even with a gods-eye-view we need more than statistics... since those events which count as "Rain -> ShoeWet" have to be selected on a non-statistical basis. For the athelete catching a ball, or the scientist designing the experiment, we're interested only in those causal cases. For sure P(A|B) is a (approximate, statistical) model of P(A| B->A) -- but it's a very restricted, limited model. The athlete needs to estimate P(ball-stops | catch -> ball-stops) NOT P(ball-stops | catch) which is just any case of the ball-stopping given any case of catching.

YeGoblynQueenne 4y ago
Wow, that's a nice way to put it. I haven't seen that P(A|B -> A) notation before. Where does it come from? But I think the OP is arguing, essentially, that P(A|B -> A) is only an interpretation of P(A|B) that we have chosen, among the many possible interpretations of P(A|B). Which I think evokes the problem of induction. How do we know that P(A| B -> A) when all we can observe ourselves is P(A|B)?
- mjburgess 4y ago
  > when all we can observe ourselves is P(A|B)? No, we actually observe P(A | B -> A) where `B` is our body and `A` is some action we take on the world. Hume was WRONG. Very wrong. Statistical AI has the problem of induction; we have bodies, so we do not. ---- As for notation, I'm riffing of Judea Pearl's do notation. He'd say, P(A|do(B)) but his `do` operator is slightly more general Google: do-operator, causal analysis, judea pearl, etc.
  - YeGoblynQueenne 4y ago
    Ah, I thought it might be something to do with Judea Pearl. >> Hume was WRONG. Very wrong. Oh boy :) I can see what you're saying about having bodies, but bodies are very limited things and that's just making Hume's point. We can only know so much by experiencing it with our bodies. We've learned a lot more about the world, and its foundations, thanks to our ability to draw inferences without having to engage our bodies. For example, all of mathematics, including logic that studies inference, is "things we do without having to engage our bodies". And those very things have shown us the limits of our own abilities, or at least our ability to create formal systems that can describe the world in its entirety. They have shown us the limits of our ability for inductive inference (and in a very concrete manner - see Mark E. Gold's Language Identification in the Limit). Machine learning systems are more limited than ourselves, that's right. And that's because we have created them, and we are limited beings that cannot know the entirety of the world just by looking at it, or reasoning about it.
    - 4y ago

YeGoblynQueenne 4y ago

I very disagree but have an upvote for a well-argued comment. >> The question is, does an LLM in the course of modelling asymmetric correlations, develop something analogous to an explanatory model. I think so, in the sense that a good statistical model will intrinsically capture explanatory relations. A statistical model may "capture" explanatory relations, but can it use them? A data scientist showing a plot to a bunch of people is explaining something using a statistical model, so obviously the statistical model has some explanatory power. But it's the data scientist that is using the model as an explanation. I think the discussion is whether a statistical model can exist that doesn't just "capture" an explanation, but can also make use of that explanation like a human would, for example as background knowledge to build new explanations. That seems very far fetched: a statistical model that doesn't just model, but also introspects and has agency. Anyway I find it very hard to think of language models as explanatory models. They're predictive models, they are black boxes, they model language, but what do they explain? And to whom? The big debate is that (allegedly) "we don't understand language models" in the first place. We have a giant corpus of incomprehensible data; we train a giant black box model on it; now we have a giant incomprehensible model of the data. What did we explain? >> But this is in the solution space of good models of statistical regularity of an external system. To maximally predict the next token in a sequence just requires a model of the process that generates that sequence. Let's call that model M* for clarity. The search space of models, let's call it S. There are any number of models in S that can generate many of the same sequences as M* without being M*. The question is, and has always been, in machine learning, how do we find M* in S, without being distracted by M_1, M_2, M_3, ..., ... that are not M*. Given that we have a very limited way to test the capabilities of models, and that models are getting bigger and bigger (in machine learning anyway) which makes it harder and harder to get a good idea of what, exactly, they are modelling, how can we say which model we got a hold of?

hackinthebochs 4y ago
>A statistical model may "capture" explanatory relations, but can it use them? That's the beauty of autoregressive training, the model is rewarded for capturing and utilizing explanatory relations because they have an outsized effect on prediction. It's the difference between frequency counting while taking the past context as an opaque unit vs decomposing the past context and leveraging relevant tokens for generation while ignoring irrelevant ones. Self-attention does this by searching over all pairs of tokens in the context window for relevant associations. Induction heads[1] are a fully worked out example of this and help explain in-context learning in LLMs. >Anyway I find it very hard to think of language models as explanatory models. They're predictive models, they are black boxes, they model language, but what do they explain? And to whom? The model encodes explanatory relationships of phenomena in the world and it uses these relationships to successfully generalize its generation out-of-distribution. Basically, these models genuinely understand some things about the world. LLMs exhibit linguistic competence as it engages with subject matter to accurately respond to unseen variations in prompts of that subject matter. At least in some cases. I argue this point in some detail here[2]. >how can we say which model we got a hold of? More sophisticated tests, ideally that can isolate exactly what was in the training data in comparison to what was generated. I think the example of the wide variety of poetry these models generate should strongly raise one's credence that they capture a sufficiently accurate model of poetry. I go into detail on this example in the link I mentioned. Aside from that, various ways of testing in-context learning can do a lot of work here[3]. [1] https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html https://transformer-circuits.pub/2022/in-context-learning-an... [2] https://www.reddit.com/r/naturalism/comments/1236vzf/ https://www.reddit.com/r/naturalism/comments/1236vzf/ [3] https://twitter.com/leopoldasch/status/1638848881558704129 https://twitter.com/leopoldasch/status/1638848881558704129