4 ms·

>an AI can't reason, because it is built to stochastically predicts tokens, which is not reasoning and which is different from the activity of reasoning. A fea

>an AI can't reason, because it is built to stochastically predicts tokens, which is not reasoning and which is different from the activity of reasoning. A feature indispensable in the generation of a sequence is recovered in the limit of predicting the sequence. So prediction does not exclude higher level cognitive processes. Transformers being universal sequence2sequence modelers give reason to believe they can reach the limit.

mjburgess 3y ago
There are an infinite number of models which generate the same exact (infinite) sequence. No model is "recovered in the limit". And, more severely, we're blind to the future. So of the infinite models of all of history, we're not even interested in the ones which are maximally retrospectively predictive. Almost all of those are maximally non-predictive of the future, and indeed, much worse than the ones which fail to predict the past well. So your 'recovering in the limit' alas is a dangerous kind of pseudoscience: fitting the past. We want models which are wrong for the right reasons. Not models which are right for the wrong ones. The latter fail catastrophically. The models we require enable us to simulate unrealised futures; the causal, abductive models of science (, say).
- hackinthebochs 3y ago
  >There are an infinite number of models which generate the same exact (infinite) sequence. Only if there is no constraint on the model. But the fixed number of parameters, and the inductive bias of attention limits the space of models that are learnable. In the limit of infinite data but finite capacity for memorization, the only available solution will be to recover the information dynamic of the generating process. >So of the infinite models of all of history, we're not even interested in the ones which are maximally retrospectively predictive. Presumably the same processes that operated in the past will operate in the future. So accurately modelling the past is certainly very informative for the future.
  - mjburgess 3y ago
    > Only if there is no constraint on the model. But the fixed number of parameters Nope. cf. the under-determination of evidence by theory. > So accurately modelling the past is certainly very informative for the future. Nope. cf. we don't have infinite measures on infinite events in the past. Consider that any given measure of an event, say M(E) is really a measure of a near infinite number of causes, say C1...n. Now, how many independent measures of these do we have? A handful. so we really have, M(E|Controlling(Handful C1..n))) do we want to model that?!? No.. that's insane. That's superstition. That's what all of science stands against. Here's what we do: we build models not fit to any data. We build models that can generate the data we observe, but we build them by familiarity with reality *NOT* with the measures M. How do we do that? Many ways, but the body is a key component of the answer. We have practically certain causal models of the body. ConceptCup = MySensoryMotorActions(RealCups) I_Touch = MySensoryMotorActions(Me) P(I_Touch_Cup|Cup_Exists, ConceptCut) ~= 1 P(I_Touch_Cup|-Cup_Exists, ConceptCup) ~= 0 by iterating through such sensory-motor actions we can find concepts which produce these discrete splits. We call these concepts 'objects'. And from these we assemble models of reality whose shadow is our measures. We handle the cup, and by handling the cup can imagine the cup, and by imagine the cup can generate measures of the cup. You cannot reverse from a shadow of a vase to the clay vase itself: there are an infinite number, given abribtarily infinite parameters. The task is to find the right parameterisation; not to suppose that any given one has one solution (it doesnt, so a fool's hope anyway). Such a condition is just a formalisation of supersition: my model is decided by the data; my parameterisation is 'free'. My model of the world is cooincidence. Indeed, the truth is, in a sense, only a single parameter.
    - hackinthebochs 3y ago
      >Nope. cf. the under-determination of evidence by theory. You mean the underdetermination of theory by evidence? This isn't really relevant. Given no prior information and no constraints on the model, the theory is underdetermined. But given some strong constraints, there will be one model that best explains the data. Of course a model can't deal with an arbitrary distribution shift, but no one expects a model to be insensitive to a change in the underlying data. The question is whether the constraints of natural systems and an LLM with a given inductive bias and finite capacity are sufficient constraints. It's not totally clear that it is or isn't, but this isn't decided by the underdetermination principle. >Nope. cf. we don't have infinite measures on infinite events in the past. I don't see how this is relevant to the point it is in response to. >Such a condition is just a formalisation of supersition: my model is decided by the data; my parameterisation is 'free'. My model of the world is cooincidence. Everything is data in the end; your interactions with the world by touch are just data. Our brains have an inductive bias towards certain interpretations of certain data, and these interpretations tend to promote proliferation in organisms. But LLMs have an inductive bias as well, one that allows them to search for circuits that generate training data. LLMs tend to land on interesting models that result in non-trivial generalization abilities in some contexts. This ability goes beyond just frequency modeling. Of course LLMs are limited to what it can know of the world through its data paradigm. But so are we. Such a limit in itself doesn't imply an in principle limit to modeling/understanding in LLMs.

mjburgess 3y ago

There are an infinite number of models which generate the same exact (infinite) sequence. No model is "recovered in the limit". And, more severely, we're blind to the future. So of the infinite models of all of history, we're not even interested in the ones which are maximally retrospectively predictive. Almost all of those are maximally non-predictive of the future, and indeed, much worse than the ones which fail to predict the past well. So your 'recovering in the limit' alas is a dangerous kind of pseudoscience: fitting the past. We want models which are wrong for the right reasons. Not models which are right for the wrong ones. The latter fail catastrophically. The models we require enable us to simulate unrealised futures; the causal, abductive models of science (, say).

hackinthebochs 3y ago
>There are an infinite number of models which generate the same exact (infinite) sequence. Only if there is no constraint on the model. But the fixed number of parameters, and the inductive bias of attention limits the space of models that are learnable. In the limit of infinite data but finite capacity for memorization, the only available solution will be to recover the information dynamic of the generating process. >So of the infinite models of all of history, we're not even interested in the ones which are maximally retrospectively predictive. Presumably the same processes that operated in the past will operate in the future. So accurately modelling the past is certainly very informative for the future.
- mjburgess 3y ago
  > Only if there is no constraint on the model. But the fixed number of parameters Nope. cf. the under-determination of evidence by theory. > So accurately modelling the past is certainly very informative for the future. Nope. cf. we don't have infinite measures on infinite events in the past. Consider that any given measure of an event, say M(E) is really a measure of a near infinite number of causes, say C1...n. Now, how many independent measures of these do we have? A handful. so we really have, M(E|Controlling(Handful C1..n))) do we want to model that?!? No.. that's insane. That's superstition. That's what all of science stands against. Here's what we do: we build models not fit to any data. We build models that can generate the data we observe, but we build them by familiarity with reality *NOT* with the measures M. How do we do that? Many ways, but the body is a key component of the answer. We have practically certain causal models of the body. ConceptCup = MySensoryMotorActions(RealCups) I_Touch = MySensoryMotorActions(Me) P(I_Touch_Cup|Cup_Exists, ConceptCut) ~= 1 P(I_Touch_Cup|-Cup_Exists, ConceptCup) ~= 0 by iterating through such sensory-motor actions we can find concepts which produce these discrete splits. We call these concepts 'objects'. And from these we assemble models of reality whose shadow is our measures. We handle the cup, and by handling the cup can imagine the cup, and by imagine the cup can generate measures of the cup. You cannot reverse from a shadow of a vase to the clay vase itself: there are an infinite number, given abribtarily infinite parameters. The task is to find the right parameterisation; not to suppose that any given one has one solution (it doesnt, so a fool's hope anyway). Such a condition is just a formalisation of supersition: my model is decided by the data; my parameterisation is 'free'. My model of the world is cooincidence. Indeed, the truth is, in a sense, only a single parameter.
  - hackinthebochs 3y ago
    >Nope. cf. the under-determination of evidence by theory. You mean the underdetermination of theory by evidence? This isn't really relevant. Given no prior information and no constraints on the model, the theory is underdetermined. But given some strong constraints, there will be one model that best explains the data. Of course a model can't deal with an arbitrary distribution shift, but no one expects a model to be insensitive to a change in the underlying data. The question is whether the constraints of natural systems and an LLM with a given inductive bias and finite capacity are sufficient constraints. It's not totally clear that it is or isn't, but this isn't decided by the underdetermination principle. >Nope. cf. we don't have infinite measures on infinite events in the past. I don't see how this is relevant to the point it is in response to. >Such a condition is just a formalisation of supersition: my model is decided by the data; my parameterisation is 'free'. My model of the world is cooincidence. Everything is data in the end; your interactions with the world by touch are just data. Our brains have an inductive bias towards certain interpretations of certain data, and these interpretations tend to promote proliferation in organisms. But LLMs have an inductive bias as well, one that allows them to search for circuits that generate training data. LLMs tend to land on interesting models that result in non-trivial generalization abilities in some contexts. This ability goes beyond just frequency modeling. Of course LLMs are limited to what it can know of the world through its data paradigm. But so are we. Such a limit in itself doesn't imply an in principle limit to modeling/understanding in LLMs.