6 ms·

To paraphrase what you're saying: an AI can't reason, because it is built to stochastically predicts tokens, which is not reasoning and which is different from

by lukev 3y ago

To paraphrase what you're saying: an AI can't reason, because it is built to stochastically predicts tokens, which is not reasoning and which is different from the activity of reasoning. Which I agree with. But also, by observation, it can (at least some of the time) emit token sequences that emulate reasoning (at least on some simple tasks.) So perhaps it can reason in the same way a submarine can swim.

hackinthebochs 3y ago
>an AI can't reason, because it is built to stochastically predicts tokens, which is not reasoning and which is different from the activity of reasoning. A feature indispensable in the generation of a sequence is recovered in the limit of predicting the sequence. So prediction does not exclude higher level cognitive processes. Transformers being universal sequence2sequence modelers give reason to believe they can reach the limit.
- mjburgess 3y ago
  There are an infinite number of models which generate the same exact (infinite) sequence. No model is "recovered in the limit". And, more severely, we're blind to the future. So of the infinite models of all of history, we're not even interested in the ones which are maximally retrospectively predictive. Almost all of those are maximally non-predictive of the future, and indeed, much worse than the ones which fail to predict the past well. So your 'recovering in the limit' alas is a dangerous kind of pseudoscience: fitting the past. We want models which are wrong for the right reasons. Not models which are right for the wrong ones. The latter fail catastrophically. The models we require enable us to simulate unrealised futures; the causal, abductive models of science (, say).
  - hackinthebochs 3y ago
    >There are an infinite number of models which generate the same exact (infinite) sequence. Only if there is no constraint on the model. But the fixed number of parameters, and the inductive bias of attention limits the space of models that are learnable. In the limit of infinite data but finite capacity for memorization, the only available solution will be to recover the information dynamic of the generating process. >So of the infinite models of all of history, we're not even interested in the ones which are maximally retrospectively predictive. Presumably the same processes that operated in the past will operate in the future. So accurately modelling the past is certainly very informative for the future.
    - mjburgess 3y ago
      > Only if there is no constraint on the model. But the fixed number of parameters Nope. cf. the under-determination of evidence by theory. > So accurately modelling the past is certainly very informative for the future. Nope. cf. we don't have infinite measures on infinite events in the past. Consider that any given measure of an event, say M(E) is really a measure of a near infinite number of causes, say C1...n. Now, how many independent measures of these do we have? A handful. so we really have, M(E|Controlling(Handful C1..n))) do we want to model that?!? No.. that's insane. That's superstition. That's what all of science stands against. Here's what we do: we build models not fit to any data. We build models that can generate the data we observe, but we build them by familiarity with reality *NOT* with the measures M. How do we do that? Many ways, but the body is a key component of the answer. We have practically certain causal models of the body. ConceptCup = MySensoryMotorActions(RealCups) I_Touch = MySensoryMotorActions(Me) P(I_Touch_Cup|Cup_Exists, ConceptCut) ~= 1 P(I_Touch_Cup|-Cup_Exists, ConceptCup) ~= 0 by iterating through such sensory-motor actions we can find concepts which produce these discrete splits. We call these concepts 'objects'. And from these we assemble models of reality whose shadow is our measures. We handle the cup, and by handling the cup can imagine the cup, and by imagine the cup can generate measures of the cup. You cannot reverse from a shadow of a vase to the clay vase itself: there are an infinite number, given abribtarily infinite parameters. The task is to find the right parameterisation; not to suppose that any given one has one solution (it doesnt, so a fool's hope anyway). Such a condition is just a formalisation of supersition: my model is decided by the data; my parameterisation is 'free'. My model of the world is cooincidence. Indeed, the truth is, in a sense, only a single parameter.
      - hackinthebochs 3y ago
        >Nope. cf. the under-determination of evidence by theory. You mean the underdetermination of theory by evidence? This isn't really relevant. Given no prior information and no constraints on the model, the theory is underdetermined. But given some strong constraints, there will be one model that best explains the data. Of course a model can't deal with an arbitrary distribution shift, but no one expects a model to be insensitive to a change in the underlying data. The question is whether the constraints of natural systems and an LLM with a given inductive bias and finite capacity are sufficient constraints. It's not totally clear that it is or isn't, but this isn't decided by the underdetermination principle. >Nope. cf. we don't have infinite measures on infinite events in the past. I don't see how this is relevant to the point it is in response to. >Such a condition is just a formalisation of supersition: my model is decided by the data; my parameterisation is 'free'. My model of the world is cooincidence. Everything is data in the end; your interactions with the world by touch are just data. Our brains have an inductive bias towards certain interpretations of certain data, and these interpretations tend to promote proliferation in organisms. But LLMs have an inductive bias as well, one that allows them to search for circuits that generate training data. LLMs tend to land on interesting models that result in non-trivial generalization abilities in some contexts. This ability goes beyond just frequency modeling. Of course LLMs are limited to what it can know of the world through its data paradigm. But so are we. Such a limit in itself doesn't imply an in principle limit to modeling/understanding in LLMs.
mjburgess 3y ago
So what's happening is this: valid arguments have a structure which can be modelled statistically. But validity isn't a statistical notion: P(-A|A) is zero. Since P(Premise|Contradiction) is frequently above zero in LLMs they engage in randomly "irrational" reasoning. That's what makes them particularly unreliable. The reason for any given P(premise|contradiction) having any given freq is just that "its that freq in the corpus". This is a pretty insane reason, and so incomprehensibly insane that many -- i guess -- cannot fathom how an LLM can appear to reason without being able to. I suppose there's a sort of truman show effect: the reality of the underlying mechanism is so outside anything most people could analogise to, they fail to be able to see the trick taking place. People talk about "human failures" but we never ascribe confidences to propositions based on their frequency in a text corpus; and the apparent "Reasoning" which arises out of this is incomprehensible. That is without some systematic training in applied stats so you can set up the model and reason about it regardless of its outputs -- which are irrelevant to its mechanism
- lukev 3y ago
  It's true. Validity isn't a statistical notion, and that's why I agree with you that LLMs can't "reason" if we're being precise with language. Just like they can't "know" or "believe" or basically any other verb we use in epistemology. But the argument is that by dint of repetition, the models actually encode some of the structure of basic logical primitives: syllogisms, entailment, modus ponens/tollens, etc. This then weights their output so that (for simple enough stuff) they're more likely to emit outputs that are logically sound, than outputs that aren't logically sound. Indeed, this has to be the case, or they couldn't maintain any level of coherency in their output at all (which, like it or not, they can.) Like you, I'm not comfortable calling this reasoning. But it's also something that is not 100% entirely unlike reasoning, either, at least in terms of output.
  - mjburgess 3y ago
    The heart of science is distinguishing illusion from reality: from measures of events from models of events. Shadows from what casts them. The spherical shadow of an object here alas, isnt caused by a spherical object. It's a spikey deformed object whose shadow is spherical with the right prompt. This is easy to show. Engineers are people who put on light shows. Engineers make the magic lanterns --- theatres of people who believe there's another world in front of them. Scientists are interested in the quality of the film grain. The spikiness of the suppose sphere. Or, the failure of an LLM's ability to "reason"

by lukev 3y ago

hackinthebochs 3y ago

>an AI can't reason, because it is built to stochastically predicts tokens, which is not reasoning and which is different from the activity of reasoning. A feature indispensable in the generation of a sequence is recovered in the limit of predicting the sequence. So prediction does not exclude higher level cognitive processes. Transformers being universal sequence2sequence modelers give reason to believe they can reach the limit.

mjburgess 3y ago
There are an infinite number of models which generate the same exact (infinite) sequence. No model is "recovered in the limit". And, more severely, we're blind to the future. So of the infinite models of all of history, we're not even interested in the ones which are maximally retrospectively predictive. Almost all of those are maximally non-predictive of the future, and indeed, much worse than the ones which fail to predict the past well. So your 'recovering in the limit' alas is a dangerous kind of pseudoscience: fitting the past. We want models which are wrong for the right reasons. Not models which are right for the wrong ones. The latter fail catastrophically. The models we require enable us to simulate unrealised futures; the causal, abductive models of science (, say).
- hackinthebochs 3y ago
  >There are an infinite number of models which generate the same exact (infinite) sequence. Only if there is no constraint on the model. But the fixed number of parameters, and the inductive bias of attention limits the space of models that are learnable. In the limit of infinite data but finite capacity for memorization, the only available solution will be to recover the information dynamic of the generating process. >So of the infinite models of all of history, we're not even interested in the ones which are maximally retrospectively predictive. Presumably the same processes that operated in the past will operate in the future. So accurately modelling the past is certainly very informative for the future.
  - mjburgess 3y ago
    > Only if there is no constraint on the model. But the fixed number of parameters Nope. cf. the under-determination of evidence by theory. > So accurately modelling the past is certainly very informative for the future. Nope. cf. we don't have infinite measures on infinite events in the past. Consider that any given measure of an event, say M(E) is really a measure of a near infinite number of causes, say C1...n. Now, how many independent measures of these do we have? A handful. so we really have, M(E|Controlling(Handful C1..n))) do we want to model that?!? No.. that's insane. That's superstition. That's what all of science stands against. Here's what we do: we build models not fit to any data. We build models that can generate the data we observe, but we build them by familiarity with reality *NOT* with the measures M. How do we do that? Many ways, but the body is a key component of the answer. We have practically certain causal models of the body. ConceptCup = MySensoryMotorActions(RealCups) I_Touch = MySensoryMotorActions(Me) P(I_Touch_Cup|Cup_Exists, ConceptCut) ~= 1 P(I_Touch_Cup|-Cup_Exists, ConceptCup) ~= 0 by iterating through such sensory-motor actions we can find concepts which produce these discrete splits. We call these concepts 'objects'. And from these we assemble models of reality whose shadow is our measures. We handle the cup, and by handling the cup can imagine the cup, and by imagine the cup can generate measures of the cup. You cannot reverse from a shadow of a vase to the clay vase itself: there are an infinite number, given abribtarily infinite parameters. The task is to find the right parameterisation; not to suppose that any given one has one solution (it doesnt, so a fool's hope anyway). Such a condition is just a formalisation of supersition: my model is decided by the data; my parameterisation is 'free'. My model of the world is cooincidence. Indeed, the truth is, in a sense, only a single parameter.
    - hackinthebochs 3y ago
      >Nope. cf. the under-determination of evidence by theory. You mean the underdetermination of theory by evidence? This isn't really relevant. Given no prior information and no constraints on the model, the theory is underdetermined. But given some strong constraints, there will be one model that best explains the data. Of course a model can't deal with an arbitrary distribution shift, but no one expects a model to be insensitive to a change in the underlying data. The question is whether the constraints of natural systems and an LLM with a given inductive bias and finite capacity are sufficient constraints. It's not totally clear that it is or isn't, but this isn't decided by the underdetermination principle. >Nope. cf. we don't have infinite measures on infinite events in the past. I don't see how this is relevant to the point it is in response to. >Such a condition is just a formalisation of supersition: my model is decided by the data; my parameterisation is 'free'. My model of the world is cooincidence. Everything is data in the end; your interactions with the world by touch are just data. Our brains have an inductive bias towards certain interpretations of certain data, and these interpretations tend to promote proliferation in organisms. But LLMs have an inductive bias as well, one that allows them to search for circuits that generate training data. LLMs tend to land on interesting models that result in non-trivial generalization abilities in some contexts. This ability goes beyond just frequency modeling. Of course LLMs are limited to what it can know of the world through its data paradigm. But so are we. Such a limit in itself doesn't imply an in principle limit to modeling/understanding in LLMs.

mjburgess 3y ago

So what's happening is this: valid arguments have a structure which can be modelled statistically. But validity isn't a statistical notion: P(-A|A) is zero. Since P(Premise|Contradiction) is frequently above zero in LLMs they engage in randomly "irrational" reasoning. That's what makes them particularly unreliable. The reason for any given P(premise|contradiction) having any given freq is just that "its that freq in the corpus". This is a pretty insane reason, and so incomprehensibly insane that many -- i guess -- cannot fathom how an LLM can appear to reason without being able to. I suppose there's a sort of truman show effect: the reality of the underlying mechanism is so outside anything most people could analogise to, they fail to be able to see the trick taking place. People talk about "human failures" but we never ascribe confidences to propositions based on their frequency in a text corpus; and the apparent "Reasoning" which arises out of this is incomprehensible. That is without some systematic training in applied stats so you can set up the model and reason about it regardless of its outputs -- which are irrelevant to its mechanism

lukev 3y ago
It's true. Validity isn't a statistical notion, and that's why I agree with you that LLMs can't "reason" if we're being precise with language. Just like they can't "know" or "believe" or basically any other verb we use in epistemology. But the argument is that by dint of repetition, the models actually encode some of the structure of basic logical primitives: syllogisms, entailment, modus ponens/tollens, etc. This then weights their output so that (for simple enough stuff) they're more likely to emit outputs that are logically sound, than outputs that aren't logically sound. Indeed, this has to be the case, or they couldn't maintain any level of coherency in their output at all (which, like it or not, they can.) Like you, I'm not comfortable calling this reasoning. But it's also something that is not 100% entirely unlike reasoning, either, at least in terms of output.
- mjburgess 3y ago
  The heart of science is distinguishing illusion from reality: from measures of events from models of events. Shadows from what casts them. The spherical shadow of an object here alas, isnt caused by a spherical object. It's a spikey deformed object whose shadow is spherical with the right prompt. This is easy to show. Engineers are people who put on light shows. Engineers make the magic lanterns --- theatres of people who believe there's another world in front of them. Scientists are interested in the quality of the film grain. The spikiness of the suppose sphere. Or, the failure of an LLM's ability to "reason"