3 ms·

So what's happening is this: valid arguments have a structure which can be modelled statistically. But validity isn't a statistical notion: P(-A|A) is zero. S

So what's happening is this: valid arguments have a structure which can be modelled statistically. But validity isn't a statistical notion: P(-A|A) is zero. Since P(Premise|Contradiction) is frequently above zero in LLMs they engage in randomly "irrational" reasoning. That's what makes them particularly unreliable. The reason for any given P(premise|contradiction) having any given freq is just that "its that freq in the corpus". This is a pretty insane reason, and so incomprehensibly insane that many -- i guess -- cannot fathom how an LLM can appear to reason without being able to. I suppose there's a sort of truman show effect: the reality of the underlying mechanism is so outside anything most people could analogise to, they fail to be able to see the trick taking place. People talk about "human failures" but we never ascribe confidences to propositions based on their frequency in a text corpus; and the apparent "Reasoning" which arises out of this is incomprehensible. That is without some systematic training in applied stats so you can set up the model and reason about it regardless of its outputs -- which are irrelevant to its mechanism

lukev 3y ago
It's true. Validity isn't a statistical notion, and that's why I agree with you that LLMs can't "reason" if we're being precise with language. Just like they can't "know" or "believe" or basically any other verb we use in epistemology. But the argument is that by dint of repetition, the models actually encode some of the structure of basic logical primitives: syllogisms, entailment, modus ponens/tollens, etc. This then weights their output so that (for simple enough stuff) they're more likely to emit outputs that are logically sound, than outputs that aren't logically sound. Indeed, this has to be the case, or they couldn't maintain any level of coherency in their output at all (which, like it or not, they can.) Like you, I'm not comfortable calling this reasoning. But it's also something that is not 100% entirely unlike reasoning, either, at least in terms of output.
- mjburgess 3y ago
  The heart of science is distinguishing illusion from reality: from measures of events from models of events. Shadows from what casts them. The spherical shadow of an object here alas, isnt caused by a spherical object. It's a spikey deformed object whose shadow is spherical with the right prompt. This is easy to show. Engineers are people who put on light shows. Engineers make the magic lanterns --- theatres of people who believe there's another world in front of them. Scientists are interested in the quality of the film grain. The spikiness of the suppose sphere. Or, the failure of an LLM's ability to "reason"
  - Gnarl 3y ago
    Bang on! Standing ovation to both of you.
YeGoblynQueenne 3y ago
Look, of course statistics isn't reasoning because you can't build a proof with statistics (or probabilities). But, we have to wonder: when people say "reasoning", do they really, really mean, drawing inferences from axioms and theorems using some set of inference rules? Or do they just mean that they can ask a question and get an answer that makes sense, back? I certainly think it's the latter. People are imprecise when they speak about reasoning, just as they are imprecise when reasoning. Most people who are going to be using LLMs will not be people looking for precise, correct answers derived by sound inference procedures. Those who need precision will seek it elsewhere, where precision can be obtained. The rest will be happy with "reasoning", quote-unquote. Basically the use case for LLMs reminds me of a couple pieces of work that were published in the past, where people used neural nets to approximate the results of a precise calculation. One team trained a neural net to predict the chaotic motion of three astronomical bodies ("the three body problem"). Another trained a neural net to approximate an automated planner programmed to drive a drone around in a thicket of trees without crashing. In both cases the trained model could obviously only return approximately correct results, and it could only approximate results that had already been calculated classically, but, at least in the second case, if I remember correctly, there was a significant speedup of the operation of the drone, compared with the automated planner- very reasonably so, since the model approximating the planner didn't have to do all the hard work of actually, you know, calculating a plan. My bottom line is that no matter what you (or I, or the article's author, or anyone else) can say or show about the true capabilities of LLMs to reason, people are totally going to use them _as if_ they could reason, and the job of proving that they can't is going to get all that much harder for that experience, misguided as it may be. Ultimately the question of whether LLMs can reason is going to be, outside specialist circles, as relevant as "Does Netflix distribute art?". Sure, it doesn't. But people watch it anyway. Most people seemingly don't need art as much, and they don't need reasoning as much, either. Which is a tragic conclusion, of course. At least some of us are still working on AI whose purpose is to make humans better at thinking, not that takes away their motivation to think.

lukev 3y ago

It's true. Validity isn't a statistical notion, and that's why I agree with you that LLMs can't "reason" if we're being precise with language. Just like they can't "know" or "believe" or basically any other verb we use in epistemology. But the argument is that by dint of repetition, the models actually encode some of the structure of basic logical primitives: syllogisms, entailment, modus ponens/tollens, etc. This then weights their output so that (for simple enough stuff) they're more likely to emit outputs that are logically sound, than outputs that aren't logically sound. Indeed, this has to be the case, or they couldn't maintain any level of coherency in their output at all (which, like it or not, they can.) Like you, I'm not comfortable calling this reasoning. But it's also something that is not 100% entirely unlike reasoning, either, at least in terms of output.

mjburgess 3y ago
The heart of science is distinguishing illusion from reality: from measures of events from models of events. Shadows from what casts them. The spherical shadow of an object here alas, isnt caused by a spherical object. It's a spikey deformed object whose shadow is spherical with the right prompt. This is easy to show. Engineers are people who put on light shows. Engineers make the magic lanterns --- theatres of people who believe there's another world in front of them. Scientists are interested in the quality of the film grain. The spikiness of the suppose sphere. Or, the failure of an LLM's ability to "reason"
- Gnarl 3y ago
  Bang on! Standing ovation to both of you.

YeGoblynQueenne 3y ago

Look, of course statistics isn't reasoning because you can't build a proof with statistics (or probabilities). But, we have to wonder: when people say "reasoning", do they really, really mean, drawing inferences from axioms and theorems using some set of inference rules? Or do they just mean that they can ask a question and get an answer that makes sense, back? I certainly think it's the latter. People are imprecise when they speak about reasoning, just as they are imprecise when reasoning. Most people who are going to be using LLMs will not be people looking for precise, correct answers derived by sound inference procedures. Those who need precision will seek it elsewhere, where precision can be obtained. The rest will be happy with "reasoning", quote-unquote. Basically the use case for LLMs reminds me of a couple pieces of work that were published in the past, where people used neural nets to approximate the results of a precise calculation. One team trained a neural net to predict the chaotic motion of three astronomical bodies ("the three body problem"). Another trained a neural net to approximate an automated planner programmed to drive a drone around in a thicket of trees without crashing. In both cases the trained model could obviously only return approximately correct results, and it could only approximate results that had already been calculated classically, but, at least in the second case, if I remember correctly, there was a significant speedup of the operation of the drone, compared with the automated planner- very reasonably so, since the model approximating the planner didn't have to do all the hard work of actually, you know, calculating a plan. My bottom line is that no matter what you (or I, or the article's author, or anyone else) can say or show about the true capabilities of LLMs to reason, people are totally going to use them _as if_ they could reason, and the job of proving that they can't is going to get all that much harder for that experience, misguided as it may be. Ultimately the question of whether LLMs can reason is going to be, outside specialist circles, as relevant as "Does Netflix distribute art?". Sure, it doesn't. But people watch it anyway. Most people seemingly don't need art as much, and they don't need reasoning as much, either. Which is a tragic conclusion, of course. At least some of us are still working on AI whose purpose is to make humans better at thinking, not that takes away their motivation to think.