4 ms·
The problem with claims like these that models are not doing “actual reasoning” is that they are often hot takes and not thought through very well. For example
by WhitneyLand 2y ago
The problem with claims like these that models are not doing “actual reasoning” is that they are often hot takes and not thought through very well.
For example, since reasoning doesn’t yet have any consensus definition that can be applied as a yes/no test - you have to explain what you specifically mean by it, or else the claim is hollow.
Clarify your definition, give a concrete example under that definition of something that’s your version of true scottsman reasoning and something that’s not, then let’s talk.
- martin-t 2y agoI am not gonna pretend to have one true definition but there are hopefully components of reasoning we can agree on: - The ability to hold a collection of beliefs/statements and perform operations on them such as noticing one contradicts the others. Claude in my example clearly failed this. - The ability to make logical inferences. Since logical steps can be represented by text, current LLMs seem to simulate this on the token stream and seem to have decent results most of the time. True inferences would be representing statements in some abstract form and being able to perform operations on them according to rules. The key here is that once it has a collection of statements and is given ("decides to use") an operation to perform, then it can do this correctly 100% of the time (ignoring hardware errors).
- jryan49 2y agoIs it possible the models do something entirely different? I'm not sure why everyone needs to compare them to human intelligence. It's very obvious llms work nothing like our brains why would the intelligence they exhibit be like ours?
- YeGoblynQueenne 2y agoExplain this to me please: we don't have any consensus definition of _mathematics_ that can be applied as a yes/no test. Does that mean we don't know how to do mathematics, or that we don't know whether something, is, or, more importantly, isn't mathematics? For example, if I throw a bunch of sticks in the air and look at their patterns to divine the future- can I call that "mathematics" just because nobody has a "consensus definition of mathematics that can be applied as a yes/no test"? Can I just call anything I like mathematics and nobody can tell me it's wrong because ... no definition? We, as a civilisation, have studied both formal and informal reasoning since at least a couple thousand years go, starting with Aristotle and his syllogisms (a formalisation of rigorous arguments) and continuing through the years with such figures as Leibniz, Boole, Bayes, Frege, Pierce, Quine, Russel, Godel, Turing, etc etc. There are entire research disciplines that are dedicated to the study of reasoning: philosophy, computer science, and, of course, all of mathematics itself. In AI research reasoning is a major topic studied by fields like automated theorem proving, planning and scheduling, program verification and model checking, etc, everything one finds in Russel & Norvig really. It is only in machine learning circles that reasoning seems to be such a big mystery that nobody can agree what it is; and in discussions on the internet about whether LLMs reason or not. And it should be clear that never in the history of human civilisation did "reasoning" mean "predict the most likely answer according to some training corpus".
- WhitneyLand 2y agoYeah sure there’s lots of research on reasoning. The papers I’ve seen that make claims about it are usually pretty precise about what it means in the context of that work and that specific claim, at least in the hard sciences listed.
- YeGoblynQueenne 2y agoI'm complaining because I haven't seen any such papers. Which ones do you have in mind?
- WhitneyLand 2y agoExamples go back 50 years, across many of the disciplines you’ve mentioned, but to throw out one that’s recent, on topic, and highly cited, there’s: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models https://arxiv.org/pdf/2501.19201 https://arxiv.org/pdf/2501.19201 It effectively treats “reasoning” as the ability to generate intermediate steps leading to a correct conclusion. Now, is this valid reasoning? Well, depends on the claim and the definition of reasoning. When someone just says AI can’t reason, I could argue for or against that depending on the specifics. It’s not enough to just say yes or no.
- YeGoblynQueenne 2y agoThanks for the link. >> It effectively treats “reasoning” as the ability to generate intermediate steps leading to a correct conclusion. Is "effectively" the same as "pretty precise" as per your previous comment? I don't see that because I searched the paper for all occurrences of "reasoning" and noticed two things: first that while the term is used to saturation there is no attempt to define it even informally, let alone precisely; and second that I could have replaced "reasoning" with any buzzword of the day and it would not change the impact of the paper. As far as I can tell the paper uses "reasoning" just because it happens to be what's currently trending in LLM circles. And still of course no attempt to engage with the common understanding of reasoning I discuss above, or any hint that the authors are aware of it. Sorry to be harsh, but you promised "examples that go back 50 years" and this is the kind of thing I've seen consistently in the last 15 or so.
- nuancebydefault 2y agoI like the true scottsman requirement. About any definition man can come up with for 'true' reasoning easily falls prey to a 'true scottsman fallacy'.