4 ms·
> Among other factors it can be argued that true reasoning involves symbolic logic and abstractions, and LLM are next token predictors. I think this is circula
by brookst 2y ago
> Among other factors it can be argued that true reasoning involves symbolic logic and abstractions, and LLM are next token predictors.
I think this is circular?
If an LLM is "merely" predicting the next tokens to put together a description of symbolic reasoning and abstractions... how is that different from really exercisng those things?
Can you give me an example of symbolic reasoning that I can't handwave away as just the likely next words given the starting place?
I'm not saying that LLMs have those capabilities; I'm question whether there is any utility in distinguishing the "actual" capability from identical outputs.
- griomnib 2y agoMathematical reasoning is the most obvious area where it breaks down. This paper does an excellent job of proving this point with some elegant examples: https://arxiv.org/pdf/2410.05229 https://arxiv.org/pdf/2410.05229
- brookst 2y agoSure, but people fail at mathematical reasoning. That doesn't mean people are incapable of reasoning. I'm not saying LLMs are perfect reasoners, I'm questioning the value of asserting that they cannot reason with some kind of "it's just text that looks like reasoning" argument.
- dartos 2y agoPeople can communicate each step, and review each step as that communication is happening. LLMs must be prompted for everything and don’t act on their own. The value in the assertion is in preventing laymen from seeing a statistical guessing machine be correct and assuming that it always will be. It’s dangerous to put so much faith in what in reality is a very good guessing machine. You can ask it to retrace its steps, but it’s just guessing at what it’s steps were, since it didn’t actually go through real reasoning, just generated text that reads like reasoning steps.
- brookst 2y ago> since it didn’t actually go through real reasoning, just generated text that reads like reasoning steps. Can you elaborate on the difference? Are you bringing sentience into it? It kind of sounds like it from "don't act on their own". But reasoning and sentience are wildly different things. > It’s dangerous to put so much faith in what in reality is a very good guessing machine Yes, exactly. That's why I think it is good we are supplementing fallible humans with fallible LLMs; we already have the processes in place to assume that not every actor is infallible.
- david-gpu 2y agoSo true. People who argue that we should not trust/use LLMs because they sometimes get it wrong are holding them to a higher standard than people -- we make mistakes too! Do we blindly trust or believe every single thing we hear from another person? Of course not. But hearing what they have to say can still be fruitful, and it is not like we have an oracle at our disposal who always speaks the absolute truth, either. We make do with what we have, and LLMs are another tool we can use.
- vundercind 2y ago> Can you elaborate on the difference? They’ll fail in different ways than something that thinks (and doesn’t have some kind of major disease of the brain going on) and often smack in the middle of appearing to think.
- ben_w 2y ago> People can communicate each step, and review each step as that communication is happening. Can, but don't by default. Just as LLMs can be asked for chain of thought, but the default for most users is just chat. This behaviour of humans is why we software developers have daily standup meetings, version control, and code review. > LLMs must be prompted for everything and don’t act on their own And this is why we humans have task boards like JIRA, and quarterly goals set by management.
- int_19h 2y agoA human brain in a vat doesn't act on its own, either.
- NBJack 2y agoThe idea is the average person would, sure. A mathematically oriented person would fair far better. Throw all the math problems you want at a LLM for training; it will still fail if you step outside of the familiar.
- ben_w 2y ago> it will still fail if you step outside of the familiar. To which I say: ᛋᛟ᛬ᛞᛟ᛬ᚻᚢᛗᚪᚾᛋ
- trashtester 2y agoᛒᚢᛏ ᚻᚢᛗᚪᚾ ᚻᚢᛒᚱᛁᛋ ᛈᚱᛖᚹᛖᚾᛏ ᚦᛖᛗ ᚠᚱᛟᛗ ᚱᛖᚪᛚᛁᛉᛁᚾᚷ ᚦᚻᚪᛏ
- ben_w 2y agoᛁᚾᛞᛖᛖᛞ᛬ᛁᛏ᛬ᛁᛋ᛬ᚻᚢᛒᚱᛁᛋ ᛁ᛬ᚻᚪᚹᛖ᛬ᛟᚠᛏᛖᚾ᛬ᛋᛖᛖᚾ᛬ᛁᚾ᛬ᛞᛁᛋᚲᚢᛋᛋᛁᛟᚾᛋ᛬ᛋᚢᚲ᛬ᚪᛋ᛬ᚦᛁᛋ᛬ᚲᛚᚪᛁᛗᛋ᛬ᚦᚪᛏ᛬ᚻᚢᛗᚪᚾ᛬ᛗᛁᚾᛞᛋ᛬ᚲᚪᚾ᛬ᛞᛟ᛬ᛁᛗᛈᛟᛋᛋᛁᛒᛚᛖ᛬ᚦᛁᛝᛋ᛬ᛋᚢᚲ᛬ᚪᛋ᛬ᚷᛖᚾᛖᚱᚪᛚᛚᚣ᛬ᛋᛟᛚᚹᛖ᛬ᚦᛖ᛬ᚻᚪᛚᛏᛁᛝ᛬ᛈᚱᛟᛒᛚᛖᛗ edit: Snap, you said the same in your other comment :)
- trashtester 2y agoSwitching back to latin letters... It seems to me that the idea of the Universal Turing Machine is quite misleading for a lot of people, such as David Deutsch. My impression is that the amount of compute to solve most problems that can really only be solved by Turing Machines is always going to remain inaccessible (unless they're trivally small). But at the same time, the universe seems to obey a principle of locality (as long as we only consider the Quantum Wave Function, and don't postulate that it collapses). Also, the quantum fields are subject to some simple (relative to LLMs) geometric symmetries, such as invariance under the U(1)xSU(2)xSU(3) group. As it turns out, similar group symmetries can be found in all sorts of places in the real world. Also it seems to me that at some level, both ANN's and biological brains set up a similar system to this physical reality, which may explain why brains develop this way and why both kinds are so good at simulating at least some aspects of the physical world, such as translation, rotation, some types of deformation, gravity, sound, light etc. And when biological brains that initially developed to predict the physical world is then use to create language, that language is bound to use the same type of machinere. And this may be why LLM's do language so well with a similar architecture.
- Workaccount2 2y agoMaybe I am not understanding the paper correctly, but it seems they tested "state of the art models" which is almost entirely composed of open source <27B parameter models. Mostly 8B and 3B models. This is kind of like giving algebra problems to 7 year olds to "test human algebra ability." If you are holding up a 3B parameter model as an example of "LLM's can't reason" I'm not sure if the authors are confused or out of touch. I mean, they do test 4o and O1 preview, but their performance is notablely absent from the paper's conclusion.
- dartos 2y agoIt’s difficult to reproducibly test openai models, since they can change from under you and you don’t have control over every hyperparameter. It would’ve been nice to see one of the larger llama models though.
- famouswaffles 2y agoThe results are there, it's just hidden away in the appendix. The result is that those models they don't actually suffer drops on 4/5 of their modified benchmarks. The one benchmark that does see actual drops that aren't explained by margin of error is the benchmark that adds "seemingly relevant but ultimately irrelevant information to problems" Those results are absent from the conclusion because the conclusion falls apart otherwise.
- dartos 2y agoThere isn’t much utility, but tbf the outputs aren’t identical. One danger is the human assumption that, since something appears to have that capability in some settings, it will have that capability in all settings. Thats a recipe for exploding bias, as we’ve seen with classic statistical crime detection systems.
- NBJack 2y agoInferring patterns in unfamiliar problems. Take a common word problem in a 5th grade math text book. Now, change as many words as possible; instead of two trains, make it two different animals; change the location to a rarely discussed town; etc. Even better, invent words/names to identify things. Someone who has done a word problem like that will very likely recognize the logic, even if the setting is completely different. Word tokenization alone should fail miserably.
- djmips 2y agoI have noted over my life that a lot of problems end up being a variation on solved problems from another more familiar domain but frustratingly take a long time to solve before realizing this was just like that thing you had already solved. Nevertheless, I do feel like humans do benefit from identifying meta patterns but as the chess example shows even we might be weak in unfamiliar areas.
- Propelloni 2y agoLearn how to solve one problem and apply the approach, logic and patterns to different problems. In German that's called "Transferleistung" (roughly "transfer success") and a big thing at advanced schools. Or, at least my teacher friends never stop talking about it. We get better at it over time, as probably most of us can attest.
- roywiggins 2y agoA lot of LLMs do weird things on the question "A farmer needs to get a bag of grain across a river. He has a boat that can transport himself and the grain. How does he do this?" (they often pattern-match on the farmer/grain/sheep/fox puzzle and start inventing pointless trips ("the farmer returns alone. Then, he crosses again.") in a way that a human wouldn't)
- vidarh 2y agoIt is. As it stands, throw a loop around an LLM and act as the tape, and an LLM can obviously be made Turing complete (you can get it to execute all the steps of a minimal Turing machine, so drop temperature so its deterministic, and you have a Turing complete system). To argue that they can't be made to reason is effectively to argue that there is some unknown aspect of the brain that allows us to compute functions not in the Turing computable set, which would be an astounding revelation if it could be proven. Until someone comes up with evidence for that, it is more reasonable to assume that it is a question of whether we have yet found a training mechanism that can lead to reasoning or not, not whether or not LLMs can learn to.
- vundercind 2y agoIt doesn’t follow that because a system is Turing complete the approach being used will eventually achieve reasoning.
- vidarh 2y agoNo, but that was also not the claim I made. The point is that as the person I replied to pointed out, that LLM's are "next token predictors" is a meaningless dismissal, as they can be both next token predictors and Turing complete, and given that unless reasoning requires functions outside the Turing computable (we know of no way of constructing such functions, or no way for them to exist) calling them "next token predictors" says nothing about their capabilities.