10 ms·

There was a whole bunch of people who claimed LLMs can't reason at all and that everything is a regurgitation. I wonder what they have to say about this. Like,

There was a whole bunch of people who claimed LLMs can't reason at all and that everything is a regurgitation. I wonder what they have to say about this. Like, what exactly is going on here with chain of thought reasoning from their expert perspective?

teej 2y ago
My mental model for chain-of-thought is not “reasoning”. It’s more of an iterative search through the latent space of the model.
- valine 2y ago
  And human reasoning is somehow more magical? Really struggling to understand the distinction between searching through a latent space and “reasoning”.
  - Exoristos 2y ago
    As you're demonstrating, it depends on the human.
    - error_logic 2y ago
      Indeed, some of us are more willing to believe ourselves somehow special and block out our own irrationalities and missteps. There is a deep need in some people to ignore their own flaws so they can present themselves as more competent. More reasonably, people try to have a thought ready before starting to talk, or will stop and think again if they notice a contradiction in what they're saying. Other people will keep right on going and double down if they've made a mistake, refusing to acknowledge the opportunity for learning and blocking out inconvenient contradiction. Interestingly this is itself quite relevant to the AI alignment problem. Competence requires being able to accurately reflect [on] the patterns, whereas goal-seeking and morality require ignoring some options as costly/dangerous distractions--but not to the point of incompetence. Balancing the two is tricky, and neither people nor governments have solved this universally, let alone an AI.
  - exe34 2y ago
    no, humans do the same thing - even the "intuition" of things to try are probably the results of searches, but we don't have conscious access to them. certainly it's the simplest explanation that fits the observations.
    - sgt101 2y ago
      Given the observations of the workings of brains though (slow, low energy) we're back to some magic to make this happen?
      - exe34 2y ago
        does a machine with ten trillion components need magic to function? maybe. but probably not.
  - sgt101 2y ago
    It would be startling if humans could reason this way with two kilos of meat consuming 40w. Even more surprising given that we get answers in ~1s with a cross brain comms time of about 0.25s To me it's clear that human reasoning is different from a massive search of a latent space. I can say that when I am thinking I maybe try half a dozen ideas or scenarios, but very rarely more. I can't say where those ideas come from or how I make them up though. Maybe we can't frame what it is and how it works with human languages though, which might make it seem magical in some way. Or maybe there's a good framing that I don't know - would love to learn!
    - naasking 2y ago
      > I can say that when I am thinking I maybe try half a dozen ideas or scenarios, but very rarely more You mean you consciously try half a dozen ideas. That's not really reflective of what's happening subconsciously though, which could be a broader kind of search that filters out other possibilities and only bubbles up some options you would consider promising.
    - daveguy 2y ago
      It could be a matter of massively parallel computation (trillions of synapses functioning at once) plus efficiency of chemical-analog vs transistor. But, personally I believe, there are a lot more algorithmic issues between what we are doing now and what the brain is doing. Even if it's just a matter of scale, we are at least a million fold away from the bandwidth and processing power of a single human brain vs exaflop supercomputers, especially in efficiency. Closing that gap, will require 20 years of Moore's law + bandwidth + efficiency + equivalent of 18 years grounded training (probably faster than 18y). We are not close to having our most powerful computers as flexible and general purpose intelligence as a single human. The 20 year limitation is based on processing, efficiency, and bandwidth. Hopefully the algorithms will catch up by then.
      - valine 2y ago
        Even if everything you said is true it doesn’t mean general AI is 20 years away. Computers are many orders of magnitude faster at non-parallel tasks. Who’s to say raw clock speeds won’t easily compensate for any deficiencies vs the brain. You can run a trillion parameter model on a single machine today. We’re maybe 4 or 5 years away from that same 1T model running on an iPhone.
  - numba888 2y ago
    Yes, it has at least two distinct parts: consciousness and sub. First is visible to 'us', it's inner monologue or vision, or other senses. But we don't 'know' what happens in subconsciousness. The answer just pops up from nowhere. 'Reasoning' LLMs for now have the first part. The 'sub' part is questionable, depends how you look at latent space.
- qnleigh 2y ago
  Can you elaborate? I think this is a really interesting question. It comes up over and over again, but it often feels like the two sides of the debate talk past each other. What does the mental model of 'iterative search through latent space' convey that 'reasoning' doesn't? Human reasoning also often searches through a space of potential solution methods and similar problems, and keeps applying them until making progress. I appreciate that there might be danger in using words like 'thinking' and 'reasoning' in that they cause us to anthropomorphize LLMs, but if we are careful not to do so then this is a separate issue.
  - naasking 2y ago
    "Search" a class of fairly well defined algorithms, "reasoning" is vague / ambiguous. If reasoning can be reduced to some kind of search, that makes its meaning more precise and understandable.
    - qnleigh 2y ago
      Sure one word is more precise than the other, but in the context of this question that's beside the point. If one makes the claim 'it's not reasoning, it's just X,' the burden of proof is that X is district from reasoning. That 'reasoning' is an ambiguous term, long makes it harder to argue that X is not compatible with. I was pointing out that, at least superficially, they seem potentially compatible. Again I'm not at all trying to argue that LLMs do reason. I'm trying to understand how one definitively argues that they don't.
      - naasking 2y ago
        I agree I wouldn't a priori exclude reasoning as being a form of search, I was just addressing the question of reduction providing clarity. I do think people are far too dismissive of classifying something as reasoning simply because the latter is vague where the reduction is generally something understood.
- dyauspitr 2y ago
  That sounds like Sheldon from BBT describing human reasoning.
dartos 2y ago
You can still regurgitate a chain of thought response… It’s all still tokens…
- ninetyninenine 2y ago
  Oh stop. The neural network is piecing together the tokens in a way that indicates reasoning. Clearly. I don't really need to say this, we all know it now and so do you. Your statement here is just weak. It's really embarrassing the stubborn stance people were taking that LLMs weren't intelligent and wouldn't make any progress towards agi. I sometimes wonder how people live with themselves when they realize their wrong outlook on things is just as canned and biased as the hallucinations of LLMs themselves.
  - dartos 2y ago
    Personal attacks really make your argument stronger.
    - ninetyninenine 2y ago
      I said your statement was weak I didn't say YOU were weak. Don't take it personally. It wasn't meant to be that way. If you take offense then I apologize. That being said, my argument was a statement about a general fact that is very true. The sentiment not too long ago was these things are just regurgitators with no similarity to human intelligence. I think it's clear now that all the previous claims were just completely baseless.
      - Jensson 2y ago
        > That being said, my argument was a statement about a general fact that is very true. Is this a general fact that is very true? It sounds like you are judging rather than stating a fact. > It's really embarrassing the stubborn stance people were taking that LLMs weren't intelligent and wouldn't make any progress towards agi. I sometimes wonder how people live with themselves when they realize their wrong outlook on things is just as canned and biased as the hallucinations of LLMs themselves.
calibas 2y ago
I don't get this whole debate, surely what's meant by "reason" can be strictly defined and measured? Then we can conclusively say whether or not it's happening with LLMs. It seems to me like the debate is largely just semantics about how to define "reason".
- ninetyninenine 2y ago
  It's semantics. But there's a general motivation behind it that's less technical. Basically if it can reason, it implies human level intelligence. That's the line separating man from machine. Once you cross that line there's a whole bunch of economic, cultural and existential changes that happen to society that are permanent. We can't go back. This is what people are debating about. Many many people don't want to believe we crossed the line with LLMs. It brings about a sort of existential dread. Especially to programmers who's pride is entirely dependent upon their intelligence and ability to program.
  - eastbound 2y ago
    It’s not about being afraid, it’s that the auto-reconfiguration of neurons seems advanced to decompile it at this time, and it surprising that LLM, which are just a probabilistic model of guessing the next word, could be capable of actual thought. The day it happens, we’ll believe it. There are only 100bn neurons in a brain, after all, and many more than this in modern machines, so it is theoretically possible. Just LLMs seemed too simple for that.
  - sgt101 2y ago
    We've had "reasoning" machines for a long time - I learned chess playing against a computer in the 1980's. But we don't have reasoning that can be applied generally in the open world yet. Or at least I haven't seen it. In terms of society it should be easy to track if this is true or not. Healthcare and elder care settings will be a very early canary of this because there is huge pressure for improvement and change in these. General reasoning machines will make a very significant, clear and early impact here. I have seen note taking apps for nurses - but not much else so far.
  - 2y ago
littlestymaar 2y ago
> There was a whole bunch of people who claimed LLMs can't reason at all and that everything is a regurgitation. I wonder what they have to say about this. I don't see that as a refutation of the former actually: model trained to be stochastic parrots with next-token prediction as only learning target were indeed stochastic parrots and now we've moved to a completely different technology that features reinforcement learning in its training so it will go farther and farther from stochastic parrots and more and more towards “intelligence”. If anything, the fact that the entire industry has now moved to RL instead of just cramming through trillions of tokens to make progress is a pretty strong acknowledgement that the “stochastic parrots” crowd was right.
csomar 2y ago
They sure do have a certain amount of "reasoning". Here is R1 trying to multiply a large number (successfully): https://gist.github.com/omarabid/038678cc269a3f2db756a7e082526f17 https://gist.github.com/omarabid/038678cc269a3f2db756a7e0825... If you pick a random combination, there is a very good chance that the combination and the product do not exist anywhere. So the LLM has to "create" it somehow. It sure goes through a lot (hundreds of lines of self-reflection) but it successfully does the math. I don't think it is the same kind of "reasoning" as humans, but there is an emergent kind of structure happening here that is allowing for this reasoning.
- Jensson 2y ago
  That is like saying a calculator is reasoning since it spends many cycles thinking about the problem before answering. You could say yes a calculator is reasoning, but most would say a calculator isn't really reasoning. > there is an emergent kind of structure happening here that is allowing for this reasoning. Yes, but that structure doesn't need to be more complicated than a calculator. The complicated bit is the fuzzy lookup that returns fuzzy patterns (every LLM does that though so its old), and then how it recourses that, but seeing how that can result in it recursing into a solution for a multiplication is no harder than understanding how a calculator works. So to add basic reasoning like this you just have to add "functions" for the LLM to fuzzy lookup, and then when looking up those "functions" it will eventually recourse into a solution. The hard part is finding a set of such functions that solves a wide range of problems, and that was solved here.
- mkl 2y ago
  I think it is very human-like reasoning. I reason exactly like this when doing numerical calculations in my head, and I'm a mathematician (no, I can't work with numbers this big in my head). It's quite funny where late in the piece it says it's checking with a calculator, which a human would do if possible (if they didn't start out with that) but then its statements are pretty much the same as before, and it probably didn't actually use a calculator.
- daveguy 2y ago
  I think that demonstrates how far we are from reasoning and from true self-reflection. If either were happening it would know that it has the capability to multiply four numbers in nanoseconds and know that it doesn't even matter the order it multiplies them. The first reasoning step, I have 4 numbers to multiply, should be the only one necessary.
Vampiero 2y ago
The test is very simple but people simply don't realize it. When LLMs are good at Prolog, it means they're good at logic, which means they're good at reasoning. Until then, you can't trust them.
- daveguy 2y ago
  What exactly do you mean by "good at Prolog"?
  - Vampiero 2y ago
    It means being good at first order predicate logic. And possibly higher order too when you consider `call/n` and lambdas. It means being good at generalization, at reasoning in causal terms, at understanding structure and grammar, at encoding problems as graphs and querying them for solutions, and much more. Basically it's what current LLMs lack. They're good at spewing coherent text but they lack the building blocks of reason, which are made of logic, and which confer the quality of being consistent. A implies B.

teej 2y ago

My mental model for chain-of-thought is not “reasoning”. It’s more of an iterative search through the latent space of the model.

valine 2y ago
And human reasoning is somehow more magical? Really struggling to understand the distinction between searching through a latent space and “reasoning”.
- Exoristos 2y ago
  As you're demonstrating, it depends on the human.
  - error_logic 2y ago
    Indeed, some of us are more willing to believe ourselves somehow special and block out our own irrationalities and missteps. There is a deep need in some people to ignore their own flaws so they can present themselves as more competent. More reasonably, people try to have a thought ready before starting to talk, or will stop and think again if they notice a contradiction in what they're saying. Other people will keep right on going and double down if they've made a mistake, refusing to acknowledge the opportunity for learning and blocking out inconvenient contradiction. Interestingly this is itself quite relevant to the AI alignment problem. Competence requires being able to accurately reflect [on] the patterns, whereas goal-seeking and morality require ignoring some options as costly/dangerous distractions--but not to the point of incompetence. Balancing the two is tricky, and neither people nor governments have solved this universally, let alone an AI.
- exe34 2y ago
  no, humans do the same thing - even the "intuition" of things to try are probably the results of searches, but we don't have conscious access to them. certainly it's the simplest explanation that fits the observations.
  - sgt101 2y ago
    Given the observations of the workings of brains though (slow, low energy) we're back to some magic to make this happen?
    - exe34 2y ago
      does a machine with ten trillion components need magic to function? maybe. but probably not.
- sgt101 2y ago
  It would be startling if humans could reason this way with two kilos of meat consuming 40w. Even more surprising given that we get answers in ~1s with a cross brain comms time of about 0.25s To me it's clear that human reasoning is different from a massive search of a latent space. I can say that when I am thinking I maybe try half a dozen ideas or scenarios, but very rarely more. I can't say where those ideas come from or how I make them up though. Maybe we can't frame what it is and how it works with human languages though, which might make it seem magical in some way. Or maybe there's a good framing that I don't know - would love to learn!
  - naasking 2y ago
    > I can say that when I am thinking I maybe try half a dozen ideas or scenarios, but very rarely more You mean you consciously try half a dozen ideas. That's not really reflective of what's happening subconsciously though, which could be a broader kind of search that filters out other possibilities and only bubbles up some options you would consider promising.
  - daveguy 2y ago
    It could be a matter of massively parallel computation (trillions of synapses functioning at once) plus efficiency of chemical-analog vs transistor. But, personally I believe, there are a lot more algorithmic issues between what we are doing now and what the brain is doing. Even if it's just a matter of scale, we are at least a million fold away from the bandwidth and processing power of a single human brain vs exaflop supercomputers, especially in efficiency. Closing that gap, will require 20 years of Moore's law + bandwidth + efficiency + equivalent of 18 years grounded training (probably faster than 18y). We are not close to having our most powerful computers as flexible and general purpose intelligence as a single human. The 20 year limitation is based on processing, efficiency, and bandwidth. Hopefully the algorithms will catch up by then.
    - valine 2y ago
      Even if everything you said is true it doesn’t mean general AI is 20 years away. Computers are many orders of magnitude faster at non-parallel tasks. Who’s to say raw clock speeds won’t easily compensate for any deficiencies vs the brain. You can run a trillion parameter model on a single machine today. We’re maybe 4 or 5 years away from that same 1T model running on an iPhone.
- numba888 2y ago
  Yes, it has at least two distinct parts: consciousness and sub. First is visible to 'us', it's inner monologue or vision, or other senses. But we don't 'know' what happens in subconsciousness. The answer just pops up from nowhere. 'Reasoning' LLMs for now have the first part. The 'sub' part is questionable, depends how you look at latent space.
qnleigh 2y ago
Can you elaborate? I think this is a really interesting question. It comes up over and over again, but it often feels like the two sides of the debate talk past each other. What does the mental model of 'iterative search through latent space' convey that 'reasoning' doesn't? Human reasoning also often searches through a space of potential solution methods and similar problems, and keeps applying them until making progress. I appreciate that there might be danger in using words like 'thinking' and 'reasoning' in that they cause us to anthropomorphize LLMs, but if we are careful not to do so then this is a separate issue.
- naasking 2y ago
  "Search" a class of fairly well defined algorithms, "reasoning" is vague / ambiguous. If reasoning can be reduced to some kind of search, that makes its meaning more precise and understandable.
  - qnleigh 2y ago
    Sure one word is more precise than the other, but in the context of this question that's beside the point. If one makes the claim 'it's not reasoning, it's just X,' the burden of proof is that X is district from reasoning. That 'reasoning' is an ambiguous term, long makes it harder to argue that X is not compatible with. I was pointing out that, at least superficially, they seem potentially compatible. Again I'm not at all trying to argue that LLMs do reason. I'm trying to understand how one definitively argues that they don't.
    - naasking 2y ago
      I agree I wouldn't a priori exclude reasoning as being a form of search, I was just addressing the question of reduction providing clarity. I do think people are far too dismissive of classifying something as reasoning simply because the latter is vague where the reduction is generally something understood.
dyauspitr 2y ago
That sounds like Sheldon from BBT describing human reasoning.

dartos 2y ago

You can still regurgitate a chain of thought response… It’s all still tokens…

ninetyninenine 2y ago
Oh stop. The neural network is piecing together the tokens in a way that indicates reasoning. Clearly. I don't really need to say this, we all know it now and so do you. Your statement here is just weak. It's really embarrassing the stubborn stance people were taking that LLMs weren't intelligent and wouldn't make any progress towards agi. I sometimes wonder how people live with themselves when they realize their wrong outlook on things is just as canned and biased as the hallucinations of LLMs themselves.
- dartos 2y ago
  Personal attacks really make your argument stronger.
  - ninetyninenine 2y ago
    I said your statement was weak I didn't say YOU were weak. Don't take it personally. It wasn't meant to be that way. If you take offense then I apologize. That being said, my argument was a statement about a general fact that is very true. The sentiment not too long ago was these things are just regurgitators with no similarity to human intelligence. I think it's clear now that all the previous claims were just completely baseless.
    - Jensson 2y ago
      > That being said, my argument was a statement about a general fact that is very true. Is this a general fact that is very true? It sounds like you are judging rather than stating a fact. > It's really embarrassing the stubborn stance people were taking that LLMs weren't intelligent and wouldn't make any progress towards agi. I sometimes wonder how people live with themselves when they realize their wrong outlook on things is just as canned and biased as the hallucinations of LLMs themselves.

calibas 2y ago

I don't get this whole debate, surely what's meant by "reason" can be strictly defined and measured? Then we can conclusively say whether or not it's happening with LLMs. It seems to me like the debate is largely just semantics about how to define "reason".

ninetyninenine 2y ago
It's semantics. But there's a general motivation behind it that's less technical. Basically if it can reason, it implies human level intelligence. That's the line separating man from machine. Once you cross that line there's a whole bunch of economic, cultural and existential changes that happen to society that are permanent. We can't go back. This is what people are debating about. Many many people don't want to believe we crossed the line with LLMs. It brings about a sort of existential dread. Especially to programmers who's pride is entirely dependent upon their intelligence and ability to program.
- eastbound 2y ago
  It’s not about being afraid, it’s that the auto-reconfiguration of neurons seems advanced to decompile it at this time, and it surprising that LLM, which are just a probabilistic model of guessing the next word, could be capable of actual thought. The day it happens, we’ll believe it. There are only 100bn neurons in a brain, after all, and many more than this in modern machines, so it is theoretically possible. Just LLMs seemed too simple for that.
- sgt101 2y ago
  We've had "reasoning" machines for a long time - I learned chess playing against a computer in the 1980's. But we don't have reasoning that can be applied generally in the open world yet. Or at least I haven't seen it. In terms of society it should be easy to track if this is true or not. Healthcare and elder care settings will be a very early canary of this because there is huge pressure for improvement and change in these. General reasoning machines will make a very significant, clear and early impact here. I have seen note taking apps for nurses - but not much else so far.
- 2y ago

littlestymaar 2y ago

> There was a whole bunch of people who claimed LLMs can't reason at all and that everything is a regurgitation. I wonder what they have to say about this. I don't see that as a refutation of the former actually: model trained to be stochastic parrots with next-token prediction as only learning target were indeed stochastic parrots and now we've moved to a completely different technology that features reinforcement learning in its training so it will go farther and farther from stochastic parrots and more and more towards “intelligence”. If anything, the fact that the entire industry has now moved to RL instead of just cramming through trillions of tokens to make progress is a pretty strong acknowledgement that the “stochastic parrots” crowd was right.

csomar 2y ago

They sure do have a certain amount of "reasoning". Here is R1 trying to multiply a large number (successfully): https://gist.github.com/omarabid/038678cc269a3f2db756a7e082526f17 https://gist.github.com/omarabid/038678cc269a3f2db756a7e0825... If you pick a random combination, there is a very good chance that the combination and the product do not exist anywhere. So the LLM has to "create" it somehow. It sure goes through a lot (hundreds of lines of self-reflection) but it successfully does the math. I don't think it is the same kind of "reasoning" as humans, but there is an emergent kind of structure happening here that is allowing for this reasoning.

Jensson 2y ago
That is like saying a calculator is reasoning since it spends many cycles thinking about the problem before answering. You could say yes a calculator is reasoning, but most would say a calculator isn't really reasoning. > there is an emergent kind of structure happening here that is allowing for this reasoning. Yes, but that structure doesn't need to be more complicated than a calculator. The complicated bit is the fuzzy lookup that returns fuzzy patterns (every LLM does that though so its old), and then how it recourses that, but seeing how that can result in it recursing into a solution for a multiplication is no harder than understanding how a calculator works. So to add basic reasoning like this you just have to add "functions" for the LLM to fuzzy lookup, and then when looking up those "functions" it will eventually recourse into a solution. The hard part is finding a set of such functions that solves a wide range of problems, and that was solved here.
mkl 2y ago
I think it is very human-like reasoning. I reason exactly like this when doing numerical calculations in my head, and I'm a mathematician (no, I can't work with numbers this big in my head). It's quite funny where late in the piece it says it's checking with a calculator, which a human would do if possible (if they didn't start out with that) but then its statements are pretty much the same as before, and it probably didn't actually use a calculator.
daveguy 2y ago
I think that demonstrates how far we are from reasoning and from true self-reflection. If either were happening it would know that it has the capability to multiply four numbers in nanoseconds and know that it doesn't even matter the order it multiplies them. The first reasoning step, I have 4 numbers to multiply, should be the only one necessary.

Vampiero 2y ago

The test is very simple but people simply don't realize it. When LLMs are good at Prolog, it means they're good at logic, which means they're good at reasoning. Until then, you can't trust them.

daveguy 2y ago
What exactly do you mean by "good at Prolog"?
- Vampiero 2y ago
  It means being good at first order predicate logic. And possibly higher order too when you consider `call/n` and lambdas. It means being good at generalization, at reasoning in causal terms, at understanding structure and grammar, at encoding problems as graphs and querying them for solutions, and much more. Basically it's what current LLMs lack. They're good at spewing coherent text but they lack the building blocks of reason, which are made of logic, and which confer the quality of being consistent. A implies B.