4 ms·

If it's mimicry of reason is indistinguishable from real reasoning, how is it not reasoning? Ultimately, an LLM models language and the process behind it's cre

If it's mimicry of reason is indistinguishable from real reasoning, how is it not reasoning? Ultimately, an LLM models language and the process behind it's creation to some degree of accuracy or another. If that model includes a way to approximate the act of reasoning, then it is reasoning to some extent. The extent I am happy to agree is open for discussion, but that reasoning is taking place at all is a little harder to attack.

onemoresoop 2y ago
No, it is distinguishable from real reasoning. Real reasoning, while flawed in various ways, goes through personal experience of the evaluator. LLMs don't have that capability at all. They're just sifting though tokens and associate statistical parameters to it with no skin in the game so to speak.
- danenania 2y ago
  It seems like an arbitrary distinction. If an LLM can accomplish a task that we’d all agree requires reasoning for a human to do, we can’t call that reasoning just because the mechanics are a bit different?
  - Barrin92 2y ago
    Yes because it isn't an arbitrary distinction. My good old TI-83 can do calculations that I can't even do in my head but unlike me it isn't reasoning about them, that's actually why it's able to do them so fast, and it has some pretty big implications about what it can't do. If you want to understand where a systems limitations are you need to understand not just what it does but how it does it, I feel like we need to start teaching classes on Behaviorism again.
    - danenania 2y ago
      An LLM’s mechanics are algorithmically much closer to the human brain (which the LLM is modeled on) than a TI-83, a CPU, or any other Turing machine. Which is why, like the brain, it can solve problems that no individual Turing machine can. Are you sure you aren’t just defining reasoning as something only a human can do?
      - meroes 2y ago
        My prior is reasoning is a conscious activity. There is a first person perspective. LLMs are so far removed mechanically from brains the idea they reason is not even remotely worth considering. Modeling neurons can be done with a series of pipes and flowing water, and that is not expected to give arise to consciousness either. Nor are nuerons and synapses likely to be sufficient for consciousness. You know how we insert ourselves into the process of coming up with a delicious recipe? That first person perspective might be also necessary for reasoning. No computer knows the taste of mint, it must be given parameters about it. So if a computer comes up with a recipe with mint, we know it wasn’t via tasting anything ever. A calculator doesn’t reason. A facsimile of something we have no idea about its role in consciousness has the same outlook as the calculator.
        danenania 2y ago
        Are we then conferring some kind of supernatural or religious properties to the brain’s particular implementation of neurons? If not, then why shouldn’t differently constructed but algorithmically similar systems be able to produce similar phenomena?
        oblio 2y ago
        Because we know practically nothing about brains so comparing them to LLMs is useless and nature is so complex that we're constantly discovering signs of hubris in human research. See C-sections versus natural birth. Formula versus mother's milk. Etc.
        Grimblewald 2y ago
        I think you'd benefit from reading Helen Keller's autobigoraphy "the world i live in", you might reach the same conclusions I did, this being that perhaps conciousness is flavoured by our unique way of experiencing our world, but not strictly neccesary for conciousness of some kind or another to form. I beleive conciousness to be a tool a sufficently complex neural network will develop in order for it to achieve whatever objective it has been given to optimize for.
        2y ago
    - Lerc 2y ago
      Taking a different tack from others in this thread. I don't think you can say that a TI-83 is not reasoning if it is doing calculations. Certainly it is not aware of any concepts of numbers and has no meaningful sense of the operation, but those are attributes of sentience, not reasoning. The reasoning ability of a calculator is extremely limited but what make those capabilities that it does have, non reasoning. What non-sentience based property do you think something should have to be considered reasoning. Do you consider sentience and reasoning to be one and the same? If not then you should be able to indicate what distinguishes one from the other. I doubt anyone here is arguing that chatGPT is sentient, yet plenty accept that it can reason to some extent.
      - Barrin92 2y ago
        >Do you consider sentience and reasoning to be one and the same? No, but I think they share some similarities. You can be sentient without doing any reasoning, just through experience, there's probably a lot of simple life forms in that category. Where they overlap I think, is in that they require a degree of reflection. Reasoning I'd say is the capacity to distinguish between truth and falsehoods, to have mental content of the object you're reasoning about and as a consequence have a notion of understanding and an interior or subjective view. The distinction I'd make is that calculation or memorization is not reasoning at all. My TI-83 or Stockfish can calculate math or chess but they have no notion of math or chess, they're basically Chinese rooms, they just perform mechanical operations. They can appear as if they reason, even a chess engine purely looking up results in a table base and with very simplistic brute force can play very strong chess but it doesn't know anything about chess. And with the LLMs you need to be careful because the "large" part does a lot of work. They often can sound like they reason but when they have to explain their reasoning they'll start to make up obvious falsehoods or contradictions. A good benchmark if something can reason is probably if it can.. reason about its reasoning coherently. I do think the very new chain-of-thought models are more of a step into that direction, the further you get away from relying on data the more likely you're building something that reasons but we're probably very early into systems like that.
- llm_trw 2y ago
  >goes through personal experience of the evaluator Real reasoning is being able to manipulate symbolic expressions in a consistent manner while preserving some invariants. Personal experience as logic is how you end up with the Holocaust.
- Grimblewald 2y ago
  LLM's have personal option by virtue of the fact they make statements of things they understand to the extent their training data allows. Their training data is not perfect, and in addition, through random chance the LLM will latch onto specific topics as a function of weight initialization and training data order. This would form a filter not unlike, yet distinct from, our understanding of personal experience. you could make the exact same argument against humans, we just learn to make sounds that elicit favourable responses. Besides, they have plenty "skin in the game", about the same as you or I.
- pjs_ 2y ago
  You say they are distinguishable. How would you experimentally distinguish two systems, one of which "goes through personal experience" and therefore is doing "real reasoning", vs one which is "sifting through tokens and associating statistical parameters"? Can you define a way to discriminate between these two situations?
lanstin 2y ago
I am getting two contradictory but plausible seeming replies when I ask about a certain set being the same when adding 1 to every value in the set, asked on how I ask the question. Correct answer: https://chatgpt.com/share/67a9500b-2360-8007-b70e-0bc2b84bc103 https://chatgpt.com/share/67a9500b-2360-8007-b70e-0bc2b84bc1... Incorrect answer (I think): https://chatgpt.com/share/67a950df-d4e0-8007-8105-95a9e5be1983 https://chatgpt.com/share/67a950df-d4e0-8007-8105-95a9e5be19...
- Grimblewald 2y ago
  What led you to beleive that mathematics is a good tool for evaluating an LLM? It is a thing they currently dont do well, since it is wildly out of domain of their training corpus - down the very way we structure information for an LLM to ingest. If we start doing the same for humans, most humans are in deep trouble.
  - lanstin 2y ago
    Well I am studying mathematics, and I use the LLM to help me learn. They aren't terrible, and they have all of arXiv to train on. Terrence Tao is doing some cool stuff with it - the idea will be an LLM to generate Lean proofs. https://mathstodon.xyz/@tao/113132502735585408 https://mathstodon.xyz/@tao/113132502735585408 https://terrytao.wordpress.com/2024/12/05/ai-for-math-fund/ https://terrytao.wordpress.com/2024/12/05/ai-for-math-fund/ (Professor Tao is probably the best or at least most productive in the most fields current mathematician).
    - lanstin 2y ago
      And I can assure you when I start to talk about these topics with the average human person that doesn't know the material, they just laugh at me. Even my wife who has a PhD in physics. Here's some cool math I learned from a regular book, not an LLM: https://en.wikipedia.org/wiki/Khinchin%27s_constant https://en.wikipedia.org/wiki/Khinchin%27s_constant

onemoresoop 2y ago

No, it is distinguishable from real reasoning. Real reasoning, while flawed in various ways, goes through personal experience of the evaluator. LLMs don't have that capability at all. They're just sifting though tokens and associate statistical parameters to it with no skin in the game so to speak.

danenania 2y ago
It seems like an arbitrary distinction. If an LLM can accomplish a task that we’d all agree requires reasoning for a human to do, we can’t call that reasoning just because the mechanics are a bit different?
- Barrin92 2y ago
  Yes because it isn't an arbitrary distinction. My good old TI-83 can do calculations that I can't even do in my head but unlike me it isn't reasoning about them, that's actually why it's able to do them so fast, and it has some pretty big implications about what it can't do. If you want to understand where a systems limitations are you need to understand not just what it does but how it does it, I feel like we need to start teaching classes on Behaviorism again.
  - danenania 2y ago
    An LLM’s mechanics are algorithmically much closer to the human brain (which the LLM is modeled on) than a TI-83, a CPU, or any other Turing machine. Which is why, like the brain, it can solve problems that no individual Turing machine can. Are you sure you aren’t just defining reasoning as something only a human can do?
    - meroes 2y ago
      My prior is reasoning is a conscious activity. There is a first person perspective. LLMs are so far removed mechanically from brains the idea they reason is not even remotely worth considering. Modeling neurons can be done with a series of pipes and flowing water, and that is not expected to give arise to consciousness either. Nor are nuerons and synapses likely to be sufficient for consciousness. You know how we insert ourselves into the process of coming up with a delicious recipe? That first person perspective might be also necessary for reasoning. No computer knows the taste of mint, it must be given parameters about it. So if a computer comes up with a recipe with mint, we know it wasn’t via tasting anything ever. A calculator doesn’t reason. A facsimile of something we have no idea about its role in consciousness has the same outlook as the calculator.
      - danenania 2y ago
        Are we then conferring some kind of supernatural or religious properties to the brain’s particular implementation of neurons? If not, then why shouldn’t differently constructed but algorithmically similar systems be able to produce similar phenomena?
        oblio 2y ago
        Because we know practically nothing about brains so comparing them to LLMs is useless and nature is so complex that we're constantly discovering signs of hubris in human research. See C-sections versus natural birth. Formula versus mother's milk. Etc.
      - Grimblewald 2y ago
        I think you'd benefit from reading Helen Keller's autobigoraphy "the world i live in", you might reach the same conclusions I did, this being that perhaps conciousness is flavoured by our unique way of experiencing our world, but not strictly neccesary for conciousness of some kind or another to form. I beleive conciousness to be a tool a sufficently complex neural network will develop in order for it to achieve whatever objective it has been given to optimize for.
      - 2y ago
  - Lerc 2y ago
    Taking a different tack from others in this thread. I don't think you can say that a TI-83 is not reasoning if it is doing calculations. Certainly it is not aware of any concepts of numbers and has no meaningful sense of the operation, but those are attributes of sentience, not reasoning. The reasoning ability of a calculator is extremely limited but what make those capabilities that it does have, non reasoning. What non-sentience based property do you think something should have to be considered reasoning. Do you consider sentience and reasoning to be one and the same? If not then you should be able to indicate what distinguishes one from the other. I doubt anyone here is arguing that chatGPT is sentient, yet plenty accept that it can reason to some extent.
    - Barrin92 2y ago
      >Do you consider sentience and reasoning to be one and the same? No, but I think they share some similarities. You can be sentient without doing any reasoning, just through experience, there's probably a lot of simple life forms in that category. Where they overlap I think, is in that they require a degree of reflection. Reasoning I'd say is the capacity to distinguish between truth and falsehoods, to have mental content of the object you're reasoning about and as a consequence have a notion of understanding and an interior or subjective view. The distinction I'd make is that calculation or memorization is not reasoning at all. My TI-83 or Stockfish can calculate math or chess but they have no notion of math or chess, they're basically Chinese rooms, they just perform mechanical operations. They can appear as if they reason, even a chess engine purely looking up results in a table base and with very simplistic brute force can play very strong chess but it doesn't know anything about chess. And with the LLMs you need to be careful because the "large" part does a lot of work. They often can sound like they reason but when they have to explain their reasoning they'll start to make up obvious falsehoods or contradictions. A good benchmark if something can reason is probably if it can.. reason about its reasoning coherently. I do think the very new chain-of-thought models are more of a step into that direction, the further you get away from relying on data the more likely you're building something that reasons but we're probably very early into systems like that.
llm_trw 2y ago
>goes through personal experience of the evaluator Real reasoning is being able to manipulate symbolic expressions in a consistent manner while preserving some invariants. Personal experience as logic is how you end up with the Holocaust.
Grimblewald 2y ago
LLM's have personal option by virtue of the fact they make statements of things they understand to the extent their training data allows. Their training data is not perfect, and in addition, through random chance the LLM will latch onto specific topics as a function of weight initialization and training data order. This would form a filter not unlike, yet distinct from, our understanding of personal experience. you could make the exact same argument against humans, we just learn to make sounds that elicit favourable responses. Besides, they have plenty "skin in the game", about the same as you or I.
pjs_ 2y ago
You say they are distinguishable. How would you experimentally distinguish two systems, one of which "goes through personal experience" and therefore is doing "real reasoning", vs one which is "sifting through tokens and associating statistical parameters"? Can you define a way to discriminate between these two situations?

lanstin 2y ago

I am getting two contradictory but plausible seeming replies when I ask about a certain set being the same when adding 1 to every value in the set, asked on how I ask the question. Correct answer: https://chatgpt.com/share/67a9500b-2360-8007-b70e-0bc2b84bc103 https://chatgpt.com/share/67a9500b-2360-8007-b70e-0bc2b84bc1... Incorrect answer (I think): https://chatgpt.com/share/67a950df-d4e0-8007-8105-95a9e5be1983 https://chatgpt.com/share/67a950df-d4e0-8007-8105-95a9e5be19...

Grimblewald 2y ago
What led you to beleive that mathematics is a good tool for evaluating an LLM? It is a thing they currently dont do well, since it is wildly out of domain of their training corpus - down the very way we structure information for an LLM to ingest. If we start doing the same for humans, most humans are in deep trouble.
- lanstin 2y ago
  Well I am studying mathematics, and I use the LLM to help me learn. They aren't terrible, and they have all of arXiv to train on. Terrence Tao is doing some cool stuff with it - the idea will be an LLM to generate Lean proofs. https://mathstodon.xyz/@tao/113132502735585408 https://mathstodon.xyz/@tao/113132502735585408 https://terrytao.wordpress.com/2024/12/05/ai-for-math-fund/ https://terrytao.wordpress.com/2024/12/05/ai-for-math-fund/ (Professor Tao is probably the best or at least most productive in the most fields current mathematician).
  - lanstin 2y ago
    And I can assure you when I start to talk about these topics with the average human person that doesn't know the material, they just laugh at me. Even my wife who has a PhD in physics. Here's some cool math I learned from a regular book, not an LLM: https://en.wikipedia.org/wiki/Khinchin%27s_constant https://en.wikipedia.org/wiki/Khinchin%27s_constant