6 ms·

I'm sick of people saying LLMs don't reason as if they know unequivocally. We dont' understand what's going on with LLMs. We have evidence of them reasoning su

I'm sick of people saying LLMs don't reason as if they know unequivocally. We dont' understand what's going on with LLMs. We have evidence of them reasoning successfully. And we have evidence of them Failing to reason. That evidence DOES not logically lead to "LLMs don't reason". There are multiple possibilities here. 1. LLMs can reason, but they can't tell the difference between hallucination and reasoning. 2. LLMs can reason, but they choose to lie. 3. LLMs can't reason, when they get something right, it's pure coincidence. There is not a single person who can prove or disprove ANY of those 3 points. What most people end up doing is ironically Identical to what the LLM does. They hallucinate an answer: "LLMs cannot truly reason." Think about it. There is NO EVIDENCE or insight into how an LLM works that can even tell us how an LLM arrived at a specific response. We HAVE NOTHING. Yet why do I see everywhere people like this guy, who makes claims out of nowhere? Which brings us back full circle: do humans reason? How similar is human hallucination to LLM hallucination?

staunton 1y ago
4. We don't actually have a coherent concept of what "reasoning" means in general. Same for "hallucination"...
- ninetyninenine 1y ago
  We do. Reasoning is clearly defined. Sentience or consciousness is not. Reasoning is basically the same as using logic to arrive at a conclusion when given a set of rules and axioms. Hallucination is producing a conclusion by not following rules or axioms.
  - staunton 1y ago
    You could go further and say that anything a computation could ever possibly do is a matter of using logic to arrive at a conclusion given just the right rules and axioms. That would mean anything going beyond that (potentially "sentience" or "consciousness", possibly even -- as you seemed to suggest -- "hallucination", though apparently a computer program can do it) would have to be "magic" that's impossible to even describe using logic-based science, let alone simulate by a program. (Could we have science that transcends logic?) My original claim refers to the mundane situations where people get an LLM output which isn't what they wanted and go "see, it isn't reasoning! That's the problem!". For each such instance (or at least most of them), the "rules and axioms" that would have to be used to arrive at the desired conclusion would be different, sometimes incompatible, and in any case the people demanding "that it reason" wouldn't be able to spell those rules and axioms out, often even given years to do it, at least without allowing them to be obviously contradictory or trivially close to the specific conclusion or output they want. (Incidentally, that's why LLMs are interesting in the first place!) So sure, you can trick an LLM into failing on a syllogism problem, and that does mean "it can't reason" in a certain narrow sense, but I don't think that's what we're actually talking about when someone is contrasting "reasoning" and "hallucinating", which is what I was originally objecting to. That's what I meant when I said that we don't have a coherent concept of what "reasoning" means in general.
tbrownaw 1y ago
The is zero (legitimate) question about what LLMs do. They're math, not magic. They highlight fun philosophical / definitional questions like the Chinese room thought experiment, but that's it.
- ninetyninenine 1y ago
  And where is your evidence of this? Can you even prove this for a single query/response pair? Can you trace the genesis of the signals travelling through the neural network and prove to me via the structure that no reasoning was performed at all? You don't. You arrived at this claim with NO evidence. How do you arrive at a claim without evidence? There's a word for it. It's called an Hallucination. Humans, like LLMs, have trouble saying "I don't know."
  - antihipocrat 1y ago
    Pretty sure the people creating the best models understand how these things work. Plenty of papers explain the process and you and I can create our own basic model ourselves from scratch with enough aptitude and drive. There is a lot of art involved with the best models. We aren't dealing with determinism with regard to the corpus used to train the model (too much information to curate for accuracy) nor the LLM output (probabilistic by design) nor the prompt input (humans language is dynamic and open to multiple interpretations) nor the human subjective assessment of the output. That there is a product available that manages to produce great results despite these huge challenges is amazing and this is the part that is not quantifiable - in the sense that the data scientists decisions made with regard to temperature settings etc are not derived from any fundamental property but more from an inspired group of people with qualitative instincts aligned with producing great results
    - ninetyninenine 1y ago
      https://futurism.com/anthropic-ceo-admits-ai-ignorance https://futurism.com/anthropic-ceo-admits-ai-ignorance Let me tell you something. You can go online, find a tutorial on how to make an LLM, and actually make one at home. The only thing stopping you from making an OpenAI scale LLM is compute resources. If you happen to make an LLM from scratch you won't know how it works, and neither do the people who are at OpenAi or anthropic.
antihipocrat 1y ago
I don't think the three options provided are exhaustive. For example: 4. LLMs can't reason, when they get something right, it's because the corpus of information used to create the model provides a very high probability that the response to that specific prompt is correct.
- ninetyninenine 1y ago
  They're not exhaustive, but add as many options as you want to the ones given. Most of those options can't be proven or disproven. We don't understand what's going on with LLMs. We've built something from scratch that we can't understand or control. That is what an LLM is.
proc0 1y ago
You can derive #3 from first principles of how the underlying gradient descent and backpropagation work. They approximate functions, therefore when it gets something wrong it means something went wrong with either the gradient descent or back propagation, or the training data. We're not talking about getting a certain math question wrong (humans make mistakes like this). We're talking about ridiculous mistakes that are completely random and that humans would never make. I would even go as far to say that calling them mistakes is a stretch. The algorithm simply did not capture some mapping between the prompt and the output answer. This is not how humans make mistakes. Humans build knowledge in some kind of logical knowledge tree, and mistakes arise from how this mechanism works (which we don't know exactly how, but we can certainly conclude it is completely different than transformers and LLMs). Most humans don't make random mistakes but rather something like logical mistakes. tl;dr; It's clear to me LLMs make mistakes in such a way that exposes their simple underlying mechanism, as oppose to humans who make mistakes that contain rich layers of logical reasoning.
- ninetyninenine 1y ago
  >You can derive #3 from first principles of how the underlying gradient descent and backpropagation work. They approximate functions, therefore when it gets something wrong it means something went wrong with either the gradient descent or back propagation, or the training data. Then derive it from first principles. Use mathematical notation. You can't even draw this curve... it has so many dimensions. The crazy thing is you started hallucinating to me AFTER you told me you can derive it from first principles. You claimed you can derive it, then proceeded to NOT derive it. >We're not talking about getting a certain math question wrong (humans make mistakes like this). We're talking about ridiculous mistakes that are completely random and that humans would never make. I would even go as far to say that calling them mistakes is a stretch. The algorithm simply did not capture some mapping between the prompt and the output answer. Humans make plenty of ridiculous mistakes. Even so humans can lie. How do you know it's not lying? Again. Prove it. >This is not how humans make mistakes. Humans build knowledge in some kind of logical knowledge tree, and mistakes arise from how this mechanism works (which we don't know exactly how, but we can certainly conclude it is completely different than transformers and LLMs). Most humans don't make random mistakes but rather something like logical mistakes. Please derive how humans reason from first principles. I mean this by showing me experimental evidence that shows me the genesis of a signal traveling through the human brain and branching through billions of neurons to produce "reasoning". Oh you can't? Well it looks like you're just making an approximation here? Possibly an Hallucination. Sound familiar? >tl;dr; It's clear to me LLMs make mistakes in such a way that exposes their simple underlying mechanism, as oppose to humans who make mistakes that contain rich layers of logical reasoning. You had to do make several assumptions and leaps in creativity to arrive at your conclusion. It's an hallucination through and through.
asciimov 1y ago
Extraordinary claims require extraordinary evidence. I don't have to prove that the LLM can't reason. The claims that they can have yet to be proven.
- ninetyninenine 1y ago
  There was literally only One claim in this entire thread: "LLMs can't reason" No OTHER claim was made. So given the fact there was only one claim, Where does the burden of proof lie? Hint: the person who made the claim.
tombert 1y ago
Anti-AI people are kind of weird. I got put on probation on SomethingAwful because I created a thread talking about an AI project I was working on (An Icecast radio station that uses OpenAI to generate DJ chatter and commercials), and everyone acted like I was simultaneously a completely uncreative moron also also somehow stealing work from people I would have hired, like I was somehow depriving a DJ of a job by not hiring one. I am an unemployed software person who is building something for fun, "hiring a dedicated 24 hour DJ" was never on the table. In this thread, I noticed a lot of assertions that were just being accepted as axiomatically true, like asserting the AI is worse for the environment than humans doing the equivalent labor (which is not nearly as cut and dry), and that AI can't reason and that anything that involves any AI is inherently "theft". These assertions are completely unqualified and people just eat it up. I don't know if LLMs "reason" by any consistent definition of the word. They might, they might not, I'm not going to pretend to know, but I find it a little irritating how people just assert that they don't and people just gobble it up.
- GiorgioG 1y ago
  > Anti-AI people are kind of weird. I'm not Anti-AI, I'm Anti-shit that doesn't provide meaningful value and for building reliable professional software...they are not as valuable as "they" would have you believe.
- deleted 1y ago
  [deleted]
- ninetyninenine 1y ago
  Yeah it’s annoying af. This is supposed to be a forum where more intelligent people gather but people just don’t get it.
- idiot_slaiyer 1y ago
  op you asked, what about the energy expended to generate the art. the answer is this: the average human runs on about 2000-2500kcal of energy a day. if you were to spend 25 hours working on a piece of art, the energy expenditure is between 2000 and 2500kcal. if you were to generate one piece of AI art, the energy expended is several magnitudes higher. but if you understood basic mathematics and human biology, you would have understood this going into the discussion
deleted 1y ago
[deleted]
meroes 1y ago
2 years of RLHF I don’t see reasoning. It’s clearly a text-similarity game, plus training of course. My mental model to work best is not that I’m interacting with a thing with reasoning. That differs from my mental model of interacting with humans. I would be worse at my job if I interacted with LLMs as if they reasoned.
- ninetyninenine 1y ago
  Where's the evidence? You have none. Again you made a leap here with ZERO evidence which is equivalent to an hallucination.

staunton 1y ago

4. We don't actually have a coherent concept of what "reasoning" means in general. Same for "hallucination"...

ninetyninenine 1y ago
We do. Reasoning is clearly defined. Sentience or consciousness is not. Reasoning is basically the same as using logic to arrive at a conclusion when given a set of rules and axioms. Hallucination is producing a conclusion by not following rules or axioms.
- staunton 1y ago
  You could go further and say that anything a computation could ever possibly do is a matter of using logic to arrive at a conclusion given just the right rules and axioms. That would mean anything going beyond that (potentially "sentience" or "consciousness", possibly even -- as you seemed to suggest -- "hallucination", though apparently a computer program can do it) would have to be "magic" that's impossible to even describe using logic-based science, let alone simulate by a program. (Could we have science that transcends logic?) My original claim refers to the mundane situations where people get an LLM output which isn't what they wanted and go "see, it isn't reasoning! That's the problem!". For each such instance (or at least most of them), the "rules and axioms" that would have to be used to arrive at the desired conclusion would be different, sometimes incompatible, and in any case the people demanding "that it reason" wouldn't be able to spell those rules and axioms out, often even given years to do it, at least without allowing them to be obviously contradictory or trivially close to the specific conclusion or output they want. (Incidentally, that's why LLMs are interesting in the first place!) So sure, you can trick an LLM into failing on a syllogism problem, and that does mean "it can't reason" in a certain narrow sense, but I don't think that's what we're actually talking about when someone is contrasting "reasoning" and "hallucinating", which is what I was originally objecting to. That's what I meant when I said that we don't have a coherent concept of what "reasoning" means in general.

tbrownaw 1y ago

The is zero (legitimate) question about what LLMs do. They're math, not magic. They highlight fun philosophical / definitional questions like the Chinese room thought experiment, but that's it.

ninetyninenine 1y ago
And where is your evidence of this? Can you even prove this for a single query/response pair? Can you trace the genesis of the signals travelling through the neural network and prove to me via the structure that no reasoning was performed at all? You don't. You arrived at this claim with NO evidence. How do you arrive at a claim without evidence? There's a word for it. It's called an Hallucination. Humans, like LLMs, have trouble saying "I don't know."
- antihipocrat 1y ago
  Pretty sure the people creating the best models understand how these things work. Plenty of papers explain the process and you and I can create our own basic model ourselves from scratch with enough aptitude and drive. There is a lot of art involved with the best models. We aren't dealing with determinism with regard to the corpus used to train the model (too much information to curate for accuracy) nor the LLM output (probabilistic by design) nor the prompt input (humans language is dynamic and open to multiple interpretations) nor the human subjective assessment of the output. That there is a product available that manages to produce great results despite these huge challenges is amazing and this is the part that is not quantifiable - in the sense that the data scientists decisions made with regard to temperature settings etc are not derived from any fundamental property but more from an inspired group of people with qualitative instincts aligned with producing great results
  - ninetyninenine 1y ago
    https://futurism.com/anthropic-ceo-admits-ai-ignorance https://futurism.com/anthropic-ceo-admits-ai-ignorance Let me tell you something. You can go online, find a tutorial on how to make an LLM, and actually make one at home. The only thing stopping you from making an OpenAI scale LLM is compute resources. If you happen to make an LLM from scratch you won't know how it works, and neither do the people who are at OpenAi or anthropic.

antihipocrat 1y ago

I don't think the three options provided are exhaustive. For example: 4. LLMs can't reason, when they get something right, it's because the corpus of information used to create the model provides a very high probability that the response to that specific prompt is correct.

ninetyninenine 1y ago
They're not exhaustive, but add as many options as you want to the ones given. Most of those options can't be proven or disproven. We don't understand what's going on with LLMs. We've built something from scratch that we can't understand or control. That is what an LLM is.

proc0 1y ago

You can derive #3 from first principles of how the underlying gradient descent and backpropagation work. They approximate functions, therefore when it gets something wrong it means something went wrong with either the gradient descent or back propagation, or the training data. We're not talking about getting a certain math question wrong (humans make mistakes like this). We're talking about ridiculous mistakes that are completely random and that humans would never make. I would even go as far to say that calling them mistakes is a stretch. The algorithm simply did not capture some mapping between the prompt and the output answer. This is not how humans make mistakes. Humans build knowledge in some kind of logical knowledge tree, and mistakes arise from how this mechanism works (which we don't know exactly how, but we can certainly conclude it is completely different than transformers and LLMs). Most humans don't make random mistakes but rather something like logical mistakes. tl;dr; It's clear to me LLMs make mistakes in such a way that exposes their simple underlying mechanism, as oppose to humans who make mistakes that contain rich layers of logical reasoning.

ninetyninenine 1y ago
>You can derive #3 from first principles of how the underlying gradient descent and backpropagation work. They approximate functions, therefore when it gets something wrong it means something went wrong with either the gradient descent or back propagation, or the training data. Then derive it from first principles. Use mathematical notation. You can't even draw this curve... it has so many dimensions. The crazy thing is you started hallucinating to me AFTER you told me you can derive it from first principles. You claimed you can derive it, then proceeded to NOT derive it. >We're not talking about getting a certain math question wrong (humans make mistakes like this). We're talking about ridiculous mistakes that are completely random and that humans would never make. I would even go as far to say that calling them mistakes is a stretch. The algorithm simply did not capture some mapping between the prompt and the output answer. Humans make plenty of ridiculous mistakes. Even so humans can lie. How do you know it's not lying? Again. Prove it. >This is not how humans make mistakes. Humans build knowledge in some kind of logical knowledge tree, and mistakes arise from how this mechanism works (which we don't know exactly how, but we can certainly conclude it is completely different than transformers and LLMs). Most humans don't make random mistakes but rather something like logical mistakes. Please derive how humans reason from first principles. I mean this by showing me experimental evidence that shows me the genesis of a signal traveling through the human brain and branching through billions of neurons to produce "reasoning". Oh you can't? Well it looks like you're just making an approximation here? Possibly an Hallucination. Sound familiar? >tl;dr; It's clear to me LLMs make mistakes in such a way that exposes their simple underlying mechanism, as oppose to humans who make mistakes that contain rich layers of logical reasoning. You had to do make several assumptions and leaps in creativity to arrive at your conclusion. It's an hallucination through and through.

asciimov 1y ago

Extraordinary claims require extraordinary evidence. I don't have to prove that the LLM can't reason. The claims that they can have yet to be proven.

ninetyninenine 1y ago
There was literally only One claim in this entire thread: "LLMs can't reason" No OTHER claim was made. So given the fact there was only one claim, Where does the burden of proof lie? Hint: the person who made the claim.

tombert 1y ago

Anti-AI people are kind of weird. I got put on probation on SomethingAwful because I created a thread talking about an AI project I was working on (An Icecast radio station that uses OpenAI to generate DJ chatter and commercials), and everyone acted like I was simultaneously a completely uncreative moron also also somehow stealing work from people I would have hired, like I was somehow depriving a DJ of a job by not hiring one. I am an unemployed software person who is building something for fun, "hiring a dedicated 24 hour DJ" was never on the table. In this thread, I noticed a lot of assertions that were just being accepted as axiomatically true, like asserting the AI is worse for the environment than humans doing the equivalent labor (which is not nearly as cut and dry), and that AI can't reason and that anything that involves any AI is inherently "theft". These assertions are completely unqualified and people just eat it up. I don't know if LLMs "reason" by any consistent definition of the word. They might, they might not, I'm not going to pretend to know, but I find it a little irritating how people just assert that they don't and people just gobble it up.

GiorgioG 1y ago
> Anti-AI people are kind of weird. I'm not Anti-AI, I'm Anti-shit that doesn't provide meaningful value and for building reliable professional software...they are not as valuable as "they" would have you believe.
deleted 1y ago
[deleted]
ninetyninenine 1y ago
Yeah it’s annoying af. This is supposed to be a forum where more intelligent people gather but people just don’t get it.
idiot_slaiyer 1y ago
op you asked, what about the energy expended to generate the art. the answer is this: the average human runs on about 2000-2500kcal of energy a day. if you were to spend 25 hours working on a piece of art, the energy expenditure is between 2000 and 2500kcal. if you were to generate one piece of AI art, the energy expended is several magnitudes higher. but if you understood basic mathematics and human biology, you would have understood this going into the discussion

deleted 1y ago

[deleted]

meroes 1y ago

2 years of RLHF I don’t see reasoning. It’s clearly a text-similarity game, plus training of course. My mental model to work best is not that I’m interacting with a thing with reasoning. That differs from my mental model of interacting with humans. I would be worse at my job if I interacted with LLMs as if they reasoned.

ninetyninenine 1y ago
Where's the evidence? You have none. Again you made a leap here with ZERO evidence which is equivalent to an hallucination.