4 ms·

Anyone else annoyed how the ML/AI community just adopted the word "reasoning" when it seems like it is being used very out of context when looking at what the m

Anyone else annoyed how the ML/AI community just adopted the word "reasoning" when it seems like it is being used very out of context when looking at what the model actually does?

ramoz 2y ago
These models take an instruction, along with any contextual information, and are trained to produce valid output. That production of output is a form of reasoning via _some_ type of logical processing. No? Maybe better to say computational reasoning. That’s a mouthful.
- nightski 2y ago
  Static computation is not reasoning (these models are not building up an argument from premises, they are merely finding statistically likely completions). Computational thinking/reasoning would be breaking down a problem into an algorithmic steps. The model is doing neither. I wouldn't confuse the fact that it can break it into steps if you ask it, because again that is just regurgitation. It's not going through that process without your prompt. That is not part of its process to arrive at an answer.
  - ramoz 2y ago
    The point is emergent capabilities in LLMs go beyond statistical extrapolation, as they demonstrate reasoning by combining learned patterns. When asked, “If Alice has 3 apples and gives 2 to Bob, how many does she have left?”, the model doesn’t just retrieve a memorized answer—it infers the logical steps (subtracting 2 from 3) to generate the correct result, showcasing reasoning built on the interplay of its scale and architecture rather than explicit data recall.
  - thelastbender12 2y ago
    I kinda agree with you but I can also see why it isn't that far from "reasoning" in the sense humans do it. To wit, if I am doing a high school geometry proof, I come up with a sequence of steps. If the proof is correct, each step follows logically from the one before it. However, when I go from step 2 to step 3, there are multiple options for step-3 I could have chose. Is it so different from a "most-likely-prediction" an LLM makes? I suppose the difference is humans can filter out logically-incorrect steps, or prune chains-of-steps that won't lead to the actual theorem quicker. But an LLM predictor coupled with a verifier doesn't feel that different from it.
  - int_19h 2y ago
    Models trained to do chain-of-thought do it without any specific prompting, and a specific system prompt that enables this behavior is often part of the setup, because that is what the model was trained on. I don't see how that is "regurgitation", either, if it performs the reasoning steps first, and only then arrives at the answer.
w10-1 2y ago
Does it help to explore the annoyance using gap analysis? I think of it as heuristics. As with humans, it's the pragmatic "whatever seems to work" where "seems" is determined via training. It's neither reasoning from first principles (system 1) nor just selecting the most likely/prevalent answer (system 2). And chaining heuristics doesn't make it reasoning, either. But where there's evidence that it's working from a model, then it becomes interesting, and begins to comply with classical epistemology wrt "reasoning". Unfortunately, information theory seems to treat any compression as a model leading to some pretty subtle delusions.
resource_waste 2y ago
These kind of simplifications continue to make me an expert in LLM applications. So... its a trade secret to know how it actually works...

Anyone else annoyed how the ML/AI community just adopted the word "reasoning" when it seems like it is being used very out of context when looking at what the model actually does?

ramoz 2y ago

These models take an instruction, along with any contextual information, and are trained to produce valid output. That production of output is a form of reasoning via _some_ type of logical processing. No? Maybe better to say computational reasoning. That’s a mouthful.

nightski 2y ago
Static computation is not reasoning (these models are not building up an argument from premises, they are merely finding statistically likely completions). Computational thinking/reasoning would be breaking down a problem into an algorithmic steps. The model is doing neither. I wouldn't confuse the fact that it can break it into steps if you ask it, because again that is just regurgitation. It's not going through that process without your prompt. That is not part of its process to arrive at an answer.
- ramoz 2y ago
  The point is emergent capabilities in LLMs go beyond statistical extrapolation, as they demonstrate reasoning by combining learned patterns. When asked, “If Alice has 3 apples and gives 2 to Bob, how many does she have left?”, the model doesn’t just retrieve a memorized answer—it infers the logical steps (subtracting 2 from 3) to generate the correct result, showcasing reasoning built on the interplay of its scale and architecture rather than explicit data recall.
- thelastbender12 2y ago
  I kinda agree with you but I can also see why it isn't that far from "reasoning" in the sense humans do it. To wit, if I am doing a high school geometry proof, I come up with a sequence of steps. If the proof is correct, each step follows logically from the one before it. However, when I go from step 2 to step 3, there are multiple options for step-3 I could have chose. Is it so different from a "most-likely-prediction" an LLM makes? I suppose the difference is humans can filter out logically-incorrect steps, or prune chains-of-steps that won't lead to the actual theorem quicker. But an LLM predictor coupled with a verifier doesn't feel that different from it.
- int_19h 2y ago
  Models trained to do chain-of-thought do it without any specific prompting, and a specific system prompt that enables this behavior is often part of the setup, because that is what the model was trained on. I don't see how that is "regurgitation", either, if it performs the reasoning steps first, and only then arrives at the answer.

w10-1 2y ago

Does it help to explore the annoyance using gap analysis? I think of it as heuristics. As with humans, it's the pragmatic "whatever seems to work" where "seems" is determined via training. It's neither reasoning from first principles (system 1) nor just selecting the most likely/prevalent answer (system 2). And chaining heuristics doesn't make it reasoning, either. But where there's evidence that it's working from a model, then it becomes interesting, and begins to comply with classical epistemology wrt "reasoning". Unfortunately, information theory seems to treat any compression as a model leading to some pretty subtle delusions.

resource_waste 2y ago

These kind of simplifications continue to make me an expert in LLM applications. So... its a trade secret to know how it actually works...