3 ms·

There cannot be any reasoning embedded in the model. The algorithm is literally "predict the most likely next token". Anything beyond that is just patterns in t

by gmueckl 4mo ago

There cannot be any reasoning embedded in the model. The algorithm is literally "predict the most likely next token". Anything beyond that is just patterns in the predictions fooling us humans into ascribing more to the system than it is actually producing.

thepasch 4mo ago
> The algorithm is literally "predict the most likely next token". That's confusing the training objective with the learned behavior. It's like saying "Stockfish's algorithm is literally 'minimize this number', and therefore, it can't actually play Chess."
- gmueckl 4mo ago
  Not a valid comparison. Chess algorithms are built around the rules of chess, most notably the turn taking nature of the game (min/max with alpha/beta pruning based on lists of valid moves in any position).
  - etskinner 4mo ago
    Who's to say the "rules of reasoning" aren't just predicting the next thing that an intelligent person (you) would do? Emergent behavior isn't magic, it's just emergent.
wizzwizz4 4mo ago
LLMs are great big finite state machines. Finite state machines can perform mechanical reasoning. A priori, there can be reasoning embedded in the model. I agree that (these) LLMs don't generally reason (even when they're writing words like "I reason that, since X, we have Y, therefore Z"), but that's not because a model inherently cannot do that.
aspenmartin 4mo ago
This is just a misconception of how LLMs work and also what reasoning is. “There cannot be any reasoning embedded in the model” a strong statement, what do you mean by reasoning because by any reasonable definition I’m aware of, they clearly are able to exhibit reasoning. The fact that the pre training objective is next token loss has nothing to do with capabilities or their ability to reason. To be highly successful at next token prediction you NEED to reason. I’m quite confused here.
- gmueckl 4mo ago
  LLM output produces the illusion of reasoning. The underlying computation, however, is not reasoning.
  - aspenmartin 4mo ago
    If you don’t mind actually taking a few more words to be more specific that would be helpful because what you’re saying doesn’t really make sense at all. You don’t need to trust that the reasoning traces are all faithful representation of an internal reasoning trace. Plenty of other ways to probe models (see anthropics work using circuit tracing).
    - gmueckl 4mo ago
      What else is there to say? LLMs can at most regurgitate approximations of human reasoning steps in the limited forms in which they may be expressed in the training data or interpolations thereof. That's the core essence of what they are. There is no proper reasoning to be found.
      - aspenmartin 4mo ago
        "at most" is wrong. RL with verifiable rewards takes you beyond quality and skills represented in training data, I'm not aware of meaningful fundamental limits here if you scale compute enough even though right now it's highly sample inefficient. Since you refuse to actually define what you consider to be reasoning let me at least put one out there: a system exhibits reasoning when an answer depends on nontrivial intermediate computation over the problem. If you find problems with this, fine, but just make an effort to contribute an alternative. If you increase test time compute you get better performance. If the model was just "interpolating" this wouldn't really work would it? Models can do FrontierMath expert problems (unpublished, expert authored, peer reviewed math problems) that require an insane amount of compositional reasoning. If they were regurgitating training data, that wouldn't really work would it? Chain of thought, while not always faithful to internal computation, improves performance. If the models were just regurgitating information, it wouldn't work that well would it? "regurgitating training data" is also of course misleading. Yea they can memorize parts of the training data, but they generalize very well.
linzhangrun 4mo ago
Quantity change leads to quality change. You can check out this Kurzgesagt video on emergence: https://www.youtube.com/watch?v=16W7c0mb-rE https://www.youtube.com/watch?v=16W7c0mb-rE

by gmueckl 4mo ago

thepasch 4mo ago

> The algorithm is literally "predict the most likely next token". That's confusing the training objective with the learned behavior. It's like saying "Stockfish's algorithm is literally 'minimize this number', and therefore, it can't actually play Chess."

gmueckl 4mo ago
Not a valid comparison. Chess algorithms are built around the rules of chess, most notably the turn taking nature of the game (min/max with alpha/beta pruning based on lists of valid moves in any position).
- etskinner 4mo ago
  Who's to say the "rules of reasoning" aren't just predicting the next thing that an intelligent person (you) would do? Emergent behavior isn't magic, it's just emergent.

wizzwizz4 4mo ago

LLMs are great big finite state machines. Finite state machines can perform mechanical reasoning. A priori, there can be reasoning embedded in the model. I agree that (these) LLMs don't generally reason (even when they're writing words like "I reason that, since X, we have Y, therefore Z"), but that's not because a model inherently cannot do that.

aspenmartin 4mo ago

This is just a misconception of how LLMs work and also what reasoning is. “There cannot be any reasoning embedded in the model” a strong statement, what do you mean by reasoning because by any reasonable definition I’m aware of, they clearly are able to exhibit reasoning. The fact that the pre training objective is next token loss has nothing to do with capabilities or their ability to reason. To be highly successful at next token prediction you NEED to reason. I’m quite confused here.

gmueckl 4mo ago
LLM output produces the illusion of reasoning. The underlying computation, however, is not reasoning.
- aspenmartin 4mo ago
  If you don’t mind actually taking a few more words to be more specific that would be helpful because what you’re saying doesn’t really make sense at all. You don’t need to trust that the reasoning traces are all faithful representation of an internal reasoning trace. Plenty of other ways to probe models (see anthropics work using circuit tracing).
  - gmueckl 4mo ago
    What else is there to say? LLMs can at most regurgitate approximations of human reasoning steps in the limited forms in which they may be expressed in the training data or interpolations thereof. That's the core essence of what they are. There is no proper reasoning to be found.
    - aspenmartin 4mo ago
      "at most" is wrong. RL with verifiable rewards takes you beyond quality and skills represented in training data, I'm not aware of meaningful fundamental limits here if you scale compute enough even though right now it's highly sample inefficient. Since you refuse to actually define what you consider to be reasoning let me at least put one out there: a system exhibits reasoning when an answer depends on nontrivial intermediate computation over the problem. If you find problems with this, fine, but just make an effort to contribute an alternative. If you increase test time compute you get better performance. If the model was just "interpolating" this wouldn't really work would it? Models can do FrontierMath expert problems (unpublished, expert authored, peer reviewed math problems) that require an insane amount of compositional reasoning. If they were regurgitating training data, that wouldn't really work would it? Chain of thought, while not always faithful to internal computation, improves performance. If the models were just regurgitating information, it wouldn't work that well would it? "regurgitating training data" is also of course misleading. Yea they can memorize parts of the training data, but they generalize very well.

linzhangrun 4mo ago

Quantity change leads to quality change. You can check out this Kurzgesagt video on emergence: https://www.youtube.com/watch?v=16W7c0mb-rE https://www.youtube.com/watch?v=16W7c0mb-rE