5 ms·

Sure, it's not quite true, but it's close enough for the analysis to be correct. ie., if we take a family of problems: P1..n each statable with prompts P1..n,

Sure, it's not quite true, but it's close enough for the analysis to be correct. ie., if we take a family of problems: P1..n each statable with prompts P1..n, each having an 'essential complexity to solve' O(P1)...O(Pn) then it's trivial to show that no curve-fitting statistical algorithm has the 'correct' computational budgets associated to each. The budget is basically constant, and is not increasing correctly as the problem complexity increases. (Eg., consider the prompt: by applying Dijkstra's algorithm, find a path....on graph G1, G2..Gn -- with ever more complex graphs). It is right to say that step-by-step prompting expands the computational resources available to the LLM and therefore often improves the quality of the answer.. but it is incorrect to say that it does so by allocating computation according to that complexity. At best you could say that it is we, the human prompter, seeing the answer is bad, are increasing its budget -- so we are partially modelling the complexity of the problem for the LLM. If you could create an LLM that "prompted itself" according to the complexity of the problem, and measured its energy/time/etc. to be as we would expect.. then you'd be on firmer grounds with this claim. But i'd be inclined to deny this was possible given what an LLM is... ie., I claim that I can always find a family of problems P1..Pn where the computational distribution will rule out that reasoning is taking place. Why? Because the LLM is only sampling from a compression of training data, I deny it is at all sensitive to the meaning of the terms. So whereas a person will understand a novel problem (family) on the basis of its meaning, and expend time appropriately, the LLM is limited to sampling from prior examples of such problems in training data.

viraptor 2y ago
> ie., I claim that I can always find a family of problems P1..Pn where the computational distribution will rule out that reasoning is taking place. Finding edge cases is not a great way to evaluate probabilistic processes. You'll almost always find a pathological case. I'm sure we can find a series of questions of increasing difficulty where humans think they're of decreasing difficulty. That wouldn't prove humans don't reason. > If you could create an LLM that "prompted itself" according to the complexity of the problem You don't have to create one. We know that in-context learning is roughly equivalent to training, so you can add instructions about evaluating task difficulty and different approaches depending on the result to the system prompt. You'll see it's possible to do that estimate. Agents use that all the time to decide when to split a task into subtasks and when to proceed with solving. > So whereas a person will understand a novel problem (family) on the basis of its meaning, and expend time appropriately, the LLM is limited to sampling from prior examples of such problems in training data. Can you prove humans aren't just sampling from training data + preserving context? I don't believe anyone has really documented how human thought works at that level. I'm not saying it's true or false, just - do we actually know enough to validate the claim you made?
- mjburgess 2y ago
  > Finding edge cases is not a great way to evaluate probabilistic processes. These aren't "edge cases". They are proofs by contradiction: Claim 1: LLMs have a fixed computation budget per token Claim 2: LLMs engage in reasoning Claim 3: Reasoning is an unbounded process whose time is a function of the complexity of the problem Claim 4 Therefore: LLMs compute proportional to the complexity of the problem Problem: Contradiction in Claims 1, 4. There are an infinite number of "edge" cases to illustrate this point; but the point has nothing to do with LLMs failing on some algorithms. It's not an argument about LLM accuracy. > Can you prove humans aren't just sampling from training data + preserving context Yes. Above, Claim 1': Humans do not have a fixed computation budget per token. So there's no contradiction. It will take you an appropriately long time to say "Yes" or "No" as the task complexity increases. Hence you are actually engaged in reasoning. Though I don't find your style of argument good-faith, but wish fulfilment and arguments from ignorance. "I do not know anything about human beings, and I will tell you that you do not, so that I can preserve what I wish to be true" -- this is the position of almost all defenders of AI i've encountered, and I find it disagreeably religious.
  - viraptor 2y ago
    This falls apart at claim 1, because it's for a token, not the answer. You can start a reasoning process, but I don't know of anyone seriously claiming it happens in a single step. LLMs have a fixed budget per token, but the result you normally want is multiple tokens long. Engaging in chain-of-thought answer means a variable number of tokens, with the length depending on the complexity of the task. In this thread I mentioned the prompt + generated subtasks, so it's not like we're talking about single-token outputs. Or in a simple example, the answer to "using step by step reasoning, what's the colour of a white sheet" is significantly shorter than "using step by step reasoning, why is Chewbacca defence a fallacy". The more complex task automatically got more steps and more tokens dedicated to it. > Humans do not have a fixed computation budget per token. That's not even wrong. Humans don't operate on tokens. That just doesn't make sense.
    - mjburgess 2y ago

viraptor 2y ago

> ie., I claim that I can always find a family of problems P1..Pn where the computational distribution will rule out that reasoning is taking place. Finding edge cases is not a great way to evaluate probabilistic processes. You'll almost always find a pathological case. I'm sure we can find a series of questions of increasing difficulty where humans think they're of decreasing difficulty. That wouldn't prove humans don't reason. > If you could create an LLM that "prompted itself" according to the complexity of the problem You don't have to create one. We know that in-context learning is roughly equivalent to training, so you can add instructions about evaluating task difficulty and different approaches depending on the result to the system prompt. You'll see it's possible to do that estimate. Agents use that all the time to decide when to split a task into subtasks and when to proceed with solving. > So whereas a person will understand a novel problem (family) on the basis of its meaning, and expend time appropriately, the LLM is limited to sampling from prior examples of such problems in training data. Can you prove humans aren't just sampling from training data + preserving context? I don't believe anyone has really documented how human thought works at that level. I'm not saying it's true or false, just - do we actually know enough to validate the claim you made?

mjburgess 2y ago
> Finding edge cases is not a great way to evaluate probabilistic processes. These aren't "edge cases". They are proofs by contradiction: Claim 1: LLMs have a fixed computation budget per token Claim 2: LLMs engage in reasoning Claim 3: Reasoning is an unbounded process whose time is a function of the complexity of the problem Claim 4 Therefore: LLMs compute proportional to the complexity of the problem Problem: Contradiction in Claims 1, 4. There are an infinite number of "edge" cases to illustrate this point; but the point has nothing to do with LLMs failing on some algorithms. It's not an argument about LLM accuracy. > Can you prove humans aren't just sampling from training data + preserving context Yes. Above, Claim 1': Humans do not have a fixed computation budget per token. So there's no contradiction. It will take you an appropriately long time to say "Yes" or "No" as the task complexity increases. Hence you are actually engaged in reasoning. Though I don't find your style of argument good-faith, but wish fulfilment and arguments from ignorance. "I do not know anything about human beings, and I will tell you that you do not, so that I can preserve what I wish to be true" -- this is the position of almost all defenders of AI i've encountered, and I find it disagreeably religious.
- viraptor 2y ago
  This falls apart at claim 1, because it's for a token, not the answer. You can start a reasoning process, but I don't know of anyone seriously claiming it happens in a single step. LLMs have a fixed budget per token, but the result you normally want is multiple tokens long. Engaging in chain-of-thought answer means a variable number of tokens, with the length depending on the complexity of the task. In this thread I mentioned the prompt + generated subtasks, so it's not like we're talking about single-token outputs. Or in a simple example, the answer to "using step by step reasoning, what's the colour of a white sheet" is significantly shorter than "using step by step reasoning, why is Chewbacca defence a fallacy". The more complex task automatically got more steps and more tokens dedicated to it. > Humans do not have a fixed computation budget per token. That's not even wrong. Humans don't operate on tokens. That just doesn't make sense.
  - mjburgess 2y ago