4 ms·

So it seems 'obvious' to me that a network about 50 layers deep (for example) can only reason about symbolic questions for 50 'steps' (in quotes because it's no

So it seems 'obvious' to me that a network about 50 layers deep (for example) can only reason about symbolic questions for 50 'steps' (in quotes because it's not a step as we think about it). It only seems there's more complexity because it's 50 steps in one or more learned subspaces that the model has been trained in (which might mean the model can accomplish more than one 'human step' in its 'step'). Humans (well intelligent humans at least) seem able to obviously reason beyond those steps, but we all know it requires real thinking and deliberation and perhaps a notepad to be able to do that. It's quite something to, for example, expect ChatGPT to be able to correctly do 4 digit multiplications without any thought or recourse to 'paper' when very few human beings can do that.

blackbear_ 3y ago
This paper does indeed follow your intuition to investigate the limits of transformers on compositional tasks (i.e., those that require multi-step reasoning, including your multiplication example): https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2305.18654 > Our empirical findings suggest that transformer LLMs solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills. To round off our empirical study, we provide theoretical arguments on abstract multi-step reasoning problems that highlight how autoregressive generations' performance can rapidly decay with increased task complexity.
- anon291 3y ago
  Ah good... This is definitely a research path I've been looking into. Great to see someone else has already gone there!
- visarga 3y ago
  Maybe the Skill Mix paper is relevant here. They define a list of 100 skills, and then randomly sample tuples of n skills (usually less than 6) and generate a test example using those skills. Apparently only GPT-4 (at the time of the paper) was able to compose 5 skills, the other models just 3 or 2. Beyond 5 skills even GPT-4 was doing much worse. The interesting finding of the paper is that GPT-4 couldn't have seen all the (topic, skill-tuple) combinations in the training set. If you have 10,000 examples on a topic, and use 5 out of 100 skills, you would need 100^5 training examples to cover all combinations. In conclusion GPT-4 generalizes to new skill combinations, thus it is not a stochastic parrot. https://arxiv.org/abs/2310.17567 https://arxiv.org/abs/2310.17567
radarsat1 3y ago
This is true but you have to also consider the autoregressive component. In your example, it's 50 steps per iteration of the model, where the model is executed once for each token in the output. So practically speaking it's a bit more complicated to calculate how much the model can "think". Of course once a token is output it is committed to that (in the most basic scenario), but that doesn't mean it is not still "thinking" as it produces subsequent tokens. > perhaps a notepad Exactly, the context and previously output tokens can be considered such a notepad since they are input for the next steps of the model.
- Closi 3y ago
  Agreed - also prompt engineering encourages LLM's to do this too (i.e. asking the LLM to explain the steps it will take to solve an answer, prior to answering - e.g. Zero-Shot CoT 'Let's think step by step')
  - carlmr 3y ago
    Teachers asking students to show their work are human prompt engineers.
- anon291 3y ago
  So part of my general issue with this kind of thinking is that, if we take this as the main means of creating complexity, then shorter prompts are worse for reasoning than longer ones, because longer ones automatically give the model more 'space', to think. Now, I realize that the research community knows this, but I like papers like this that explicitly seek ways to enable the model to 'breathe' a bit,.
  - danielmarkbruce 3y ago
    This doesn't make sense. The responses can be long.
visarga 3y ago
You are missing an important detail here - number of tokens - yes, you have 50 "steps" in network depth, but you could have extra tokens. Assuming you don't run out of tape, there is no reason for LLMs to be limited to simple operations.
danielmarkbruce 3y ago
This doesn't make a lot of sense when you consider how backprop works. Layers aren't limited to working independently. This also doesn't make a lot of sense when you consider models are autoregressive.

blackbear_ 3y ago

This paper does indeed follow your intuition to investigate the limits of transformers on compositional tasks (i.e., those that require multi-step reasoning, including your multiplication example): https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2305.18654 > Our empirical findings suggest that transformer LLMs solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills. To round off our empirical study, we provide theoretical arguments on abstract multi-step reasoning problems that highlight how autoregressive generations' performance can rapidly decay with increased task complexity.

anon291 3y ago
Ah good... This is definitely a research path I've been looking into. Great to see someone else has already gone there!
visarga 3y ago
Maybe the Skill Mix paper is relevant here. They define a list of 100 skills, and then randomly sample tuples of n skills (usually less than 6) and generate a test example using those skills. Apparently only GPT-4 (at the time of the paper) was able to compose 5 skills, the other models just 3 or 2. Beyond 5 skills even GPT-4 was doing much worse. The interesting finding of the paper is that GPT-4 couldn't have seen all the (topic, skill-tuple) combinations in the training set. If you have 10,000 examples on a topic, and use 5 out of 100 skills, you would need 100^5 training examples to cover all combinations. In conclusion GPT-4 generalizes to new skill combinations, thus it is not a stochastic parrot. https://arxiv.org/abs/2310.17567 https://arxiv.org/abs/2310.17567

radarsat1 3y ago

This is true but you have to also consider the autoregressive component. In your example, it's 50 steps per iteration of the model, where the model is executed once for each token in the output. So practically speaking it's a bit more complicated to calculate how much the model can "think". Of course once a token is output it is committed to that (in the most basic scenario), but that doesn't mean it is not still "thinking" as it produces subsequent tokens. > perhaps a notepad Exactly, the context and previously output tokens can be considered such a notepad since they are input for the next steps of the model.

Closi 3y ago
Agreed - also prompt engineering encourages LLM's to do this too (i.e. asking the LLM to explain the steps it will take to solve an answer, prior to answering - e.g. Zero-Shot CoT 'Let's think step by step')
- carlmr 3y ago
  Teachers asking students to show their work are human prompt engineers.
anon291 3y ago
So part of my general issue with this kind of thinking is that, if we take this as the main means of creating complexity, then shorter prompts are worse for reasoning than longer ones, because longer ones automatically give the model more 'space', to think. Now, I realize that the research community knows this, but I like papers like this that explicitly seek ways to enable the model to 'breathe' a bit,.
- danielmarkbruce 3y ago
  This doesn't make sense. The responses can be long.

visarga 3y ago

You are missing an important detail here - number of tokens - yes, you have 50 "steps" in network depth, but you could have extra tokens. Assuming you don't run out of tape, there is no reason for LLMs to be limited to simple operations.

danielmarkbruce 3y ago

This doesn't make a lot of sense when you consider how backprop works. Layers aren't limited to working independently. This also doesn't make a lot of sense when you consider models are autoregressive.