7 ms·

It would be really funny if the whole "Go step by step and think it through" trick only worked because it was adding more tokens for the AI to reuse.

by kaibee 3y ago

famouswaffles 3y ago
I'm sure that's part of it but also in the same way out of distribution tokens can derail its output, certain tokens will guide it. In the paper, the model doesn't really learn to use padding tokens effectively till after training so it's the only thing.
justinzollars 3y ago
is this your prompt? "Go step by step and think it through".
- joshka 3y ago
  This seems like a slight variation of "Let’s think step by step" [1] or "Let’s work this out in a step by step way to be sure we have the right answer" [2]. There's a good review of a bunch of these in https://arxiv.org/abs/2305.02897 https://arxiv.org/abs/2305.02897 1: https://arxiv.org/abs/2205.11916 https://arxiv.org/abs/2205.11916 2: https://openreview.net/forum?id=92gvk82DE- https://openreview.net/forum?id=92gvk82DE-
danbruc 3y ago
It does. Those models are glorified look-up tables, output = answers[input]. It would of course be insane to try to store this mapping explicitly but it is still there implicitly in the weight of the network. But no matter what, it is just feed-forward, the system will always be limited when it comes to learning things that require iteration. This can be overcome to some extend by doing something like chain of thought that allows dumping some state into the output stream that will then be fed back through the autoregressive loop wrapping the network. Pause tokens are more or less the same thing just that they do not pollute the output with the internal thoughts of the network.
- famouswaffles 3y ago
  Filler tokens don't work anywhere near as well on GPT-4 as CoT so that's obviously not the only reason. https://www.lesswrong.com/posts/oSZ2xTxEMZh9f3Yaz/llms-are-mostly-not-helped-by-filler-tokens https://www.lesswrong.com/posts/oSZ2xTxEMZh9f3Yaz/llms-are-m...
  - danbruc 3y ago
    I did not read the entire linked article but For instance, I tested whether asking a model to produce "a b c d e" before solving a math problem improves performance. is of course not what you would expect to work. It is not about dumping meaningless stuff into the input but intermediate results. If you have to add two long numbers but are only capable to add single digits, you want to dump the carry into the output and get this fed back as input so that you can take it into account when adding up the next two digits. Dumping unrelated stuff into the output will of course not help. Chain of thought does exactly this, it solves part of the overall problem and feeds that back for consideration when tackling the next piece of the problem.
    - famouswaffles 3y ago
      Yes and I'm saying even that doesn't work as robustly as CoT across different tasks. CoT is so obviously not just about padding for extra compute. Certain tokens will guide it to more accurate responses just as much as others derail it out of distribution. This should be obvious. Literally this paper, you can see the model only really learns how to use the padding tokens effectively after training.
      - danbruc 3y ago
        Yes and I'm saying even that doesn't work as robustly as CoT across different tasks. What does not work? CoT is so obviously not just about padding for extra compute. In a certain way it is but the crucial point is that an autoregressive neural network is pretty much a recurrent neural network and chain of thought encourages the use of the feedback path.
deleted 3y ago
[deleted]
3abiton 3y ago
Is this the same as pause tokens?

It would be really funny if the whole "Go step by step and think it through" trick only worked because it was adding more tokens for the AI to reuse.

by kaibee 3y ago

It would be really funny if the whole "Go step by step and think it through" trick only worked because it was adding more tokens for the AI to reuse.

famouswaffles 3y ago

I'm sure that's part of it but also in the same way out of distribution tokens can derail its output, certain tokens will guide it. In the paper, the model doesn't really learn to use padding tokens effectively till after training so it's the only thing.

justinzollars 3y ago

is this your prompt? "Go step by step and think it through".

joshka 3y ago
This seems like a slight variation of "Let’s think step by step" [1] or "Let’s work this out in a step by step way to be sure we have the right answer" [2]. There's a good review of a bunch of these in https://arxiv.org/abs/2305.02897 https://arxiv.org/abs/2305.02897 1: https://arxiv.org/abs/2205.11916 https://arxiv.org/abs/2205.11916 2: https://openreview.net/forum?id=92gvk82DE- https://openreview.net/forum?id=92gvk82DE-

danbruc 3y ago

It does. Those models are glorified look-up tables, output = answers[input]. It would of course be insane to try to store this mapping explicitly but it is still there implicitly in the weight of the network. But no matter what, it is just feed-forward, the system will always be limited when it comes to learning things that require iteration. This can be overcome to some extend by doing something like chain of thought that allows dumping some state into the output stream that will then be fed back through the autoregressive loop wrapping the network. Pause tokens are more or less the same thing just that they do not pollute the output with the internal thoughts of the network.

famouswaffles 3y ago
Filler tokens don't work anywhere near as well on GPT-4 as CoT so that's obviously not the only reason. https://www.lesswrong.com/posts/oSZ2xTxEMZh9f3Yaz/llms-are-mostly-not-helped-by-filler-tokens https://www.lesswrong.com/posts/oSZ2xTxEMZh9f3Yaz/llms-are-m...
- danbruc 3y ago
  I did not read the entire linked article but For instance, I tested whether asking a model to produce "a b c d e" before solving a math problem improves performance. is of course not what you would expect to work. It is not about dumping meaningless stuff into the input but intermediate results. If you have to add two long numbers but are only capable to add single digits, you want to dump the carry into the output and get this fed back as input so that you can take it into account when adding up the next two digits. Dumping unrelated stuff into the output will of course not help. Chain of thought does exactly this, it solves part of the overall problem and feeds that back for consideration when tackling the next piece of the problem.
  - famouswaffles 3y ago
    Yes and I'm saying even that doesn't work as robustly as CoT across different tasks. CoT is so obviously not just about padding for extra compute. Certain tokens will guide it to more accurate responses just as much as others derail it out of distribution. This should be obvious. Literally this paper, you can see the model only really learns how to use the padding tokens effectively after training.
    - danbruc 3y ago
      Yes and I'm saying even that doesn't work as robustly as CoT across different tasks. What does not work? CoT is so obviously not just about padding for extra compute. In a certain way it is but the crucial point is that an autoregressive neural network is pretty much a recurrent neural network and chain of thought encourages the use of the feedback path.

deleted 3y ago

[deleted]

3abiton 3y ago

Is this the same as pause tokens?