6 ms·

But then it's an LLM augmented with a tool that is solving the problem, and not the LLM itself.

by rain1 3y ago

espadrine 3y ago
The transformer neural network itself only computes the next token probabilities; the sampling of the token, and the mechanism by which the token it generates is concatenated to the input to make it generate the token after, are technically not part of the LLM. So we can either study the LLM strictly from single-token outputs, or from how it combines into larger constructs. The former excludes not just CoT, but a large part of the benchmarks on which they are tested. Looking at it another way, how would humans solve PARITY? I don’t think anyone would read the string, close their eyes, then give an answer without a single thought. At the very least, I would place my finger on each digit successively, effectively using a tool that acts as a memory storage (storing a cursor through the string). The reason I would do that, is that I would have devised an algorithm that requires going through digits one at a time.
magicalhippo 3y ago
And? Our brains aren't just doing the exact same things all over, we have centers focused on certain tasks. For example we have[1] neurons that do addition and some others that do subtraction. Our brains are limited to using neurons as building blocks, but why should an AI be limited in the same way? [1]: https://www.sciencedaily.com/releases/2022/02/220214121241.htm https://www.sciencedaily.com/releases/2022/02/220214121241.h...
- eru 3y ago
  Furthermore, people also use tools like an abacus, without any detriment to their intelligence.
- rng_civ 3y ago
  The whole point of this post is that we are trying to: * Classify the computational power of Transformers (when it stumbles on certain easier problems but can solve harder ones) * Find a "minimal" change to the Transformer that would allow it to compute these problems. Solving these 2 problems by giving LLMs arbitrary access to external plugins is a cop out. You would not: * Call youself a chef just because you own a restaraunt (you need to cook too!) * Or (more program-y), say that C code meets Rust's memory safety standards simply because you can write the main function in C and write the rest of the program in Rust Allowing arbitrary external plugins seems absurdly overkill and not 'minimal' (although that doesn't mean it isn't interesting from a practical perspective!), which is what I assumed that the rain1 was originally pointing out.
  - magicalhippo 3y ago
    My point is why must transformers be able to do everything? edit: I don't mean to dismiss the work trying to figure out what they can do. That seems reasonable and valuable. It's just, we're not trying to figure out how to tweak QR decomposition to solve arbitrary equations. It's a tool, a powerful tool, but it has some clear limitations.
SanderNL 3y ago
That is a problem how?
shawntan 3y ago
Just as a general response to the objections to this: The goal was to use XOR and PARITY as simple tasks and formal languages that exhibit a larger problem with the architecture. The unfortunate side-effect of this is that the most straightforward thing to relate them to are human reasoning tasks like arithmetic, for which the most straightforward thing to say is then: "We use calculators too!". Or worse, over-focusing on the PARITY problem, specifically. The PARITY problem requires a kind of circuit that flips a bit on and off alternatively whenever we see a '1'/ signal. Is there a subconscious process that we employ in our brains that implements this circuit? Might some aspects of linguistic processing require this? (Michael Hahn brings up 'iterated negation' here: https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00306/43545/Theoretical-Limitations-of-Self-Attention-in https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00306...) All this to say, maintaining and updating latent state may be an implicit requirement for certain tasks, and that would be more likely learned from data than from an explicit prompt or outsourcing to a calculator.
jstanley 3y ago
So? Lots of systems are made of multiple components.
benlivengood 3y ago
Humans solve all of their problems using at least one tool. Unless you are naked in the wildlands holding nothing, you are probably using a tool or a result of a tool (landscaping, clothing, etc.)

But then it's an LLM augmented with a tool that is solving the problem, and not the LLM itself.

by rain1 3y ago

But then it's an LLM augmented with a tool that is solving the problem, and not the LLM itself.

The transformer neural network itself only computes the next token probabilities; the sampling of the token, and the mechanism by which the token it generates is concatenated to the input to make it generate the token after, are technically not part of the LLM. So we can either study the LLM strictly from single-token outputs, or from how it combines into larger constructs. The former excludes not just CoT, but a large part of the benchmarks on which they are tested. Looking at it another way, how would humans solve PARITY? I don’t think anyone would read the string, close their eyes, then give an answer without a single thought. At the very least, I would place my finger on each digit successively, effectively using a tool that acts as a memory storage (storing a cursor through the string). The reason I would do that, is that I would have devised an algorithm that requires going through digits one at a time.

magicalhippo 3y ago

And? Our brains aren't just doing the exact same things all over, we have centers focused on certain tasks. For example we have[1] neurons that do addition and some others that do subtraction. Our brains are limited to using neurons as building blocks, but why should an AI be limited in the same way? [1]: https://www.sciencedaily.com/releases/2022/02/220214121241.htm https://www.sciencedaily.com/releases/2022/02/220214121241.h...

eru 3y ago
Furthermore, people also use tools like an abacus, without any detriment to their intelligence.
rng_civ 3y ago
The whole point of this post is that we are trying to: * Classify the computational power of Transformers (when it stumbles on certain easier problems but can solve harder ones) * Find a "minimal" change to the Transformer that would allow it to compute these problems. Solving these 2 problems by giving LLMs arbitrary access to external plugins is a cop out. You would not: * Call youself a chef just because you own a restaraunt (you need to cook too!) * Or (more program-y), say that C code meets Rust's memory safety standards simply because you can write the main function in C and write the rest of the program in Rust Allowing arbitrary external plugins seems absurdly overkill and not 'minimal' (although that doesn't mean it isn't interesting from a practical perspective!), which is what I assumed that the rain1 was originally pointing out.
- magicalhippo 3y ago
  My point is why must transformers be able to do everything? edit: I don't mean to dismiss the work trying to figure out what they can do. That seems reasonable and valuable. It's just, we're not trying to figure out how to tweak QR decomposition to solve arbitrary equations. It's a tool, a powerful tool, but it has some clear limitations.

SanderNL 3y ago

That is a problem how?

shawntan 3y ago

Just as a general response to the objections to this: The goal was to use XOR and PARITY as simple tasks and formal languages that exhibit a larger problem with the architecture. The unfortunate side-effect of this is that the most straightforward thing to relate them to are human reasoning tasks like arithmetic, for which the most straightforward thing to say is then: "We use calculators too!". Or worse, over-focusing on the PARITY problem, specifically. The PARITY problem requires a kind of circuit that flips a bit on and off alternatively whenever we see a '1'/ signal. Is there a subconscious process that we employ in our brains that implements this circuit? Might some aspects of linguistic processing require this? (Michael Hahn brings up 'iterated negation' here: https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00306/43545/Theoretical-Limitations-of-Self-Attention-in https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00306...) All this to say, maintaining and updating latent state may be an implicit requirement for certain tasks, and that would be more likely learned from data than from an explicit prompt or outsourcing to a calculator.

jstanley 3y ago

So? Lots of systems are made of multiple components.

benlivengood 3y ago

Humans solve all of their problems using at least one tool. Unless you are naked in the wildlands holding nothing, you are probably using a tool or a result of a tool (landscaping, clothing, etc.)