11 ms·

Neural networks are universal function approximators, transformers are universal sequence to sequence approximators. Crucially, the sequences here are the strea

by tovej 3y ago

Neural networks are universal function approximators, transformers are universal sequence to sequence approximators. Crucially, the sequences here are the stream tokens in an LLM. This means an LLM can do arbitrary language manipulation. If the reasoning task requires something that can't be done using language manipulation in embeddings-space, it won't be able to do it. Idk, maybe my mathematical intuition is wrong, but it seems obvious to me that if all you have is a function on a vector (representing a token stream), then the only transformations you can create are ones that can be expressed in that vector space. In other words, if you express a problem in a more complicated space (e.g. a visual problem, or an abstract algebra problem), you will not be able to solve it in the smaller token space, there's not enough information. You COULD first transform the vector into a different representation, perform the transformation there, and then encode back. This step would have to be different for every reasoning problem, so the underlying transformation to solution space would have to be generic and move into one of N reasoning subsystems. Also, if we hypothetically assume this limitation doesn't exist, we would still have the issue that a trained LLM represents a single such transformation function. If we want generic reasoning skills, the approximated function would have to be different for every reasoning problem. Even if we use a subset of the vectors degrees of freedoms to express each type of reasoning, it would make any truly reasoning LLM computationally intractable, because the embeddings space would blow up.

int_19h 3y ago
What couldn't you possibly represent as a bunch of numbers to the level of precision necessary to process it? We might not have sufficient compute power for LMs large enough for some tasks yet, but I don't see how it's different in principle. And yes, of course such representation would be different for different problems. We also have organs in our body that do that for various types of sensory inputs that ultimately all feed into the brain to be processed as quantized impulses. And GPT-4 is already supposed to be an "assembly of experts", i.e. different models trained for different tasks, with an orchestrating model weaving them together. This is also not a surprise, given that our brains aren't monolithic, either. You should be able to do it with a single model that's large enough, but it is likely more energy- and time-efficient to get there by putting smaller pieces together instead.
- tovej 3y ago
  It's just that there is no way for the model to map the token meanings to a iseful representation. Yes, in the general case, you can make a model for most types of logic problems (but be careful there, there are real problems you couldn't represent as "a bunch of numbers" and a single transformation), the issue is that the model cannot build that representation even for simple cases because it,'s used to operate at on language (and maybe some other fuzzy input type), which has to be _reliably_ interpreted to this number space; in different ways for different problems; without breaking the language processing.
stevenhuang 3y ago
> In other words, if you express a problem in a more complicated space (e.g. a visual problem, or an abstract algebra problem), you will not be able to solve it in the smaller token space, there's not enough information You're aware multimodel transformers do exactly this? https://github.com/bytedance/SALMONN https://github.com/bytedance/SALMONN
- tovej 3y ago
  Even that space is not large enough for the "any reasoning system" space. The functions also have to be representable in a way that the model can transform them usefully. If the model does not have the capability to transform into a different arbitrary space for every separate sentence, reasoning is not possible. Even then, reasoning would IMO be limited by other representation effects. But maybe we could have a discussion at that point. I just don't think transformers are going to get us to an intelligent agent, which is completely ok, they are a useful ML tool that we can use to create somewhat accurate models for complicated systems, and that's what they're designed for.
  - stevenhuang 3y ago
    > If the model does not have the capability to transform into a different arbitrary space for every separate sentence, reasoning is not possible. You're making an authoritative statement on how reasoning works based on conjecture. I remind you that this field is in active study. The "function" that sufficiently approximates the phenomenon of higher order reasoning may well mode collapse into a compact representation that need not exhaustively span the space of all "sentence representations". You can illustrate for yourself that it is not at all clear your premise is a necessary condition. Our brains can reason. Are you proposing its mechanism of action is by "transformation into a different arbitrary space for every sensory experience X"? I mean maybe? But what if there exists some higher order space that actually generalizes perfectly and captures the basis vectors of these seemingly orthogonal spaces? That can also be the case. Which is why in truth we don't yet know, and to say your view is correct is suspect. Hope you see now why this claim is an overstatement and missing some qualifiers.

by tovej 3y ago

int_19h 3y ago

What couldn't you possibly represent as a bunch of numbers to the level of precision necessary to process it? We might not have sufficient compute power for LMs large enough for some tasks yet, but I don't see how it's different in principle. And yes, of course such representation would be different for different problems. We also have organs in our body that do that for various types of sensory inputs that ultimately all feed into the brain to be processed as quantized impulses. And GPT-4 is already supposed to be an "assembly of experts", i.e. different models trained for different tasks, with an orchestrating model weaving them together. This is also not a surprise, given that our brains aren't monolithic, either. You should be able to do it with a single model that's large enough, but it is likely more energy- and time-efficient to get there by putting smaller pieces together instead.

tovej 3y ago
It's just that there is no way for the model to map the token meanings to a iseful representation. Yes, in the general case, you can make a model for most types of logic problems (but be careful there, there are real problems you couldn't represent as "a bunch of numbers" and a single transformation), the issue is that the model cannot build that representation even for simple cases because it,'s used to operate at on language (and maybe some other fuzzy input type), which has to be _reliably_ interpreted to this number space; in different ways for different problems; without breaking the language processing.

stevenhuang 3y ago

> In other words, if you express a problem in a more complicated space (e.g. a visual problem, or an abstract algebra problem), you will not be able to solve it in the smaller token space, there's not enough information You're aware multimodel transformers do exactly this? https://github.com/bytedance/SALMONN https://github.com/bytedance/SALMONN

tovej 3y ago
Even that space is not large enough for the "any reasoning system" space. The functions also have to be representable in a way that the model can transform them usefully. If the model does not have the capability to transform into a different arbitrary space for every separate sentence, reasoning is not possible. Even then, reasoning would IMO be limited by other representation effects. But maybe we could have a discussion at that point. I just don't think transformers are going to get us to an intelligent agent, which is completely ok, they are a useful ML tool that we can use to create somewhat accurate models for complicated systems, and that's what they're designed for.
- stevenhuang 3y ago
  > If the model does not have the capability to transform into a different arbitrary space for every separate sentence, reasoning is not possible. You're making an authoritative statement on how reasoning works based on conjecture. I remind you that this field is in active study. The "function" that sufficiently approximates the phenomenon of higher order reasoning may well mode collapse into a compact representation that need not exhaustively span the space of all "sentence representations". You can illustrate for yourself that it is not at all clear your premise is a necessary condition. Our brains can reason. Are you proposing its mechanism of action is by "transformation into a different arbitrary space for every sensory experience X"? I mean maybe? But what if there exists some higher order space that actually generalizes perfectly and captures the basis vectors of these seemingly orthogonal spaces? That can also be the case. Which is why in truth we don't yet know, and to say your view is correct is suspect. Hope you see now why this claim is an overstatement and missing some qualifiers.