4 ms·

Ok, I'll bite. I personally don't think human communication is simply pattern matching (e.g. we have drives, motives, relationships that we want to tell each o

by tovej 3y ago

Ok, I'll bite. I personally don't think human communication is simply pattern matching (e.g. we have drives, motives, relationships that we want to tell each other, these are internal systems, outside the visible band of communication). That's a little off topic though, my post was not about communication, but reasoning. Pattern matching in LLMs is specifically at the language level, as in "predict the next token in a stream", I should have probably specified this bit. You seem to refer to a more general kind of pattern matching when you say, where we're looking at a complicated structural model within which humans reason about the behavior of different system components and their interactions. There is no way to form an internal representation of such systems within an LLM, which is why I said "of course" it can't reason. An LLM is not designed with this capability in mind. If all you have is language, and predictive text, there is no way to use this to simulate and predict the rules in a higher-dimensional space than "stream of tokens", or let's put it another way: there would need to be an extremely high correlation between the pattern in the problem space and in language space for the likelihood of the model guessing right to be anywhere near human levels.

stevenhuang 3y ago
> There is no way to form an internal representation of such systems within an LLM, which is why I said "of course" it can't reason. An LLM is not designed with this capability in mind. You claim this as if it is a foregone conclusion yet it is anything but. This is an active area of study, even within human cognition https://en.m.wikipedia.org/wiki/Predictive_coding https://en.m.wikipedia.org/wiki/Predictive_coding A complex system clearly does not need to be designed to evoke some behavior. Just look to the concept of emergent phenomenon. See evolution through natural selection as an obvious counter example. Evolution selects for fitness and not reasoning ability. Yet here we are. It is not known if the same isn't happening now for the transformer model. LLMs are universal function approximators. If reasoning ability is at all representable through such a process, it is possible for LLMs to find it. https://en.m.wikipedia.org/wiki/Universal_approximation_theorem https://en.m.wikipedia.org/wiki/Universal_approximation_theo...
- tovej 3y ago
  Neural networks are universal function approximators, transformers are universal sequence to sequence approximators. Crucially, the sequences here are the stream tokens in an LLM. This means an LLM can do arbitrary language manipulation. If the reasoning task requires something that can't be done using language manipulation in embeddings-space, it won't be able to do it. Idk, maybe my mathematical intuition is wrong, but it seems obvious to me that if all you have is a function on a vector (representing a token stream), then the only transformations you can create are ones that can be expressed in that vector space. In other words, if you express a problem in a more complicated space (e.g. a visual problem, or an abstract algebra problem), you will not be able to solve it in the smaller token space, there's not enough information. You COULD first transform the vector into a different representation, perform the transformation there, and then encode back. This step would have to be different for every reasoning problem, so the underlying transformation to solution space would have to be generic and move into one of N reasoning subsystems. Also, if we hypothetically assume this limitation doesn't exist, we would still have the issue that a trained LLM represents a single such transformation function. If we want generic reasoning skills, the approximated function would have to be different for every reasoning problem. Even if we use a subset of the vectors degrees of freedoms to express each type of reasoning, it would make any truly reasoning LLM computationally intractable, because the embeddings space would blow up.
  - int_19h 3y ago
    What couldn't you possibly represent as a bunch of numbers to the level of precision necessary to process it? We might not have sufficient compute power for LMs large enough for some tasks yet, but I don't see how it's different in principle. And yes, of course such representation would be different for different problems. We also have organs in our body that do that for various types of sensory inputs that ultimately all feed into the brain to be processed as quantized impulses. And GPT-4 is already supposed to be an "assembly of experts", i.e. different models trained for different tasks, with an orchestrating model weaving them together. This is also not a surprise, given that our brains aren't monolithic, either. You should be able to do it with a single model that's large enough, but it is likely more energy- and time-efficient to get there by putting smaller pieces together instead.

by tovej 3y ago

stevenhuang 3y ago

> There is no way to form an internal representation of such systems within an LLM, which is why I said "of course" it can't reason. An LLM is not designed with this capability in mind. You claim this as if it is a foregone conclusion yet it is anything but. This is an active area of study, even within human cognition https://en.m.wikipedia.org/wiki/Predictive_coding https://en.m.wikipedia.org/wiki/Predictive_coding A complex system clearly does not need to be designed to evoke some behavior. Just look to the concept of emergent phenomenon. See evolution through natural selection as an obvious counter example. Evolution selects for fitness and not reasoning ability. Yet here we are. It is not known if the same isn't happening now for the transformer model. LLMs are universal function approximators. If reasoning ability is at all representable through such a process, it is possible for LLMs to find it. https://en.m.wikipedia.org/wiki/Universal_approximation_theorem https://en.m.wikipedia.org/wiki/Universal_approximation_theo...

tovej 3y ago
Neural networks are universal function approximators, transformers are universal sequence to sequence approximators. Crucially, the sequences here are the stream tokens in an LLM. This means an LLM can do arbitrary language manipulation. If the reasoning task requires something that can't be done using language manipulation in embeddings-space, it won't be able to do it. Idk, maybe my mathematical intuition is wrong, but it seems obvious to me that if all you have is a function on a vector (representing a token stream), then the only transformations you can create are ones that can be expressed in that vector space. In other words, if you express a problem in a more complicated space (e.g. a visual problem, or an abstract algebra problem), you will not be able to solve it in the smaller token space, there's not enough information. You COULD first transform the vector into a different representation, perform the transformation there, and then encode back. This step would have to be different for every reasoning problem, so the underlying transformation to solution space would have to be generic and move into one of N reasoning subsystems. Also, if we hypothetically assume this limitation doesn't exist, we would still have the issue that a trained LLM represents a single such transformation function. If we want generic reasoning skills, the approximated function would have to be different for every reasoning problem. Even if we use a subset of the vectors degrees of freedoms to express each type of reasoning, it would make any truly reasoning LLM computationally intractable, because the embeddings space would blow up.
- int_19h 3y ago
  What couldn't you possibly represent as a bunch of numbers to the level of precision necessary to process it? We might not have sufficient compute power for LMs large enough for some tasks yet, but I don't see how it's different in principle. And yes, of course such representation would be different for different problems. We also have organs in our body that do that for various types of sensory inputs that ultimately all feed into the brain to be processed as quantized impulses. And GPT-4 is already supposed to be an "assembly of experts", i.e. different models trained for different tasks, with an orchestrating model weaving them together. This is also not a surprise, given that our brains aren't monolithic, either. You should be able to do it with a single model that's large enough, but it is likely more energy- and time-efficient to get there by putting smaller pieces together instead.