5 ms·
What couldn't you possibly represent as a bunch of numbers to the level of precision necessary to process it? We might not have sufficient compute power for LMs
by int_19h 3y ago
What couldn't you possibly represent as a bunch of numbers to the level of precision necessary to process it? We might not have sufficient compute power for LMs large enough for some tasks yet, but I don't see how it's different in principle.
And yes, of course such representation would be different for different problems. We also have organs in our body that do that for various types of sensory inputs that ultimately all feed into the brain to be processed as quantized impulses.
And GPT-4 is already supposed to be an "assembly of experts", i.e. different models trained for different tasks, with an orchestrating model weaving them together. This is also not a surprise, given that our brains aren't monolithic, either. You should be able to do it with a single model that's large enough, but it is likely more energy- and time-efficient to get there by putting smaller pieces together instead.
- tovej 3y agoIt's just that there is no way for the model to map the token meanings to a iseful representation. Yes, in the general case, you can make a model for most types of logic problems (but be careful there, there are real problems you couldn't represent as "a bunch of numbers" and a single transformation), the issue is that the model cannot build that representation even for simple cases because it,'s used to operate at on language (and maybe some other fuzzy input type), which has to be _reliably_ interpreted to this number space; in different ways for different problems; without breaking the language processing.