3 ms·
It depends on what does "generalize beyond the training data" means. If I invent a new programming language and I teach (in-context) the language to the model a
by GaggiX 3y ago
It depends on what does "generalize beyond the training data" means. If I invent a new programming language and I teach (in-context) the language to the model and it's able to use it to solve many tasks, is it generalizing beyond the training data?
- andy99 3y agoNo. The way I'd look at it is that generalization or specifically extrapolation would mean that different features are needed to make a prediction (here, the next token) than what is seen in the training data. Something like a made up language could still result in the same patterns being relevant. That's why out-of-distribution research often uses mathematical extrapolation as a task.
- famouswaffles 3y ago"Supercharged Interpolation" is not something that actually exists. Learning in High Dimension Always Amounts to Extrapolation https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485 What you're asking for is not "generalization" but magic and humans would also fail.
- deleted 3y ago[deleted]
- spywaregorilla 3y agoCan you provide a real world example? Because this sounds like nonsense. As in, not a weakness of any architecture but just the very concept of pattern matching. What you might be asking for is a system that simply continually learns.
- vidarh 3y agoCan humans do this? Because I'm not convinced humans can do this. Or that it reasonably means anything.