4 ms·
No. The way I'd look at it is that generalization or specifically extrapolation would mean that different features are needed to make a prediction (here, the ne
by andy99 3y ago
No. The way I'd look at it is that generalization or specifically extrapolation would mean that different features are needed to make a prediction (here, the next token) than what is seen in the training data. Something like a made up language could still result in the same patterns being relevant. That's why out-of-distribution research often uses mathematical extrapolation as a task.
- famouswaffles 3y ago"Supercharged Interpolation" is not something that actually exists. Learning in High Dimension Always Amounts to Extrapolation https://arxiv.org/abs/2110.09485 https://arxiv.org/abs/2110.09485 What you're asking for is not "generalization" but magic and humans would also fail.
- deleted 3y ago[deleted]
- spywaregorilla 3y agoCan you provide a real world example? Because this sounds like nonsense. As in, not a weakness of any architecture but just the very concept of pattern matching. What you might be asking for is a system that simply continually learns.
- vidarh 3y agoCan humans do this? Because I'm not convinced humans can do this. Or that it reasonably means anything.