2 ms·
It can learn very well within its context. And it can leave notes for itself to read into future contexts. That's learning, whether you agree or not, and whet
by CamperBob2 2mo ago
It can learn very well within its context. And it can leave notes for itself to read into future contexts.
That's learning, whether you agree or not, and whether you like it or not. Baby steps, yes, but walking nevertheless.
- HarHarVeryFunny 2mo agoThere seems to be very few cases where ICL is really doing something that can be considered as learning (albeit ephemeral) rather than just utilizing in-context data via induction heads. Yes, I'm aware of the linear regression example, and a few others, but these appear to just be specific capabilities that were learnt during pre-training, presumably pursuant to reducing errors on similar-but-different training samples, not any kind of generic run-time learning capability.