Quoting Timos Moraitis a Neuromorphic PhD "For reasons like this, "in-context learning" is not an accurate term for transformers. It's projection and storage,

Quoting Timos Moraitis a Neuromorphic PhD "For reasons like this, "in-context learning" is not an accurate term for transformers. It's projection and storage, nothing is learnt. This new paper has attracted a lot of interest, and it's nice that it proves things formally and empirically, but it looks like people are surprised by it, even though it was clear." https://x.com/timos_m/status/1983625714202010111 https://x.com/timos_m/status/1983625714202010111