I agree, the in-context, few shot learning is the most interesting thing a about these transformer models, they have basically learned to learn (meta-learning).

I agree, the in-context, few shot learning is the most interesting thing a about these transformer models, they have basically learned to learn (meta-learning). Some additional interesting papers about it: Transformers learn in-context by gradient descent: https://arxiv.org/abs/2212.07677 https://arxiv.org/abs/2212.07677 What Can Transformers Learn In-Context? A Case Study of Simple Function Classes: https://arxiv.org/abs/2208.01066 https://arxiv.org/abs/2208.01066 "We also show that we can train Transformers to in-context learn more complex function classes -- namely sparse linear functions, two-layer neural networks, and decision trees -- with performance that matches or exceeds task-specific learning algorithms."