Yea, this is an example of the transfomer models' ability of in-context learning, they just don't incorporate them into their long-term fixed memory yet, but I

Yea, this is an example of the transfomer models' ability of in-context learning, they just don't incorporate them into their long-term fixed memory yet, but I imagine it's the next step. Some interesting recent papers about the in-context learning: https://www.lesswrong.com/posts/firtXAWGdvzXYAh9B/paper-transformers-learn-in-context-by-gradient-descent https://www.lesswrong.com/posts/firtXAWGdvzXYAh9B/paper-tran... What Can Transformers Learn In-Context? A Case Study of Simple Function Classes: https://arxiv.org/abs/2208.01066 https://arxiv.org/abs/2208.01066 Transformers learning to learn (meta-learning): https://openreview.net/forum?id=t6tA-KB4dO https://openreview.net/forum?id=t6tA-KB4dO