this is talking about "in-context" learning, i.e. you tell the bot some new information (here is a language, here are its rules, now translate this text) If it

this is talking about "in-context" learning, i.e. you tell the bot some new information (here is a language, here are its rules, now translate this text) If it hasn't seen a language with those rules before, it makes a lot of mistakes. The key is with those sorts of rules. If it's seen one with similar rules but ordered differently, it will do much better than with entirely novel rules. I imagine a human would too. The paper itself trained a transformer to predict what function some data points represented, either linear (x=y or similar) or sinusoid (y=sin(x) or similar). They then presented a section of the sinoid data to the transformer, so the correct output would have been something similar to the sin function. they repeated this with models trained on various mixtures of linear and sinoid distributions, making the sinoids increasingly rarer and eventually not appearing in the distribution at all. Predictably, the accuracy got worse as the model saw less of the sinoids. You may note that these are vastly less sophisticated models than GPT.