I think you’re largely right, and that current GPT results may over-represent the model’s learning ability. A couple of speakers from Microsoft at the MPPC 202

I think you’re largely right, and that current GPT results may over-represent the model’s learning ability. A couple of speakers from Microsoft at the MPPC 2023 this week indicated that OpenAI’s models were not merely exposed to e.g. poetry, programming, etc. and learned those fields. Rather they were saying that the model is more of a composite of skills that were specifically trained, building on word identification, sentences, grammar, ultimately moving on to higher order skills. Perhaps this isn’t a secret (or perhaps I misunderstood), but it means the model’s ability to perform self-directed learning is much less than I previously though.