I am dubious about the stated result. Language models have been very good at spitting out chunks of text verbatim that were in their training data. Buried in

I am dubious about the stated result. Language models have been very good at spitting out chunks of text verbatim that were in their training data. Buried in the github training data will belots of examples where people post their solutions to fun problems..including past code competitions. How often is it managing to match a description of a repository and spitting out correct code that cribbed heavily from it? That is, instead of figuring out the problem, it is pattern matching to a solution that someone else already figured out? How would we know?