The fact that the model learns to generate samples from a distribution, be it a natural language like English, or a computer language like Python, does not mean

The fact that the model learns to generate samples from a distribution, be it a natural language like English, or a computer language like Python, does not mean it is overfitting. The fact that it can't generate good APL, MUMPS, or whatever means that it does not have enough data to learn its distribution. Knowing so much about some languages like Python is not a hindrance, but likely a boon, because it teaches the model what programming languages look like. If it was smarter, it could figure things out just from the language specification, but we're not there yet.