this is sort of the bigger picture observation I have about LLMs in general: that they seem to be massively over-parameterised, which in a classical statistical

this is sort of the bigger picture observation I have about LLMs in general: that they seem to be massively over-parameterised, which in a classical statistical world would suggest they will readily over fit and it will be hard to get them to generalise beyond regurgitating their training material. Yet this is not the behaviour we see. So in some sense the "magic" we have seen flourish is that LLMs have managed to devise training mechanisms that do let them learn without over fitting in the context of a large excess of parameters. Part of it may be the sheer volume of training material but I suspect even with that you would get straight up regurgitation because it is so hard to stop the input data being degenerate. So other factors about how the training works must be contributing to this. Of course, this is all my semi-layman speculation about this and I'm curious what actually knowledgeable people think.