Well, you are right that it may learn statistical associations that we associate/generalize ourselves with/as a model when we probe it. I think the key bit her

Well, you are right that it may learn statistical associations that we associate/generalize ourselves with/as a model when we probe it. I think the key bit here is that we can influence its associations by inputting the prompt, changing the model that is presented. I'm waay ahead of myself here, but the thought is interesting, and it will likely remain an open question for at least a few months/years.