My understanding is that at some point the model needs to gain an understanding of how the world works if it needs to know the right set of words that come next

My understanding is that at some point the model needs to gain an understanding of how the world works if it needs to know the right set of words that come next to keep reducing perplexity. But I don't think this can scale to AGI but I have nothing to back it up