I can't prove it, but I suspect there is a more fundamental limitation to any language model that is purely a language model in the sense of a probability distr

I can't prove it, but I suspect there is a more fundamental limitation to any language model that is purely a language model in the sense of a probability distribution over possible words given the precedent of other words. Gaining any meaningful level of understanding without an awareness that things other than words even exist seems like it won't happen. The most obvious limitation is you can't develop a language that way. Language is a compression of reality or of some other intermediate model of reality to either an audio stream or symbol stream, so not having access to the less abstracted models, let alone to reality itself, means you can never understand anything except the existing corpus. That isn't a criticism of GPT-3 by any stretch, as comments like this seem to often get interpreted that way, but the "taking all possible jobs AGI" hype seems a bit out of control given it is just a language model. Even something with the unambiguous intellect of a human, say an actual human, but with no ability to move, no senses other than hearing, that never heard anything except speech, would not be expected by anyone to dominate all job markets and advance the intellectual frontier. This, of course, goes beyond fundamental limitations of GPT-3, as I see this as a fundamental limitation of any language model whatsoever. On its own, it isn't enough. At some point, AI research is going to have to figure out how to fuse models from many domains and get them to cooperatively model all of the various ways to explore and sense reality. That includes the corpus of existing human written knowledge, but it isn't just that.