My takeaway from all of the GPT successes and failure (the linear algebra vs riemann hypothesis cases being one example) is that language and jargon can have bu

My takeaway from all of the GPT successes and failure (the linear algebra vs riemann hypothesis cases being one example) is that language and jargon can have built in structural constraints such that by having a good LLM you can get good results because the grammatical structure happens to match reality's structure. But reality is under no constraint to conform to the grammatical structures that we use in order to communicate about related problems or solutions. And I suppose this shouldn't be too surprising considering even logic itself runs into godelian issues. For any consistent logic there exists some statement which is true, but which does not have a proof. Which feels vaguely similar to "for any functional LLM there exists some truth which defies the next statistically likely sequence of tokens".