I agree the best possible language model would possess general AI, but you can't go from no intelligence to general AI in one step. It seems likely to me that t

I agree the best possible language model would possess general AI, but you can't go from no intelligence to general AI in one step. It seems likely to me that the intermediate levels of intelligence such a system might develop early on might actually make it a worse language model. The evolutionary path from dumb language model to language model with human level AI might well lie through an unhappy valley in which the fledgling AI makes more mistakes and is easier to catch out than a pure language model. In which case, a system would never take that evolutionary path. This is a problem with systems optimising towards a single specific function. It constrains their optimisation paths.