>> You mention that a language model can't model language, only text, because it only observes examples of text, not language. However, I would argue that these

>> You mention that a language model can't model language, only text, because it only observes examples of text, not language. However, I would argue that these text examples are in fact representations of human language ability. The text that language models are trained on is a product of human language ability. This text, while not a perfect representation of language, is a direct outcome of language use and thus carries within it the patterns and structures that language models can learn. The technical term for what you describe is latent variable modelling, which I discuss in my comment above. Like I say in that comment, it is not possible to do that for human language ability because we don't have examples of it. Concretely, to train a latent variable model you need examples of both an observable variable X and the latent variable Y that is correlated with X. Without examples of both, you can't learn the correlation between them. Consider the Viterbi algorithm. One use of Viterbi is to train Part-of-Speech Taggers. This is possible because we can create a corpus of text where words are annotated with their parts of speech. We can do that because we have a fairly good knowledge of the parts of speech of different words (it's a human concept, after all). We can't do that with language ability. We can't take a corpus and annotate each sentence in the corpus with whatever sequence of operations in the human mind produced that text. Because we don't know what that sequence, is. Regarding Transformers' latent space- that refers to the correlation between words in a corpus, not the latent relation between linguistic ability and words. We can model the correlation between words, but we're still missing the hidden variable that causes this correlation, i.e. human linguistic ability. Regarding translation, language models can do that because there are examples of parallel corpora (i.e. texts translated in multiple languages) in their training data. The most obvious example is Wikipedia. No modelling of common structure underlying structure is needed.