Your analogy is reaching to the farthest edge case - one of complete non-understanding and complete mimicry. The problem is that language models do understand c

Your analogy is reaching to the farthest edge case - one of complete non-understanding and complete mimicry. The problem is that language models do understand concepts for some reasonable definitions of understanding: they will use the concept correct and with low error rate. So all you’re really pointing at here is an example where they still have poor understanding, not that they have some innate inability to understand. Alternatively, you need to provide a definition of understanding which is falsifiable and shown to be false for all concepts a language model could plausibly understand.