3 ms·

Kids learn grammar by copying, but a group of kids with no language will invent one (see Nicaraguan sign language), and empty transformer models will not do thi

Kids learn grammar by copying, but a group of kids with no language will invent one (see Nicaraguan sign language), and empty transformer models will not do this. Kids raised hearing a mix of French and African languages will also invent a new language (Haitian creole) and a LLM will not do this, it would alternate speaking one or the other. ChatGPT proves that language is not so complex that it can’t be modeled, but the core Chomsky arguments are unaffected. I haven’t looked into the alternative perspectives (emergent grammar, etc) in the discipline since undergrad so I’m not saying Chomsky is right, just saying LLMs don’t disprove it.

mcv 3y ago
> empty transformer models will not do this. Do we have evidence for that? Of course those kids did have a real world to communicate about, and that creates the need for a language. I think if you had AI models that had a world to interact with and other models to communicate with and learn from, they might be able to develop their own language too. But this does get to another of Chomsky's points: the need to have a world model. To current LLMs, the only world model they have is text they read on the internet. And they only communicate with us, and I don't think they learn from their communication with us. So that's unavoidably limiting them. But I think it would be a very interesting experiment to give them a world to interact with and other similar models to communicate with, to see if they develop some sort of language (which would be completely incomprehensible to us of course; that's another hurdle). I agree that LLMs don't disprove his inborn grammar argument, but they don't support it either. I think more research is needed.