5 ms·
>I've seen Chomsky argue that LLMs can't regurgitate his linguistical theories When has he said this? For the most part I feel Chomsky has been misunderstood w
by Kranar 3y ago
>I've seen Chomsky argue that LLMs can't regurgitate his linguistical theories
When has he said this? For the most part I feel Chomsky has been misunderstood when it comes to LLMs. As best as I can tell what Chomsky has said is that LLMs do not provide any insight into how language works, it's not really a scientific advancement so much as it's an engineering breakthrough.
The fact that LLMs exist and can mimic natural language does not in anyway give us insight into how humans construct language. People have been able to construct objects that can produce natural language for close to 100,000 years, but that doesn't mean that those people understood the nature of that language.
- calf 3y agoChomsky said that LLMs are statistical regurgitators which means LLMs can never actually reason and explain which language understanding requires. That they are a wrong model of computation by definition. It's an interesting position and I'm sympathetic toward it, he could be partly right in the end.
- Izkata 3y agoWell then he's already wrong: https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-othello-gpt-has-a-linear-emergent-world https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-o...
- ska 3y agoThat doesn't really follow from the linked research (which is interesting, though).
- Izkata 3y ago> > Chomsky said that LLMs are statistical regurgitators which means LLMs can never actually reason Othello-GPT managed to develop an internal model of the board that actually works, it doesn't just regurgitate. Hence, wrong.
- calf 3y agoIMO This an incorrect and unrigorous understanding of what "internal model" means which is why there is a valid scientific debate about this issue.
- mtlmtlmtlmtl 3y agoRegurgitators can't have internal representations? Sometimes the best way to regurgitate is to learn an internal representation. That doesn't mean it suddenly stopped being a statistical model.
- worik 3y ago> it's not really a scientific advancement so much as it's an engineering breakthrough. Yes, I agree with that. Very little science in LLMs But what utterly fantastic engineering! Totally breathtakingly fabulous engineering! I heard Noam say LLMs are "...plagiarism on an industrial scale". I agree. How incredible that modern engineers can build a machine to do plagiarism. Amazing Just a "stochastic parrot". Possible. But what are you? What am I?