3 ms·
Chomsky said that LLMs are statistical regurgitators which means LLMs can never actually reason and explain which language understanding requires. That they are
by calf 3y ago
Chomsky said that LLMs are statistical regurgitators which means LLMs can never actually reason and explain which language understanding requires. That they are a wrong model of computation by definition.
It's an interesting position and I'm sympathetic toward it, he could be partly right in the end.
- Izkata 3y agoWell then he's already wrong: https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-othello-gpt-has-a-linear-emergent-world https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-o...
- ska 3y agoThat doesn't really follow from the linked research (which is interesting, though).
- Izkata 3y ago> > Chomsky said that LLMs are statistical regurgitators which means LLMs can never actually reason Othello-GPT managed to develop an internal model of the board that actually works, it doesn't just regurgitate. Hence, wrong.
- calf 3y agoIMO This an incorrect and unrigorous understanding of what "internal model" means which is why there is a valid scientific debate about this issue.
- mtlmtlmtlmtl 3y agoRegurgitators can't have internal representations? Sometimes the best way to regurgitate is to learn an internal representation. That doesn't mean it suddenly stopped being a statistical model.