3 ms·
I suppose I can grant "feels a bit like", but unless I'm missing something (and to be fair I have not rtfa...) it hasn't actually reached refutation. The observ
by dllthomas 3y ago
I suppose I can grant "feels a bit like", but unless I'm missing something (and to be fair I have not rtfa...) it hasn't actually reached refutation. The observation that there is enough information to determine grammar rules in the training set for an LLM is only very weak evidence that there is also enough information in the much smaller training set for a typical human.
- mcv 3y agoI think it's pretty obvious that there is more than enough information in the training set of a typical human to learn grammar. Because we do. And people in different countries learn different grammars. Also keep in mind that people get more feedback over their control of language ad grammar while they're learning. We don't just listen, we communicate.
- dllthomas 3y agoThat's the point, though. As you say there is obviously enough information to learn grammar given our starting point, but Chomsky's point is that there's not enough information to distinguish between all possible languages, so (paraphrasing from a decades-old recollection and probably very much putting my spin on it) learning a (first?) language as a human is probably a matter of figuring out how to apply existing "machinery" that imposes some of its own structure rather than operating entirely from the information provided to the senses. I'm not saying it's necessarily correct, but it does seem well formed, and LLMs only undermine it in a very weak sense (so far?). To defend the conclusion itself I would have to know a whole lot more about how he estimated information content (needed and available).