3 ms·

I actually think that quite the opposite is true. Yes, children can get some idea of a language, but they need a lot more experience and exposure to reach some

by genman 3y ago

I actually think that quite the opposite is true. Yes, children can get some idea of a language, but they need a lot more experience and exposure to reach some level of proficiency in the language and this mostly includes supervised training in the school. They must read multiple books, have conversations etc to be able to really express themselves. It takes 10 years The difference is that LLM don't try to have such iterative intelligence like humans do but are more like knowledge boxes. Perhaps it would take a lot less data to gain the same level of language understanding the 3 year olds have.

jampekka 3y ago
Children are probably better in language-as-communication than current LLMs at relatively young age. Verbal (and bodily) communication is quite different from written text and its largely artificially imposed structure. Spoken language is very ungrammatical and directly litterated conversational speech is a total mess on writing standards (and a lot of information is lost in litteration). Even though e.g. Chomskian linguistics make the claim that human language is somehow a special innate syntactic construct (recursive language in the Chomsky hierarchy), I find the empirically stronger view is that the more formal grammars and vocabularies that sometimes are thought of as "language" are very far from how language is actually used when this structure is not imposed. This quite rigid structure of "correct" written language is also likely a major factor to why LLMs are so good at it.
api 3y ago
If a child learned to read at 2 and spent every waking moment reading until 10 it would still be orders of magnitude less than what we use to train even smaller base models (e.g. Mistral-7b). My 10 year old daughter doesn’t read as much as she should (too much gaming) but gets 100% on science and history tests and definitely would outperform GPT-4 on reasoning. She’d have no problem telling me how many sisters Sally has. :) A human brain consumes 20-40 watts too, not hundreds. I bet total power consumption for a human brain age 0-10 is substantially less than what it takes to train a medium sized base model. That’d be an interesting bit of math to add. GPT-4 will win on breadth of knowledge but that’s just because it’s a lossy compression blob containing a dump of the Internet. We are talking about intelligence (GI) here not just being a big dumb know it all.
- fragmede 3y ago
  That theoretical child isn't just reading words though. it's experiencing life in a human body, with a human brain, and getting full force of all their senses and biology of being human. We don't have a good measure for how much data that is, even if the majority of that is being thrown away. This theoretical child reading from 2-10 is reading fewer words, sure, but are they being exposed to far less data? The fonts being used, the smell of the books, how the spine cracks. LLMs don't get that in their datasets, just tokens, which isn't even the letters that make up a word.

by genman 3y ago

jampekka 3y ago

Children are probably better in language-as-communication than current LLMs at relatively young age. Verbal (and bodily) communication is quite different from written text and its largely artificially imposed structure. Spoken language is very ungrammatical and directly litterated conversational speech is a total mess on writing standards (and a lot of information is lost in litteration). Even though e.g. Chomskian linguistics make the claim that human language is somehow a special innate syntactic construct (recursive language in the Chomsky hierarchy), I find the empirically stronger view is that the more formal grammars and vocabularies that sometimes are thought of as "language" are very far from how language is actually used when this structure is not imposed. This quite rigid structure of "correct" written language is also likely a major factor to why LLMs are so good at it.

api 3y ago

If a child learned to read at 2 and spent every waking moment reading until 10 it would still be orders of magnitude less than what we use to train even smaller base models (e.g. Mistral-7b). My 10 year old daughter doesn’t read as much as she should (too much gaming) but gets 100% on science and history tests and definitely would outperform GPT-4 on reasoning. She’d have no problem telling me how many sisters Sally has. :) A human brain consumes 20-40 watts too, not hundreds. I bet total power consumption for a human brain age 0-10 is substantially less than what it takes to train a medium sized base model. That’d be an interesting bit of math to add. GPT-4 will win on breadth of knowledge but that’s just because it’s a lossy compression blob containing a dump of the Internet. We are talking about intelligence (GI) here not just being a big dumb know it all.

fragmede 3y ago
That theoretical child isn't just reading words though. it's experiencing life in a human body, with a human brain, and getting full force of all their senses and biology of being human. We don't have a good measure for how much data that is, even if the majority of that is being thrown away. This theoretical child reading from 2-10 is reading fewer words, sure, but are they being exposed to far less data? The fonts being used, the smell of the books, how the spine cracks. LLMs don't get that in their datasets, just tokens, which isn't even the letters that make up a word.