4 ms·

The part about data is the most fascinating, and I think it’s rather obvious that whatever the brain is doing with learning must be quite a bit more data effici

by api 3y ago

The part about data is the most fascinating, and I think it’s rather obvious that whatever the brain is doing with learning must be quite a bit more data efficient than what we are doing in AI systems. Anyone who has ever had kids knows this. Little kids learn language and fully developed world models on a tiny amount of data compared to the massive hoards we shovel into our training systems. That data is also noisy and only loosely curated. The author does mention this but I think he actually doesn’t give it enough importance. A person could read for their entire life and not approach what we shovel into GPT-3.5 or llama2, yet a three year old builds a trained language model from what must be near scratch. It must be near scratch for the initial condition because the human genome is nowhere near large enough to contain that much “priming.” There can’t be a “base model” analogue or anything like that in there.

pier25 3y ago
> and I think it’s rather obvious that whatever the brain is doing with learning must be quite a bit more data efficient than what we are doing in AI systems Yeah and the brain only needs 20 watts of power.
MVissers 3y ago
What’s also fascinating is that a child’s brain has additional plasticity/aptitude for learning different things at a different age. If he/she learns it too late, they’ll never master it. Perfect pitch is an example, ‘perfect’ motor skills for something another. Language another. Finetuning happens as a teenager/adult.
mr_toad 3y ago
> Little kids learn language and fully developed world models on a tiny amount of data compared to the massive hoards we shovel into our training systems. They don’t learn language in isolation. I doubt a human brain in a jar that was feed only literature would learn very well, if at all.
fragmede 3y ago
Kids get full motion video, sound, smell, taste, and touch, not to mention to other less known senses. How that compares to all digitized human knowledge literature, I don't know, but given that we don't have a wide standard for eg smell, I'd hesitate to call their datasets tiny.
- levitate 3y ago
  To be fair, I don't know how much of that huge input is stored for very long. Human brains are incredible at discarding unimportant information by generalizing information into broad ideas and emotions. Just think back to something that happened a few years ago and try to recall specific details. Being able to identify and discard unneeded data in realtime seems like a huge step for AI that hasn't been really implemented yet.
  - Teever 3y ago
    Maybe that discarding process and the time it takes to learn it from a combination of so many different senses is the key to developing a human mind to the level of intelligence that it is/has.
  - RoyalHenOil 3y ago
    Humans are EXTREMELY more energy efficient than computers; an adult human operates at something like 100 Watts on average (much less for children, of course), and that covers not only all of our mental processing, but also all of our physical movement, digestion, tissue regeneration/repair, etc. At no point in our evolutionary history have we had the leisure to absorb the vast quantities of energy that AI supercomputers can. I think this probably has a lot to do with why we are so effective at learning on little data; even when large quantities of data are available, it seems we may not have the energy to take it on board. We ignore and forget most of it. Maybe if we made a real effort to develop energy efficient AI, that design limitation would help us develop AI that requires far less training data.
genman 3y ago
I actually think that quite the opposite is true. Yes, children can get some idea of a language, but they need a lot more experience and exposure to reach some level of proficiency in the language and this mostly includes supervised training in the school. They must read multiple books, have conversations etc to be able to really express themselves. It takes 10 years The difference is that LLM don't try to have such iterative intelligence like humans do but are more like knowledge boxes. Perhaps it would take a lot less data to gain the same level of language understanding the 3 year olds have.
- jampekka 3y ago
  Children are probably better in language-as-communication than current LLMs at relatively young age. Verbal (and bodily) communication is quite different from written text and its largely artificially imposed structure. Spoken language is very ungrammatical and directly litterated conversational speech is a total mess on writing standards (and a lot of information is lost in litteration). Even though e.g. Chomskian linguistics make the claim that human language is somehow a special innate syntactic construct (recursive language in the Chomsky hierarchy), I find the empirically stronger view is that the more formal grammars and vocabularies that sometimes are thought of as "language" are very far from how language is actually used when this structure is not imposed. This quite rigid structure of "correct" written language is also likely a major factor to why LLMs are so good at it.
- api 3y ago
  If a child learned to read at 2 and spent every waking moment reading until 10 it would still be orders of magnitude less than what we use to train even smaller base models (e.g. Mistral-7b). My 10 year old daughter doesn’t read as much as she should (too much gaming) but gets 100% on science and history tests and definitely would outperform GPT-4 on reasoning. She’d have no problem telling me how many sisters Sally has. :) A human brain consumes 20-40 watts too, not hundreds. I bet total power consumption for a human brain age 0-10 is substantially less than what it takes to train a medium sized base model. That’d be an interesting bit of math to add. GPT-4 will win on breadth of knowledge but that’s just because it’s a lossy compression blob containing a dump of the Internet. We are talking about intelligence (GI) here not just being a big dumb know it all.

by api 3y ago

pier25 3y ago

> and I think it’s rather obvious that whatever the brain is doing with learning must be quite a bit more data efficient than what we are doing in AI systems Yeah and the brain only needs 20 watts of power.

MVissers 3y ago

What’s also fascinating is that a child’s brain has additional plasticity/aptitude for learning different things at a different age. If he/she learns it too late, they’ll never master it. Perfect pitch is an example, ‘perfect’ motor skills for something another. Language another. Finetuning happens as a teenager/adult.

mr_toad 3y ago

> Little kids learn language and fully developed world models on a tiny amount of data compared to the massive hoards we shovel into our training systems. They don’t learn language in isolation. I doubt a human brain in a jar that was feed only literature would learn very well, if at all.

fragmede 3y ago

Kids get full motion video, sound, smell, taste, and touch, not to mention to other less known senses. How that compares to all digitized human knowledge literature, I don't know, but given that we don't have a wide standard for eg smell, I'd hesitate to call their datasets tiny.

levitate 3y ago
To be fair, I don't know how much of that huge input is stored for very long. Human brains are incredible at discarding unimportant information by generalizing information into broad ideas and emotions. Just think back to something that happened a few years ago and try to recall specific details. Being able to identify and discard unneeded data in realtime seems like a huge step for AI that hasn't been really implemented yet.
- Teever 3y ago
  Maybe that discarding process and the time it takes to learn it from a combination of so many different senses is the key to developing a human mind to the level of intelligence that it is/has.
- RoyalHenOil 3y ago
  Humans are EXTREMELY more energy efficient than computers; an adult human operates at something like 100 Watts on average (much less for children, of course), and that covers not only all of our mental processing, but also all of our physical movement, digestion, tissue regeneration/repair, etc. At no point in our evolutionary history have we had the leisure to absorb the vast quantities of energy that AI supercomputers can. I think this probably has a lot to do with why we are so effective at learning on little data; even when large quantities of data are available, it seems we may not have the energy to take it on board. We ignore and forget most of it. Maybe if we made a real effort to develop energy efficient AI, that design limitation would help us develop AI that requires far less training data.

genman 3y ago

I actually think that quite the opposite is true. Yes, children can get some idea of a language, but they need a lot more experience and exposure to reach some level of proficiency in the language and this mostly includes supervised training in the school. They must read multiple books, have conversations etc to be able to really express themselves. It takes 10 years The difference is that LLM don't try to have such iterative intelligence like humans do but are more like knowledge boxes. Perhaps it would take a lot less data to gain the same level of language understanding the 3 year olds have.

jampekka 3y ago
Children are probably better in language-as-communication than current LLMs at relatively young age. Verbal (and bodily) communication is quite different from written text and its largely artificially imposed structure. Spoken language is very ungrammatical and directly litterated conversational speech is a total mess on writing standards (and a lot of information is lost in litteration). Even though e.g. Chomskian linguistics make the claim that human language is somehow a special innate syntactic construct (recursive language in the Chomsky hierarchy), I find the empirically stronger view is that the more formal grammars and vocabularies that sometimes are thought of as "language" are very far from how language is actually used when this structure is not imposed. This quite rigid structure of "correct" written language is also likely a major factor to why LLMs are so good at it.
api 3y ago
If a child learned to read at 2 and spent every waking moment reading until 10 it would still be orders of magnitude less than what we use to train even smaller base models (e.g. Mistral-7b). My 10 year old daughter doesn’t read as much as she should (too much gaming) but gets 100% on science and history tests and definitely would outperform GPT-4 on reasoning. She’d have no problem telling me how many sisters Sally has. :) A human brain consumes 20-40 watts too, not hundreds. I bet total power consumption for a human brain age 0-10 is substantially less than what it takes to train a medium sized base model. That’d be an interesting bit of math to add. GPT-4 will win on breadth of knowledge but that’s just because it’s a lossy compression blob containing a dump of the Internet. We are talking about intelligence (GI) here not just being a big dumb know it all.