6 ms·

> humans do not need 10,000 examples to tell the difference between cats and dogs, I swear, not enough people have kids. Now, is it 10k examples? No, but I th

> humans do not need 10,000 examples to tell the difference between cats and dogs, I swear, not enough people have kids. Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. One thing kids do is they'll ask for confirmation of their guess. You'll be reading a book you've read 50 times before and the kid will stop you, point at a dog in the book, and ask "dog?" And there is a development phase where this happens a lot. Also kids can get mad if they are told an object doesn't match up to the expected label, e.g. my son gets really mad if someone calls something by the wrong color. Another thing toddlers like to do is play silly labeling games, which is different than calling something the wrong name on accident, instead this is done on purpose for fun. e.g. you point to a fish and say "isn't that a lovely llama!" at which point the kid will fall down giggling at how silly you are being. The human brain develops really slowly[1], and a sense of linear time encoding doesn't really exist for quite awhile. (Even at 3, everything is either yesterday, today, or tomorrow) so who the hell knows how things are being processed, but what we do know is that kids gather information through a bunch of senses, that are operating at an absurd data collection rate 12-14 hours a day, with another 10-12 hours of downtime to process the information. [1] Watch a baby discover they have a right foot. Then a few days later figure out they also have a left foot. Watch kids who are learning to stand develop a sense of "up above me" after they bonk their heads a few time on a table bottom. Kids only learn "fast" in the sense that they have nothing else to do for years on end.

9cb14c1ec0 2y ago
> not enough people have kids. Second that. I think I've learned as much as my children have. > Watch a baby discover they have a right foot. Then a few days later figure out they also have a left foot. Watching a baby's awareness grow from pretty much nothing to a fully developed ability to understand the world around is one of the most fascinating parts of being a parent.
1024core 2y ago
> I swear, not enough people have kids. My friends toddler, who grew up with a cat in the house, would initially call all dogs "cat". :-D
- deleted 2y ago
  [deleted]
- mkl 2y ago
  My niece, 3yo, at the zoo, spent about 30 seconds trying to figure out whether a pig was a cat or a car.
Nition 2y ago
> the kid will stop you, point at a dog in the book, and ask "dog?" Of course for a human this can either mean "I have an idea about what a dog is, but I'm not sure whether this is one" or it can mean "Hey this is a... one of those, what's the word for it again?"
PheonixPharts 2y ago
> Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. I have kids so I'm presuming I'm allowed to have an opinion here. This is ignoring the fact that babies are not just learning labels, they're learning the whole of language, motion planning, sensory processing, etc. Once they have the basics down concept acquisition time shrinks rapidly and kids can easily learn their new favorite animal in as little as a single example. Compare this to LLMs which can one-shot certain tasks, but only if they have essentially already memorized enough information to know about that task. It gives the illusion that these models are learning like children do, when in reality they are not even entirely capable of learning novel concepts. Beyond just learning a new animal, humans are able to learn entirely new systems of reasoning in surprisingly few examples (though it does take quite a bit of time to process them). How many homework questions did your entire calc 1 class have? I'm guessing less than 100 and (hopefully) you successfully learned differential calculus.
- _carbyau_ 2y ago
  Two other points - I've also forgotten a bunch, but also know I could "relearn" it faster than the first time around. To continue your example, I know I've learned calculus and was lauded at the time. Now I could only give you the vagaries, nothing practical. However I know if I was pressed, I could learn it again in short order.
- p1esk 2y ago
  illusion that these models are learning like children do, when in reality they are not even entirely capable of learning novel concepts Now imagine how much would your kid learn if the only input he ever received was a sequence of words?
  - _flux 2y ago
    Are you saying it's not fair for LLMs, because of the way they are taught is different? The difference is that we don't know better methods for them, but we do know of better methods for people.
deleted 2y ago
[deleted]
resource0x 2y ago
I haven't seen 1000 cats in my entire life. I'm sure I learned how to tell a dog from a cat after being exposed to just a single instance of each.
- lostmsu 2y ago
  I'm sure you saw over 1B images of cats though, assuming 24 images per second from vision.
  - lelanthran 2y ago
    > I'm sure you saw over 1B images of cats though, assuming 24 images per second from vision. The AI models aren't seeing the same image 1B times.
    - lostmsu 2y ago
      Neither are you, during those 10 000 hours most of the time you aren't absolutely still.
      - lelanthran 2y ago
        > Neither are you, during those 10 000 hours most of the time you aren't absolutely still. So? I'm still seeing the same object. Large models aren't trained on 10k different images of a single cat.
Auracle 2y ago
That’s all true, yet my 2.5 year old sometimes one-shots specific information. I told my daughter that woodpeckers eat bugs out of trees after doing what you said and asking “what’s that noise?” for the fifth time in a few minutes when we heard some this spring. She brought it up again at least a week later, randomly. Developing brains are amazing. She also saw an eagle this spring out the car window and said “an eagle! …no, it’s a bird,” so I guess she’s still working on those image classifications ;)
bamboozled 2y ago
I think your comment over intellectualises the way children experience the world. My child experiences the world in a really pure way. They don’t care much about labels or colours or any other human inventions like that. He picks up his carrot, he doesn’t care about the name or the color . He just enjoys it through purely experiencing eating it. He can also find incredible flow state like joy from playing with river stones or looking at the moon. I personally feel bad I have to each them to label things and but things in boxes. I think your child is frustrated at times because it’s a punish of a game. The departure from “the oceanic feeling. Your comment would make sense to me if the end game of our brains and human experience is labelling things. It’s not. It’s useful but it’s not what living is about.
smusamashah 2y ago
My kid is about 3 and has been slow on language development. He can barely speak a few short sentences now. Learning names of things and concepts made a big difference for him and that's a fascinating watch and realization. This reminds of the story of Adam learning names, or how some languages can express a lot more in fewer words. And it makes sense that LLMs look intelligent to us. My kid loves repeating the names of things he learned recently. For past few weeks, after learning 'spider' and 'snake' and 'dangerous' he keeps finding spiders around, no snakes so makes up snakes from curly drawn lines and tells us they are dangerous. I think we learn fast because of stereo (3d) vision. I have no idea how these models learn and don't know if 3d vision will make multi model LLMs better and require exponentially less examples.
- Tepix 2y ago
  > I think we learn fast because of stereo (3d) vision. I think stereo vision is not that important if you can move around and get spatial clues that way also.
  - smusamashah 2y ago
    Every animal/insect I can think of has more than 1 eye. Some has lot more than 2 eyes. It has to be that important.
llm_trw 2y ago
Babies, unlike machine learning models, aren't placed in limbo when they aren't running back propagation. Babies need few examples for complex tasks because they get constant infinitely complex examples on tasks which are used for transfer learning. Current models take a nuclear reactors worth of power to run back prop on top of a small countries GDP worth of hardware. They are _not_ going to generalize to AGI because we can't afford to run them.
- larodi 2y ago
  > Current models take a nuclear reactors worth of power to run back prop on top of a small countries GDP worth of hardware. Nice one. Perhaps we are to conclude the whole transformer architecture is amazingly overblown in storage/computation costs. AGI or not, we need better approach to what transformers are doing.
ein0p 2y ago
Not to mention that babies receive petabytes of visual input to go with other stimuli. It’s up for debate how sample efficient humans actually are in the first few years of their lives.
- Tepix 2y ago
  Hardly. Visual acuity is quite low (limited to a tiny area of the FoV), your brain is filling in all the blanks for you.
  - ein0p 2y ago
    Even at that resolution (about 0.35 MP per eye for just the fovea before any processing) napkin math suggests 7.3T per day. Over 5 years you get about 13PB if my math is right, assuming 16 waking hours per day.
cess11 2y ago
I have a small kid. When they first saw some jackdaws, the first bird they noticed could fly, they thought it was terribly exciting and immediately learned the word for them, and generalised it to geese, crows, gulls and magpies (plus some less common species I don't know what they're called in english), pointing at them and screaming the equivalent of 'jackda! jackda!'.
PontifexMinimus 2y ago
> Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. If I was presented with 10 pictures of 2 species I'm unfamiliar with, about as different as cats and dogs, I expect I would be able to classify further images as either, reasonably accurately.

9cb14c1ec0 2y ago

> not enough people have kids. Second that. I think I've learned as much as my children have. > Watch a baby discover they have a right foot. Then a few days later figure out they also have a left foot. Watching a baby's awareness grow from pretty much nothing to a fully developed ability to understand the world around is one of the most fascinating parts of being a parent.

1024core 2y ago

> I swear, not enough people have kids. My friends toddler, who grew up with a cat in the house, would initially call all dogs "cat". :-D

deleted 2y ago
[deleted]
mkl 2y ago
My niece, 3yo, at the zoo, spent about 30 seconds trying to figure out whether a pig was a cat or a car.

Nition 2y ago

> the kid will stop you, point at a dog in the book, and ask "dog?" Of course for a human this can either mean "I have an idea about what a dog is, but I'm not sure whether this is one" or it can mean "Hey this is a... one of those, what's the word for it again?"

PheonixPharts 2y ago

> Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. I have kids so I'm presuming I'm allowed to have an opinion here. This is ignoring the fact that babies are not just learning labels, they're learning the whole of language, motion planning, sensory processing, etc. Once they have the basics down concept acquisition time shrinks rapidly and kids can easily learn their new favorite animal in as little as a single example. Compare this to LLMs which can one-shot certain tasks, but only if they have essentially already memorized enough information to know about that task. It gives the illusion that these models are learning like children do, when in reality they are not even entirely capable of learning novel concepts. Beyond just learning a new animal, humans are able to learn entirely new systems of reasoning in surprisingly few examples (though it does take quite a bit of time to process them). How many homework questions did your entire calc 1 class have? I'm guessing less than 100 and (hopefully) you successfully learned differential calculus.

_carbyau_ 2y ago
Two other points - I've also forgotten a bunch, but also know I could "relearn" it faster than the first time around. To continue your example, I know I've learned calculus and was lauded at the time. Now I could only give you the vagaries, nothing practical. However I know if I was pressed, I could learn it again in short order.
p1esk 2y ago
illusion that these models are learning like children do, when in reality they are not even entirely capable of learning novel concepts Now imagine how much would your kid learn if the only input he ever received was a sequence of words?
- _flux 2y ago
  Are you saying it's not fair for LLMs, because of the way they are taught is different? The difference is that we don't know better methods for them, but we do know of better methods for people.

deleted 2y ago

[deleted]

resource0x 2y ago

I haven't seen 1000 cats in my entire life. I'm sure I learned how to tell a dog from a cat after being exposed to just a single instance of each.

lostmsu 2y ago
I'm sure you saw over 1B images of cats though, assuming 24 images per second from vision.
- lelanthran 2y ago
  > I'm sure you saw over 1B images of cats though, assuming 24 images per second from vision. The AI models aren't seeing the same image 1B times.
  - lostmsu 2y ago
    Neither are you, during those 10 000 hours most of the time you aren't absolutely still.
    - lelanthran 2y ago
      > Neither are you, during those 10 000 hours most of the time you aren't absolutely still. So? I'm still seeing the same object. Large models aren't trained on 10k different images of a single cat.

Auracle 2y ago

That’s all true, yet my 2.5 year old sometimes one-shots specific information. I told my daughter that woodpeckers eat bugs out of trees after doing what you said and asking “what’s that noise?” for the fifth time in a few minutes when we heard some this spring. She brought it up again at least a week later, randomly. Developing brains are amazing. She also saw an eagle this spring out the car window and said “an eagle! …no, it’s a bird,” so I guess she’s still working on those image classifications ;)

bamboozled 2y ago

I think your comment over intellectualises the way children experience the world. My child experiences the world in a really pure way. They don’t care much about labels or colours or any other human inventions like that. He picks up his carrot, he doesn’t care about the name or the color . He just enjoys it through purely experiencing eating it. He can also find incredible flow state like joy from playing with river stones or looking at the moon. I personally feel bad I have to each them to label things and but things in boxes. I think your child is frustrated at times because it’s a punish of a game. The departure from “the oceanic feeling. Your comment would make sense to me if the end game of our brains and human experience is labelling things. It’s not. It’s useful but it’s not what living is about.

smusamashah 2y ago

My kid is about 3 and has been slow on language development. He can barely speak a few short sentences now. Learning names of things and concepts made a big difference for him and that's a fascinating watch and realization. This reminds of the story of Adam learning names, or how some languages can express a lot more in fewer words. And it makes sense that LLMs look intelligent to us. My kid loves repeating the names of things he learned recently. For past few weeks, after learning 'spider' and 'snake' and 'dangerous' he keeps finding spiders around, no snakes so makes up snakes from curly drawn lines and tells us they are dangerous. I think we learn fast because of stereo (3d) vision. I have no idea how these models learn and don't know if 3d vision will make multi model LLMs better and require exponentially less examples.

Tepix 2y ago
> I think we learn fast because of stereo (3d) vision. I think stereo vision is not that important if you can move around and get spatial clues that way also.
- smusamashah 2y ago
  Every animal/insect I can think of has more than 1 eye. Some has lot more than 2 eyes. It has to be that important.

llm_trw 2y ago

Babies, unlike machine learning models, aren't placed in limbo when they aren't running back propagation. Babies need few examples for complex tasks because they get constant infinitely complex examples on tasks which are used for transfer learning. Current models take a nuclear reactors worth of power to run back prop on top of a small countries GDP worth of hardware. They are _not_ going to generalize to AGI because we can't afford to run them.

larodi 2y ago
> Current models take a nuclear reactors worth of power to run back prop on top of a small countries GDP worth of hardware. Nice one. Perhaps we are to conclude the whole transformer architecture is amazingly overblown in storage/computation costs. AGI or not, we need better approach to what transformers are doing.

ein0p 2y ago

Not to mention that babies receive petabytes of visual input to go with other stimuli. It’s up for debate how sample efficient humans actually are in the first few years of their lives.

Tepix 2y ago
Hardly. Visual acuity is quite low (limited to a tiny area of the FoV), your brain is filling in all the blanks for you.
- ein0p 2y ago
  Even at that resolution (about 0.35 MP per eye for just the fovea before any processing) napkin math suggests 7.3T per day. Over 5 years you get about 13PB if my math is right, assuming 16 waking hours per day.

cess11 2y ago

I have a small kid. When they first saw some jackdaws, the first bird they noticed could fly, they thought it was terribly exciting and immediately learned the word for them, and generalised it to geese, crows, gulls and magpies (plus some less common species I don't know what they're called in english), pointing at them and screaming the equivalent of 'jackda! jackda!'.

PontifexMinimus 2y ago

> Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. If I was presented with 10 pictures of 2 species I'm unfamiliar with, about as different as cats and dogs, I expect I would be able to classify further images as either, reasonably accurately.