10 ms·

I see this stance of yours parroted over and over but a 3 year old can tell a dog from a cat doesn't need to be trained on millions of images. Also uses way les

I see this stance of yours parroted over and over but a 3 year old can tell a dog from a cat doesn't need to be trained on millions of images. Also uses way less energy for that.

landryraccoon 3y ago
A 3 year old still takes 3 years to train. Even a state of the art image recognition model takes way less time than that.
- beepbooptheory 3y ago
  Not sure if you meant this as a joke, but it made me smile. That poor child!
- ChatGTP 3y ago
  The moronic thing about this is, humans aren't just training, they're having a life. The "training part" is one part of it sure, but it's not the reason for existence.
  - sebzim4500 3y ago
    What's the functional difference between training and having a life?
naasking 3y ago
> but a 3 year old can tell a dog from a cat doesn't need to be trained on millions of images A 3 year old has 3 years of multimodal training data and RLHF + a few billions of years of evolution that have primed and biased our visual and cognitive systems. That requires a lot more data than machine models that literally zero inherent bias. Assuming you want a true apples to apples comparison.
AbrahamParangi 3y ago
The structure of that 3yr old’s visual cortex is itself the result of a ~500 million year optimization process.
marcinzm 3y ago
It takes basically a week on a single GPU to train AlexNet which has human level ImageNet performance. Let's say it's 500 W for the GPU versus around 10 W for a human brain. So that's 84kwh for the model and 175kwh for the baby (over 3 years at 16h/day). That's without a half billion years of architecture and initialization tuning that the baby has. I think the model performs very favorably.
- actionfromafar 3y ago
  But don’t 3 yr olds have more skills than distinguishing dogs from cats?
  - marcinzm 3y ago
    Sure but my point is that the energy costs are in the same magnitude. If you want to be pedantic then only 6% of the human brain is the visual cortex. But AlexNet is also an inefficient model so something like an optimized ResNet is 100x as efficient to train. So now you're at 10.5kwh and 1.5kwh for the baby and model respectively. You can argue details further but I'd say the energy cost of both is fairly close.
    - ShamelessC 3y ago
      You’re missing my original point which is about continued, ongoing robustness that works in the low data regime and allows pilots/astronauts to make reasonable decisions in _completely novel_ situations (as just one example). The networks we have are trained once and work efficiently for their training dataset. They are even robust to outliers in the distribution of that dataset. But they aren’t robust to surprises, unmentioned changes in assumptions/rules/patterns about the data. Even reinforcement learning is still struggling with this as self-play effectively requires being able to run your dataset/simulation quickly enough to find new policies that work. Humans don’t even have time for that, much less access to a full running simulation of their environments. Although we do generate world models and there’s some work towards that I believe. Again happy to be corrected.
  - sdenton4 3y ago
janalsncm 3y ago
This is under the assumption that brains start at random (the Tabula Rasa theory of the brain) but that doesn’t seem plausible to me. Brains have the benefit of some amount of pre training at the time of birth. That’s why spiders don’t need to be taught how to spin complex webs and why humans don’t need to learn how to manipulate abstract mental symbols (i.e. language).
sdenton4 3y ago
I can train a bird song recognition model in about two days in a v100 which performs decently well on upwards of three thousand species, and generalizes reasonably to real world data (beyond a somewhat skewed training data distribution). Humans are very bad at this task; it takes a massive effort to learn this many birds. In fact it's a great counterexample to human few shot learning ability...

I see this stance of yours parroted over and over but a 3 year old can tell a dog from a cat doesn't need to be trained on millions of images. Also uses way less energy for that.

landryraccoon 3y ago

A 3 year old still takes 3 years to train. Even a state of the art image recognition model takes way less time than that.

beepbooptheory 3y ago
Not sure if you meant this as a joke, but it made me smile. That poor child!
ChatGTP 3y ago
The moronic thing about this is, humans aren't just training, they're having a life. The "training part" is one part of it sure, but it's not the reason for existence.
- sebzim4500 3y ago
  What's the functional difference between training and having a life?

naasking 3y ago

> but a 3 year old can tell a dog from a cat doesn't need to be trained on millions of images A 3 year old has 3 years of multimodal training data and RLHF + a few billions of years of evolution that have primed and biased our visual and cognitive systems. That requires a lot more data than machine models that literally zero inherent bias. Assuming you want a true apples to apples comparison.

AbrahamParangi 3y ago

The structure of that 3yr old’s visual cortex is itself the result of a ~500 million year optimization process.

marcinzm 3y ago

It takes basically a week on a single GPU to train AlexNet which has human level ImageNet performance. Let's say it's 500 W for the GPU versus around 10 W for a human brain. So that's 84kwh for the model and 175kwh for the baby (over 3 years at 16h/day). That's without a half billion years of architecture and initialization tuning that the baby has. I think the model performs very favorably.

actionfromafar 3y ago
But don’t 3 yr olds have more skills than distinguishing dogs from cats?
- marcinzm 3y ago
  Sure but my point is that the energy costs are in the same magnitude. If you want to be pedantic then only 6% of the human brain is the visual cortex. But AlexNet is also an inefficient model so something like an optimized ResNet is 100x as efficient to train. So now you're at 10.5kwh and 1.5kwh for the baby and model respectively. You can argue details further but I'd say the energy cost of both is fairly close.
  - ShamelessC 3y ago
    You’re missing my original point which is about continued, ongoing robustness that works in the low data regime and allows pilots/astronauts to make reasonable decisions in _completely novel_ situations (as just one example). The networks we have are trained once and work efficiently for their training dataset. They are even robust to outliers in the distribution of that dataset. But they aren’t robust to surprises, unmentioned changes in assumptions/rules/patterns about the data. Even reinforcement learning is still struggling with this as self-play effectively requires being able to run your dataset/simulation quickly enough to find new policies that work. Humans don’t even have time for that, much less access to a full running simulation of their environments. Although we do generate world models and there’s some work towards that I believe. Again happy to be corrected.
- sdenton4 3y ago

janalsncm 3y ago

This is under the assumption that brains start at random (the Tabula Rasa theory of the brain) but that doesn’t seem plausible to me. Brains have the benefit of some amount of pre training at the time of birth. That’s why spiders don’t need to be taught how to spin complex webs and why humans don’t need to learn how to manipulate abstract mental symbols (i.e. language).

sdenton4 3y ago

I can train a bird song recognition model in about two days in a v100 which performs decently well on upwards of three thousand species, and generalizes reasonably to real world data (beyond a somewhat skewed training data distribution). Humans are very bad at this task; it takes a massive effort to learn this many birds. In fact it's a great counterexample to human few shot learning ability...