6 ms·

My fundamental and maybe dumb question is: when is artificial intelligence / ML going to get smarter than needing a billion images to train? Sure, the achievem

My fundamental and maybe dumb question is: when is artificial intelligence / ML going to get smarter than needing a billion images to train? Sure, the achievements of ML models lately are impressive, but it's so slow in learning. We are brute forcing the DNNs it feels to me, which is not something that smacks of great achievement. You and I have never seen even 100,000 photos in our lives. Well, maybe the video stream from our eyes is a little different. But it's not a billion fundamentally different images. Is there anything I can read about why it is so slow to learn? How will it ever get faster? What next jump will fix this, or what am I missing as a lay person?

simonw 4y ago
I recommend looking into "transfer learning". That's where you start with an existing large model, and train a new model on top of it by feeding in new images. What's fascinating about transfer learning is that you don't need to give it a lot of new images, at all. Just a few hundred extras can create a model that's frighteningly accurate for tasks like image labeling. This is pretty much how all AI models work today. Take a look at the Stable Diffusion model card: https://github.com/CompVis/stable-diffusion/blob/main/Stable_Diffusion_v1_Model_Card.md#training https://github.com/CompVis/stable-diffusion/blob/main/Stable... They ran multiple training sessions with progressively smaller (and higher quality) images to get the final result.
namose 4y ago
I think the thing that’s missing is that the AI can’t train itself. If you were asked to draw a realistic x ray of a horse’s ribcage, you’d probably google image search, do some research about horse anatomy, etc, before putting pen to paper. This thing is being trained exactly once, and can’t learn dynamically. That’ll be the next step I think.
- andreyk 4y ago
  What are you are describing is pretty much reinforcement learning (or learning with access to a query-able knowledge engine, or active learning, or all of these combined). There is work on a bunch of variations of this, but it's true that it's early days for combining it with generative systems.
  - namose 4y ago
    Yeah, I think the query-able knowledge engine is the key here, although I think it’s maybe more like “we haven’t figured out how to generalize conceptual learning”. The computer not only has to be able to query images on the internet, but also know how/what to query, which includes a bunch of actions computers are currently incapable of. In some cases, we might complete a drawing task by traveling to a new place, or taking an action (throw an egg at concrete) not querying the internet.
andreyk 4y ago
A couple of things: * We have seen more than 100,000 "photos" in the sense that photos are just images - if photos are just images, we have a constant feed of "photos" every single moment our eyes are open. Of course, that's not the same as these training datasets, but it is still worth keeping in mind. * All of these things trained on massive datasets with self-supervised learning are in a sense addressing the "slowness" of learning you mention, since self-supervised (aka no annotations are needed beyond the data itself) "pre-training" on the massive datasets can then enable training for downstream tasks with way less data. * Arguably requiring massive datasets for pre-training is still a bit lame, but then again the 4-5 years of life it takes to reach pretty advanced intelligence in humans represents a whoooole lot of data. And as with self-supervised learning on these massive models, a lot of intelligence seems to come down to learning to predict the future from sensory input. * Humans also come with a lot of pre-wiring done by evolution, whereas these models are trained from scratch. Evolutionary wiring represents its own sort of "pre-training", of course. So basically, it is not so slow to learn as it seems. Arguably it could get faster once we train multimodal models and concepts from text can reinforce learning to understand images and so on, and people are working on it (eg GATO). There may also need to be a separation between low level 'instinct' intelligence and high-level 'reasoning' intelligence; AI still sucks at the second one.
- supernova87a 4y ago
  I guess what I find really interesting is, how come we can start to self-label data we encounter in the wild, yet the DNN needs data to constantly be labeled at the same intensity per image, into the billions?
  - simonw 4y ago
    If you look at the text labels for the data used by Stable Diffusion you'll find that they are very low quality. Take a look at some here: https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im... Clearly quality of labeling isn't nearly as important once you are training on billions of images.
  - 4y ago
12ian34 4y ago
Most of us have our eyes open, looking at things ~16 hours a day, with the first 18 years of our lives heavily focused on learning about what those things are, plus we have the extra brain capacity to remember those things, and to think about them in an abstract manner. My entire photo library alone is over 100,000 photos - and since I took all of them I will have "seen" them.
Ukv 4y ago
> You and I have never seen even 100,000 photos in our lives If someone's 30, that'd only require seeing 10 images a day. For most people that quota is probably fulfilled within a couple of minutes of watching TV or browsing social media, even if the video stream from our eyes otherwise counts as nothing. We've also had about 4 billion years of evolution, slowly adjusting our genome with an unfathomable amount of data. Gradient descent is blazing fast by comparison.
- omegalulw 4y ago
  Add to that a video now a days is at least 24fps. So a two hour movie suffices haha :)
xipho 4y ago
> You and I have never seen even 100,000 photos in our lives. Well, maybe the video stream from our eyes is a little different. But it's not a billion fundamentally different images. I would argue precisely the opposite (as you alude to), it's more than 100s of billions of fundamentally (what does this even mean?) different images. Calculate the frequency at which your eyes sample, think of the times the angle changes (new images), multiple by your age, multiple by 2 for two eyes looking slightly different directions, factor in your the noise your brain has in forming the image "in your head" because you drank too much... you can continue adding factors (hours of "TV" watched on average) add-naseum. It seems that "slow to learn" has a "real" target/bound, what humans are capable of. If it takes Bob Ross decades to paint all the "images" in his head, then maybe we should go easy on the algorithims?
PeterisP 4y ago
Some experiments have demonstrated that being involved in the 'generation' of the visual data (i.e. choosing where to move, where to look, how to alter reality) gets significantly better learning than passively receiving the exact same visual data. Active learning is a good way to improve sample efficiency - however, as others note, don't underestimate the quantity of learning data that a human baby needs for certain skills even with good, evolution-optimized priors.

simonw 4y ago

I recommend looking into "transfer learning". That's where you start with an existing large model, and train a new model on top of it by feeding in new images. What's fascinating about transfer learning is that you don't need to give it a lot of new images, at all. Just a few hundred extras can create a model that's frighteningly accurate for tasks like image labeling. This is pretty much how all AI models work today. Take a look at the Stable Diffusion model card: https://github.com/CompVis/stable-diffusion/blob/main/Stable_Diffusion_v1_Model_Card.md#training https://github.com/CompVis/stable-diffusion/blob/main/Stable... They ran multiple training sessions with progressively smaller (and higher quality) images to get the final result.

namose 4y ago

I think the thing that’s missing is that the AI can’t train itself. If you were asked to draw a realistic x ray of a horse’s ribcage, you’d probably google image search, do some research about horse anatomy, etc, before putting pen to paper. This thing is being trained exactly once, and can’t learn dynamically. That’ll be the next step I think.

andreyk 4y ago
What are you are describing is pretty much reinforcement learning (or learning with access to a query-able knowledge engine, or active learning, or all of these combined). There is work on a bunch of variations of this, but it's true that it's early days for combining it with generative systems.
- namose 4y ago
  Yeah, I think the query-able knowledge engine is the key here, although I think it’s maybe more like “we haven’t figured out how to generalize conceptual learning”. The computer not only has to be able to query images on the internet, but also know how/what to query, which includes a bunch of actions computers are currently incapable of. In some cases, we might complete a drawing task by traveling to a new place, or taking an action (throw an egg at concrete) not querying the internet.

andreyk 4y ago

A couple of things: * We have seen more than 100,000 "photos" in the sense that photos are just images - if photos are just images, we have a constant feed of "photos" every single moment our eyes are open. Of course, that's not the same as these training datasets, but it is still worth keeping in mind. * All of these things trained on massive datasets with self-supervised learning are in a sense addressing the "slowness" of learning you mention, since self-supervised (aka no annotations are needed beyond the data itself) "pre-training" on the massive datasets can then enable training for downstream tasks with way less data. * Arguably requiring massive datasets for pre-training is still a bit lame, but then again the 4-5 years of life it takes to reach pretty advanced intelligence in humans represents a whoooole lot of data. And as with self-supervised learning on these massive models, a lot of intelligence seems to come down to learning to predict the future from sensory input. * Humans also come with a lot of pre-wiring done by evolution, whereas these models are trained from scratch. Evolutionary wiring represents its own sort of "pre-training", of course. So basically, it is not so slow to learn as it seems. Arguably it could get faster once we train multimodal models and concepts from text can reinforce learning to understand images and so on, and people are working on it (eg GATO). There may also need to be a separation between low level 'instinct' intelligence and high-level 'reasoning' intelligence; AI still sucks at the second one.

supernova87a 4y ago
I guess what I find really interesting is, how come we can start to self-label data we encounter in the wild, yet the DNN needs data to constantly be labeled at the same intensity per image, into the billions?
- simonw 4y ago
  If you look at the text labels for the data used by Stable Diffusion you'll find that they are very low quality. Take a look at some here: https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im... Clearly quality of labeling isn't nearly as important once you are training on billions of images.
- 4y ago

12ian34 4y ago

Most of us have our eyes open, looking at things ~16 hours a day, with the first 18 years of our lives heavily focused on learning about what those things are, plus we have the extra brain capacity to remember those things, and to think about them in an abstract manner. My entire photo library alone is over 100,000 photos - and since I took all of them I will have "seen" them.

Ukv 4y ago

> You and I have never seen even 100,000 photos in our lives If someone's 30, that'd only require seeing 10 images a day. For most people that quota is probably fulfilled within a couple of minutes of watching TV or browsing social media, even if the video stream from our eyes otherwise counts as nothing. We've also had about 4 billion years of evolution, slowly adjusting our genome with an unfathomable amount of data. Gradient descent is blazing fast by comparison.

omegalulw 4y ago
Add to that a video now a days is at least 24fps. So a two hour movie suffices haha :)

xipho 4y ago

> You and I have never seen even 100,000 photos in our lives. Well, maybe the video stream from our eyes is a little different. But it's not a billion fundamentally different images. I would argue precisely the opposite (as you alude to), it's more than 100s of billions of fundamentally (what does this even mean?) different images. Calculate the frequency at which your eyes sample, think of the times the angle changes (new images), multiple by your age, multiple by 2 for two eyes looking slightly different directions, factor in your the noise your brain has in forming the image "in your head" because you drank too much... you can continue adding factors (hours of "TV" watched on average) add-naseum. It seems that "slow to learn" has a "real" target/bound, what humans are capable of. If it takes Bob Ross decades to paint all the "images" in his head, then maybe we should go easy on the algorithims?

PeterisP 4y ago

Some experiments have demonstrated that being involved in the 'generation' of the visual data (i.e. choosing where to move, where to look, how to alter reality) gets significantly better learning than passively receiving the exact same visual data. Active learning is a good way to improve sample efficiency - however, as others note, don't underestimate the quantity of learning data that a human baby needs for certain skills even with good, evolution-optimized priors.