4 ms·
A couple of things: * We have seen more than 100,000 "photos" in the sense that photos are just images - if photos are just images, we have a constant feed of
by andreyk 4y ago
A couple of things:
* We have seen more than 100,000 "photos" in the sense that photos are just images - if photos are just images, we have a constant feed of "photos" every single moment our eyes are open. Of course, that's not the same as these training datasets, but it is still worth keeping in mind.
* All of these things trained on massive datasets with self-supervised learning are in a sense addressing the "slowness" of learning you mention, since self-supervised (aka no annotations are needed beyond the data itself) "pre-training" on the massive datasets can then enable training for downstream tasks with way less data.
* Arguably requiring massive datasets for pre-training is still a bit lame, but then again the 4-5 years of life it takes to reach pretty advanced intelligence in humans represents a whoooole lot of data. And as with self-supervised learning on these massive models, a lot of intelligence seems to come down to learning to predict the future from sensory input.
* Humans also come with a lot of pre-wiring done by evolution, whereas these models are trained from scratch. Evolutionary wiring represents its own sort of "pre-training", of course.
So basically, it is not so slow to learn as it seems. Arguably it could get faster once we train multimodal models and concepts from text can reinforce learning to understand images and so on, and people are working on it (eg GATO). There may also need to be a separation between low level 'instinct' intelligence and high-level 'reasoning' intelligence; AI still sucks at the second one.
- supernova87a 4y agoI guess what I find really interesting is, how come we can start to self-label data we encounter in the wild, yet the DNN needs data to constantly be labeled at the same intensity per image, into the billions?
- simonw 4y agoIf you look at the text labels for the data used by Stable Diffusion you'll find that they are very low quality. Take a look at some here: https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im... Clearly quality of labeling isn't nearly as important once you are training on billions of images.
- fragmede 4y agoTextual Inversion is about to flip that around (if I'm understanding the paper correctly). https://textual-inversion.github.io/ https://textual-inversion.github.io/
- andreyk 4y agoThis is only true for multi modal learning, but yeah, in that case we need text and image pairs. More than likely it's possible to pretrain image and language separately, and then use a vastly smaller number of pairs. But that's hypothetical.
- chse_cake 4y agoIf we consider a frame rate of 60 FPS then a 5 year old would have seen about ~ 6.3 billion images [60 (frames) * 60 (seconds) * 60 (minutes) * 16 (waking hours) * 365 (days) * 5 (years)]. Even with 30 FPS you can halve the number and it's still a huge number. A cool fact is that this model fit ~5B images in 900M parameter model which is tiny compared to the size of the data.
- fragmede 4y agoYeah. Humans take a long time to train. We spend years and years, starting at birth, just absorbing everything around us before we get to a point where we're considered adults.
- supernova87a 4y agoYet how do we do it with CPUs that are even less power consuming than ARM chips, in our heads?
- noobermin 4y agoBecause believe it or not matrices are not brains. People need to get over the metaphors. If you want to spend your time learn about the mathematics under the hood, there will be less "mysteries" then.
- erichocean 4y agoHuman brains are also pre-trained at birth, on faces and a whole bunch of other things.