3 ms·

I don't understand why Yann is so focused on his "Animals are smarter than AI" analogy. If compute wasn't so limited, couldn't we just train a transformer on vi

by golol 3y ago

I don't understand why Yann is so focused on his "Animals are smarter than AI" analogy. If compute wasn't so limited, couldn't we just train a transformer on video, audio and text data? I don't see why it would not learn the basic physical structure of our world just like a language transformer learns the grammatical and other structures of language. Then with this prettaines transformer you can build an agent and use some reinforcement learning too. I feel very confident that this would mirror the level of intelligence of non-human animals quite well.

jerpint 3y ago
There have been many attempts to do multi modal pre training, the difficulty is finding the right combination of data for it to be “useful” and “scalable”. It’s not trivial to just train a transformer on video, text, audio, etc. mainly due to O(N^2) on the token counts, time components with video, etc
- golol 3y ago
  I don't really see it. O(N^2) for context window length is not an issue as you don't need particularly much longer context windows than for text. You don't have to run this at 30 FPS, 12 would already be enough to understand what's going on. The dimensionality is much higher of course, but how many dimensions does the latent space of TikTok videos really have? Train an autoencoder and take only 1000 dimensions for the frames. Of course it's not that simple but my mental models atm make me feel like this should work. What do I mean by work? A grainy low fps video+subtitles transformer that looks terrible but as a plus actually has some decent physical consistency. I guess that is Phenaki etc., but I'd hope for much better understanding if the "grammar" of the physical world. That should be rather orthogonal to FPS or resolution (or number of dimensions you take).
quickthrower2 3y ago
I know intuition is often wrong, but to me a mix of dedicated visual processing AI and language AI and sound AI all somehow interacting with each other would be a good way to make an “animal” like AI rather than throw loads of attention heads at everything all at once.
- golol 3y ago
  I'm just thinking of google's GATO, it seemed to have absolutely no trouble integrating a large number of modes.
nojvek 3y ago
Animals are indeed smarter than AI for certain aspects of intelligence (physical intelligence). Evolution has optimized animal brain and bodies to survive and take care of the next generation. They have a good grasp of environment, where they are, where food is, where predators are, basic communication if they live as a group. Babies grow up and start learning. Our current AI is extremely power hungry compared to a brain. Cruise & Waymo put large power hungry supercomputers in cars. The computing system costs 100k+. They still make silly mistakes like crashing into fire trucks, blocking roads, driving into wet cement etc. ChatGPT and friends make silly mistakes for trivial math problems that require a few hierarchical planning steps. All in all, brains have some form of symbolic computation and reasoning that we haven’t been able to replicate with current AI algorithms. I’m not saying we’ll never be able to but current AI is really hyped. Kinda like crypto boom of 2019. There are some really hard algorithmic problems to be solved. Google, Microsoft, Meta could have 1000x more computing power and data, however in the grand space or all algorithms there exists a learning algorithm that is probably >10000X more efficient at generalized modeling and reasoning than what we have. The proof that we (20W biological generally intelligent computers) exist validates the hypothesis that there is a lot of advancement we can still do at the algorithm part.

by golol 3y ago

jerpint 3y ago

There have been many attempts to do multi modal pre training, the difficulty is finding the right combination of data for it to be “useful” and “scalable”. It’s not trivial to just train a transformer on video, text, audio, etc. mainly due to O(N^2) on the token counts, time components with video, etc

golol 3y ago
I don't really see it. O(N^2) for context window length is not an issue as you don't need particularly much longer context windows than for text. You don't have to run this at 30 FPS, 12 would already be enough to understand what's going on. The dimensionality is much higher of course, but how many dimensions does the latent space of TikTok videos really have? Train an autoencoder and take only 1000 dimensions for the frames. Of course it's not that simple but my mental models atm make me feel like this should work. What do I mean by work? A grainy low fps video+subtitles transformer that looks terrible but as a plus actually has some decent physical consistency. I guess that is Phenaki etc., but I'd hope for much better understanding if the "grammar" of the physical world. That should be rather orthogonal to FPS or resolution (or number of dimensions you take).

quickthrower2 3y ago

I know intuition is often wrong, but to me a mix of dedicated visual processing AI and language AI and sound AI all somehow interacting with each other would be a good way to make an “animal” like AI rather than throw loads of attention heads at everything all at once.

golol 3y ago
I'm just thinking of google's GATO, it seemed to have absolutely no trouble integrating a large number of modes.

nojvek 3y ago

Animals are indeed smarter than AI for certain aspects of intelligence (physical intelligence). Evolution has optimized animal brain and bodies to survive and take care of the next generation. They have a good grasp of environment, where they are, where food is, where predators are, basic communication if they live as a group. Babies grow up and start learning. Our current AI is extremely power hungry compared to a brain. Cruise & Waymo put large power hungry supercomputers in cars. The computing system costs 100k+. They still make silly mistakes like crashing into fire trucks, blocking roads, driving into wet cement etc. ChatGPT and friends make silly mistakes for trivial math problems that require a few hierarchical planning steps. All in all, brains have some form of symbolic computation and reasoning that we haven’t been able to replicate with current AI algorithms. I’m not saying we’ll never be able to but current AI is really hyped. Kinda like crypto boom of 2019. There are some really hard algorithmic problems to be solved. Google, Microsoft, Meta could have 1000x more computing power and data, however in the grand space or all algorithms there exists a learning algorithm that is probably >10000X more efficient at generalized modeling and reasoning than what we have. The proof that we (20W biological generally intelligent computers) exist validates the hypothesis that there is a lot of advancement we can still do at the algorithm part.