2 ms·
There have been many attempts to do multi modal pre training, the difficulty is finding the right combination of data for it to be “useful” and “scalable”. It’s
by jerpint 3y ago
There have been many attempts to do multi modal pre training, the difficulty is finding the right combination of data for it to be “useful” and “scalable”. It’s not trivial to just train a transformer on video, text, audio, etc. mainly due to O(N^2) on the token counts, time components with video, etc
- golol 3y agoI don't really see it. O(N^2) for context window length is not an issue as you don't need particularly much longer context windows than for text. You don't have to run this at 30 FPS, 12 would already be enough to understand what's going on. The dimensionality is much higher of course, but how many dimensions does the latent space of TikTok videos really have? Train an autoencoder and take only 1000 dimensions for the frames. Of course it's not that simple but my mental models atm make me feel like this should work. What do I mean by work? A grainy low fps video+subtitles transformer that looks terrible but as a plus actually has some decent physical consistency. I guess that is Phenaki etc., but I'd hope for much better understanding if the "grammar" of the physical world. That should be rather orthogonal to FPS or resolution (or number of dimensions you take).