3 ms·
Probably 5+ years unless there a significant compute or theoretical break through. Video is orders of magnitude more data than still images, and there aren’t a
by wokwokwok 4y ago
Probably 5+ years unless there a significant compute or theoretical break through. Video is orders of magnitude more data than still images, and there aren’t a lot of good labeled data sets to train on.
TikTok and instagram probably have the only meaningful (short form, highly annotated and quality marked) datasets and they won’t be sharing.
Maintaining long term coherence over time is an unsolved problem.
..but something else is going happen sooner (and it’s already happening, so the timescale is significantly shorter); and that is frame-by-frame mapping of existing video using diffusion models.
The issue is coherence between frames, but people are already addressing that by using carefully crafted “key frames” and the previous frame as an img2img prompt for the next frame.
There are already people taking crude “grey box” rendered animations and feeding them frame by frame into models to generate stylised video with “roughly this structure”.
Coherence over time isn’t there… but, it’ll get better.
I’ll be stunned if you don’t see this generating good quality videos before the end of the year.
Further more, since this basically renders the work of many VFX artists mostly irrelevant (from do everything to fix up little bits that didn’t make sense) you can bet on movie studios pouring a tonne of capital into it. You’ll see change in this space very quickly.