4 ms·
If decode is becoming so complicated and expensive the hardware can't handle it, why not just go full neural, send latents, and run decode on tensor cores? The
by avaer 5mo ago
If decode is becoming so complicated and expensive the hardware can't handle it, why not just go full neural, send latents, and run decode on tensor cores?
The answer is probably the same as for why not AV2 everything; a lot of hardware couldn't support it today. But in 10 years?
It seems we're running up against fundamental limits of human-engineered video codecs at this point. There might be a lesson in there.
- Dylan16807 5mo agoWhat makes you think that would use less resources? And it's not really hardware hitting limits, it's specifically software decoding on somewhat weaker machines.
- avaer 5mo agoIt could help in the same way that running things in a shader uses less resources than running it on CPU. You're flattening the pipeline, and you can optimize better with simpler primitives and less branches.
- itishappy 5mo agoThis is a software decoder designed to run on general purpose hardware. Adding custom hardware like tensor cores to the stack would serves a different use case.
- avaer 5mo agoRight, I'm sure it's way faster in specialized hardware. What I'm saying is the performance problem is a "code smell". The algorithms are getting so complicated that perhaps we are approaching fundamental limits of heuristiccs; we might get better + faster results ditching "smart" algorithms and just learning the codec in a much higher dimensional space. Again specialized hardware, but a different approach to it.
- Cthulhu_ 5mo agoI wouldn't want it myself, but, video games already do something like this - AI upscaling. Render the game at lower resolution / quality settings, then use AI to upscale and fill in the blanks. This could work / works for video too, give it lower resolution / quality images and AI upscale. Its predecessor would calculate intermediate frames for example.