4 ms·

A full AI video codec redesign could yield some powerful benefits (especially on the generative side), but there are solid gains even on the existing codecs. S

A full AI video codec redesign could yield some powerful benefits (especially on the generative side), but there are solid gains even on the existing codecs. Shot or scene based detection can be used to place your I, P, and B frames in better locations for example. For those that don't know, an I frame is a keyframe, a still image. P and B frame are predictions off of other frames (such as a keyframe). Frames within the same shot or scene are likely to be similar to one another, so having awareness of that contextual information can help you place for example a keyframe at precisely the start of that shot or scene, and predict everything in that shot or scene off of that.

astrange 4y ago
I-frame placement isn't hard; it's very obvious when a scene changes or when an I-frame would be profitable. You generally need to have them far more often than you'd like anyway, as otherwise you can't seek in videos. The main advantage of ML for video compression is that people accept giant ML models, so you can hide a lot of common "features" in there. The disadvantage is that classical codecs are macroblock-based for a reason; you can't just process a whole frame at a time, nobody has the memory bandwidth for that.