2 ms·
Coarsely detecting cuts is relatively simple - look for large frame-to-frame differences (e.g. encode however -> find large frames surrounded by smaller ones ->
by Groxx 6y ago
Coarsely detecting cuts is relatively simple - look for large frame-to-frame differences (e.g. encode however -> find large frames surrounded by smaller ones -> done, it's as accurate as your perceptual compression is). There are a number of ffmpeg-using tools out there doing this and other "cut to / from black" detection and it's pretty good. Not good enough for a human to say "yeah, these are all scenes", but probably good enough for picking things to re-encode like this.
The harder part is the significantly increased compute use due to re-encoding things multiple times, to detect these cuts and to try to find the best encoding. Heuristics there can be arbitrarily complex and re-calculate any number of times. I imagine it hasn't been done earlier just due to cost, though maybe they've recently achieved a better heuristic.
edit: ah, great, they link to a "dynamic optimizer" post that goes into this in some detail: https://netflixtechblog.com/dynamic-optimizer-a-perceptual-video-encoding-optimization-framework-e19f1e3a277f https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v...
- myself248 6y agoSeems to me that if the frame-to-frame difference isn't big enough to detect that way, it's not likely to benefit from a new I-frame, yeah?
- Groxx 6y agoThat's the basic idea, yeah. It falls apart in a couple places, e.g. when the cut or fast-fade goes to a very cheap frame like a mostly solid color, and it may not detect stuff like whip-cuts (since a whole chunk of frames are expensive), but so many scenes in so much of media has single-frame cuts that it's well within that "good enough" range. And for dynamic encoding like this: when it's wrong, it's not visually worse in that scene than choosing that sub-par encoding for the entire movie, which has the same "choose the best encoding" problem as individual chunks have. I assume it'd be relatively rare for it to result in anything worse than a one-shot strategy. --- ffmpeg will let you easily do frame-to-frame-diff logic that lets you chop videos into scenes, for example: https://video.stackexchange.com/a/30701 https://video.stackexchange.com/a/30701 I'm not sure how much it handles compressed-frame differences, but it shouldn't be too hard to build around it. Just might be a bit beyond bash-friendly.
- hinkley 6y agoImagine if the compression for the movie “Hero” couldn’t exploit the color themes in each act of the movie. That would be a might bigger.
- nhoughto 6y agoInteresting that it makes sense now because the economics of it have shifted, as the shift away from broadcast and towards individual stream has happened.