5 ms·

ELI5? Edit: Why the downvotes?

2bitencryption 6y ago
Instead of one optimization profile for a whole movie, Netflix is detecting when the shot of a movie changes (the camera "cuts" from one shot to another) and beginning a new optimization profile specifically for the content of that shot. At least, that is my understanding.
- purerandomness 6y ago
  That sounds so intuitive that I want to follow up with the question: Why isn't this the default? Is detecting a cut particularly difficult?
  - dexterdog 6y ago
    Even if you have to pay somebody to tag the scene changes manually that cost would be pretty insignificant to Netflix.
    - MrMorden 6y ago
      Netflix should be able to get the EDL, which would probably take care of most of the work.
  - Groxx 6y ago
    Coarsely detecting cuts is relatively simple - look for large frame-to-frame differences (e.g. encode however -> find large frames surrounded by smaller ones -> done, it's as accurate as your perceptual compression is). There are a number of ffmpeg-using tools out there doing this and other "cut to / from black" detection and it's pretty good. Not good enough for a human to say "yeah, these are all scenes", but probably good enough for picking things to re-encode like this. The harder part is the significantly increased compute use due to re-encoding things multiple times, to detect these cuts and to try to find the best encoding. Heuristics there can be arbitrarily complex and re-calculate any number of times. I imagine it hasn't been done earlier just due to cost, though maybe they've recently achieved a better heuristic. edit: ah, great, they link to a "dynamic optimizer" post that goes into this in some detail: https://netflixtechblog.com/dynamic-optimizer-a-perceptual-video-encoding-optimization-framework-e19f1e3a277f https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v...
    - myself248 6y ago
      Seems to me that if the frame-to-frame difference isn't big enough to detect that way, it's not likely to benefit from a new I-frame, yeah?
      - Groxx 6y ago
        That's the basic idea, yeah. It falls apart in a couple places, e.g. when the cut or fast-fade goes to a very cheap frame like a mostly solid color, and it may not detect stuff like whip-cuts (since a whole chunk of frames are expensive), but so many scenes in so much of media has single-frame cuts that it's well within that "good enough" range. And for dynamic encoding like this: when it's wrong, it's not visually worse in that scene than choosing that sub-par encoding for the entire movie, which has the same "choose the best encoding" problem as individual chunks have. I assume it'd be relatively rare for it to result in anything worse than a one-shot strategy. --- ffmpeg will let you easily do frame-to-frame-diff logic that lets you chop videos into scenes, for example: https://video.stackexchange.com/a/30701 https://video.stackexchange.com/a/30701 I'm not sure how much it handles compressed-frame differences, but it shouldn't be too hard to build around it. Just might be a bit beyond bash-friendly.
      - hinkley 6y ago
        Imagine if the compression for the movie “Hero” couldn’t exploit the color themes in each act of the movie. That would be a might bigger.
    - nhoughto 6y ago
      Interesting that it makes sense now because the economics of it have shifted, as the shift away from broadcast and towards individual stream has happened.
  - iwasakabukiman 6y ago
    Traditionally, you just encode the video with a new frame every X frames, regardless of shots. It might sound trivial, but it is still extra computational work to figure out when the shot changes.
  - duhi88 6y ago
    "Variable bitrate" encoding is a thing, which allows chunks of a file to be encoded at a higher or lower bitrate depending on how much is happening. I don't know exactly how it happens, but I assume that each chunk is determined by duration or size, while Netflix's new method determines each chunk by the content. In order for this to be the default, you'd need either humans, or pattern recognition algorithms to identify the chunks. You also need to quantify how by how much each chunk needs its encoding parameters tweaked. Monetary costs aside, that's increasing complexity of your pipeline with relatively small gains. I'd bet that more companies will start looking at similar approaches now that 4K HDR (and 8K) are becoming more common. Probably not worth the R&D for 1080p, but we'll see tricks like this start to trickle down, I'm sure. It's highly unlikely we'll see anything similar in FOSS tools in the near future.
    - ehsankia 6y ago
      Right, these gains are mostly relevant when you have a relatively small library compared to how many views each item gets. For example this would probably not be worth it for Youtube, expect maybe on individual hyper popular videos. Every kb that Netflix can shave off of a popular movie means terabytes of bandwidth saved.
sreekotay 6y ago
isnt this.... Vbr?
- ehsankia 6y ago
  maybe VBR just works on constant sized chunks, whereas this works on chunks that are computed to be optimal.
  - sreekotay 6y ago
    I mean --- not really - adaptive scene detection for Vbr is a long standing thing.
hinkley 6y ago
You expect a video to play at exactly 1 second per second. But compression or network bandwidth can screw that up. What do you do? Ages ago they solved this by making some frames more important than others. Key frames being the top priority. If you are falling behind you can drop everything you are working on and skip to the next key frame. Then they added other priority levels so you can dump part of the work instead of all of it when things are only a little choppy. With a zip file if you don’t have the middle of the file you can’t figure out the end. So clearly video is doing something different if they can remove chunks and still work. Many of our compression techniques for video “hang” off the key frames. All the pictures around them are described as a set of changes to the key frame. Much smaller, but also why your screen occasionally looks like a horror show. Something went wrong and the changes were shown using the wrong starting point. If the key frame is right before the scene changes completely, then you will struggle to compress the transition. There’s not enough space to describe all the changes. But if you have a program look for the transitions, you can delay or advance the key frame a bit so it lands exactly on the camera change. Bang. New picture. Then everything looks right. Today it’s gone a lot farther. There are hints and patterns that span a whole movie or a chunk of it that for instance clues in that the whole movie is shot in sepia tone or a dark cave so you can guess the next scene will be sepia too. They’re talking about lining those attributes up with big shifts in the camera work, like someone coming out of a cave into bright light.

Instead of one optimization profile for a whole movie, Netflix is detecting when the shot of a movie changes (the camera "cuts" from one shot to another) and beginning a new optimization profile specifically for the content of that shot. At least, that is my understanding.

purerandomness 6y ago
That sounds so intuitive that I want to follow up with the question: Why isn't this the default? Is detecting a cut particularly difficult?
- dexterdog 6y ago
  Even if you have to pay somebody to tag the scene changes manually that cost would be pretty insignificant to Netflix.
  - MrMorden 6y ago
    Netflix should be able to get the EDL, which would probably take care of most of the work.
- Groxx 6y ago
  Coarsely detecting cuts is relatively simple - look for large frame-to-frame differences (e.g. encode however -> find large frames surrounded by smaller ones -> done, it's as accurate as your perceptual compression is). There are a number of ffmpeg-using tools out there doing this and other "cut to / from black" detection and it's pretty good. Not good enough for a human to say "yeah, these are all scenes", but probably good enough for picking things to re-encode like this. The harder part is the significantly increased compute use due to re-encoding things multiple times, to detect these cuts and to try to find the best encoding. Heuristics there can be arbitrarily complex and re-calculate any number of times. I imagine it hasn't been done earlier just due to cost, though maybe they've recently achieved a better heuristic. edit: ah, great, they link to a "dynamic optimizer" post that goes into this in some detail: https://netflixtechblog.com/dynamic-optimizer-a-perceptual-video-encoding-optimization-framework-e19f1e3a277f https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v...
  - myself248 6y ago
    Seems to me that if the frame-to-frame difference isn't big enough to detect that way, it's not likely to benefit from a new I-frame, yeah?
    - Groxx 6y ago
      That's the basic idea, yeah. It falls apart in a couple places, e.g. when the cut or fast-fade goes to a very cheap frame like a mostly solid color, and it may not detect stuff like whip-cuts (since a whole chunk of frames are expensive), but so many scenes in so much of media has single-frame cuts that it's well within that "good enough" range. And for dynamic encoding like this: when it's wrong, it's not visually worse in that scene than choosing that sub-par encoding for the entire movie, which has the same "choose the best encoding" problem as individual chunks have. I assume it'd be relatively rare for it to result in anything worse than a one-shot strategy. --- ffmpeg will let you easily do frame-to-frame-diff logic that lets you chop videos into scenes, for example: https://video.stackexchange.com/a/30701 https://video.stackexchange.com/a/30701 I'm not sure how much it handles compressed-frame differences, but it shouldn't be too hard to build around it. Just might be a bit beyond bash-friendly.
    - hinkley 6y ago
      Imagine if the compression for the movie “Hero” couldn’t exploit the color themes in each act of the movie. That would be a might bigger.
  - nhoughto 6y ago
    Interesting that it makes sense now because the economics of it have shifted, as the shift away from broadcast and towards individual stream has happened.
- iwasakabukiman 6y ago
  Traditionally, you just encode the video with a new frame every X frames, regardless of shots. It might sound trivial, but it is still extra computational work to figure out when the shot changes.
- duhi88 6y ago
  "Variable bitrate" encoding is a thing, which allows chunks of a file to be encoded at a higher or lower bitrate depending on how much is happening. I don't know exactly how it happens, but I assume that each chunk is determined by duration or size, while Netflix's new method determines each chunk by the content. In order for this to be the default, you'd need either humans, or pattern recognition algorithms to identify the chunks. You also need to quantify how by how much each chunk needs its encoding parameters tweaked. Monetary costs aside, that's increasing complexity of your pipeline with relatively small gains. I'd bet that more companies will start looking at similar approaches now that 4K HDR (and 8K) are becoming more common. Probably not worth the R&D for 1080p, but we'll see tricks like this start to trickle down, I'm sure. It's highly unlikely we'll see anything similar in FOSS tools in the near future.
  - ehsankia 6y ago
    Right, these gains are mostly relevant when you have a relatively small library compared to how many views each item gets. For example this would probably not be worth it for Youtube, expect maybe on individual hyper popular videos. Every kb that Netflix can shave off of a popular movie means terabytes of bandwidth saved.

sreekotay 6y ago

isnt this.... Vbr?

ehsankia 6y ago
maybe VBR just works on constant sized chunks, whereas this works on chunks that are computed to be optimal.
- sreekotay 6y ago
  I mean --- not really - adaptive scene detection for Vbr is a long standing thing.

hinkley 6y ago

You expect a video to play at exactly 1 second per second. But compression or network bandwidth can screw that up. What do you do? Ages ago they solved this by making some frames more important than others. Key frames being the top priority. If you are falling behind you can drop everything you are working on and skip to the next key frame. Then they added other priority levels so you can dump part of the work instead of all of it when things are only a little choppy. With a zip file if you don’t have the middle of the file you can’t figure out the end. So clearly video is doing something different if they can remove chunks and still work. Many of our compression techniques for video “hang” off the key frames. All the pictures around them are described as a set of changes to the key frame. Much smaller, but also why your screen occasionally looks like a horror show. Something went wrong and the changes were shown using the wrong starting point. If the key frame is right before the scene changes completely, then you will struggle to compress the transition. There’s not enough space to describe all the changes. But if you have a program look for the transitions, you can delay or advance the key frame a bit so it lands exactly on the camera change. Bang. New picture. Then everything looks right. Today it’s gone a lot farther. There are hints and patterns that span a whole movie or a chunk of it that for instance clues in that the whole movie is shot in sepia tone or a dark cave so you can guess the next scene will be sepia too. They’re talking about lining those attributes up with big shifts in the camera work, like someone coming out of a cave into bright light.