4 ms·

Keyframes, InterFrame and Video Compression (2016)

17 points by woranl 6y ago

nitrogen 6y ago
Long ago I read something on HN about encoding x264 without any keyframes at all, and instead there are equivalent blocks that are swept across the image over many frames. But I can't find that reference. Does anyone know what that technique is called and/or where to read more about it?
- londons_explore 6y ago
  I'm not sure what the benefit would be... Sounds like a cool proof of concept though. In fact, with very very tiny modifications (ie. modifying the bit of the spec that says 'frame 1 must be a keyframe' and replacing it with 'the first frame is initialized with solid black'), I believe any modern video compression scheme could be used with no key frames. Compression ratio wouldn't suffer, but you'd loose the ability to seek the video.
  - nullc 6y ago
    The benefit is minimizing latency on a bandwidth constrained channel-- it also has some robustness benefits. Imagine we have a 1mbps link between us. We'd like have fluent video conferencing so we really want to have under 200ms latency. If keyframes are 10x the size of non-keyframes we'll be stuck either massively short changing the bitrate on keyframes, underutilizing our link (say using a 100kbps video), or taking a large amount of delay so that the keyframe's usage could be averaged out over the next $key_int frames. Instead, if part of the frame is constantly being updated you'll still recover from loss in finite time, but the bitrate will be more constant. > Compression ratio wouldn't suffer, but you'd loose the ability to seek the video. Just having none means you can't recover from loss-- and in any case where you care about latency, you'll likely have some loss because retransmission creates delay (and forward error correction consumes bandwidth).
    - londons_explore 6y ago
      > If keyframes are 10x the size of non-keyframes The solution to that is "non displayed frames", which VP8 allows. It means you can have a key frame which isn't part of the displayed video feed. You then send a super low res keyframe, and keep increasing the quality with further non-displayed frames until the quality is good enough to use as a reference frame for the real displayed frames. That effectively lets you spread the bandwidth of a key frame over many seconds without introducing latency. The downside is both the encoder and the decoder need to do more computation to decode all those extra frames.
    - londons_explore 6y ago
      > Instead, if part of the frame is constantly being updated you'll still recover from loss in finite time Modern video codecs tend not to be self-stabilising. For example, most codecs use some kind of huffman tables or arithmetic coding. If the probabilities for those tables depend on previous frames, and you don't know the probabilities, the resulting decode will be entirely junk, and won't get you any closer to being able to decode the next frame either.
      - nullc 6y ago
        I was insufficiently precise in my description, but having implemented it in one (admittedly simple) codec I can assure you it works. ::grin:: When this kind of rolling keyframe is used the encoder needs to track the reference structure of the video and not make any references into the "dirty zone"-- this includes constraining coding modes so that no entropy context leaks. This can be aided by (or necessitate the use of) 'tiled'/'sliced' entropy coding features that some codecs have for parallelism reasons. There is, of course, a coding performance loss-- especially if the video pans against the update boundary.
- nullc 6y ago
  I called it "rolling intra" when I implemented it with libtheora. In x264 land it's apparently called "periodic intra refresh": https://web.archive.org/web/20101017210542/http://git.videolan.org/?p=x264.git;a=commit;h=52a4fbe142926435571fbf2fb3d535e8873627d3 https://web.archive.org/web/20101017210542/http://git.videol...
- barbegal 6y ago
  This isn't possible H.264 is required to start with a key frame. Having said that you can have an H.264 stream with a single key frame.
  - mdonahoe 6y ago
    Implementations I’ve seen use a gray I-frame to bootstrap the stream, then the p-frames will completely replace it after a complete N frame cycle (typically N is 5) The frames are divided into slices where one strip of the frame contains complete blocks, but the rest is motion predicted.

17 points by woranl 6y ago

Long ago I read something on HN about encoding x264 without any keyframes at all, and instead there are equivalent blocks that are swept across the image over many frames. But I can't find that reference. Does anyone know what that technique is called and/or where to read more about it?

londons_explore 6y ago
I'm not sure what the benefit would be... Sounds like a cool proof of concept though. In fact, with very very tiny modifications (ie. modifying the bit of the spec that says 'frame 1 must be a keyframe' and replacing it with 'the first frame is initialized with solid black'), I believe any modern video compression scheme could be used with no key frames. Compression ratio wouldn't suffer, but you'd loose the ability to seek the video.
- nullc 6y ago
  The benefit is minimizing latency on a bandwidth constrained channel-- it also has some robustness benefits. Imagine we have a 1mbps link between us. We'd like have fluent video conferencing so we really want to have under 200ms latency. If keyframes are 10x the size of non-keyframes we'll be stuck either massively short changing the bitrate on keyframes, underutilizing our link (say using a 100kbps video), or taking a large amount of delay so that the keyframe's usage could be averaged out over the next $key_int frames. Instead, if part of the frame is constantly being updated you'll still recover from loss in finite time, but the bitrate will be more constant. > Compression ratio wouldn't suffer, but you'd loose the ability to seek the video. Just having none means you can't recover from loss-- and in any case where you care about latency, you'll likely have some loss because retransmission creates delay (and forward error correction consumes bandwidth).
  - londons_explore 6y ago
    > If keyframes are 10x the size of non-keyframes The solution to that is "non displayed frames", which VP8 allows. It means you can have a key frame which isn't part of the displayed video feed. You then send a super low res keyframe, and keep increasing the quality with further non-displayed frames until the quality is good enough to use as a reference frame for the real displayed frames. That effectively lets you spread the bandwidth of a key frame over many seconds without introducing latency. The downside is both the encoder and the decoder need to do more computation to decode all those extra frames.
  - londons_explore 6y ago
    > Instead, if part of the frame is constantly being updated you'll still recover from loss in finite time Modern video codecs tend not to be self-stabilising. For example, most codecs use some kind of huffman tables or arithmetic coding. If the probabilities for those tables depend on previous frames, and you don't know the probabilities, the resulting decode will be entirely junk, and won't get you any closer to being able to decode the next frame either.
    - nullc 6y ago
      I was insufficiently precise in my description, but having implemented it in one (admittedly simple) codec I can assure you it works. ::grin:: When this kind of rolling keyframe is used the encoder needs to track the reference structure of the video and not make any references into the "dirty zone"-- this includes constraining coding modes so that no entropy context leaks. This can be aided by (or necessitate the use of) 'tiled'/'sliced' entropy coding features that some codecs have for parallelism reasons. There is, of course, a coding performance loss-- especially if the video pans against the update boundary.
nullc 6y ago
I called it "rolling intra" when I implemented it with libtheora. In x264 land it's apparently called "periodic intra refresh": https://web.archive.org/web/20101017210542/http://git.videolan.org/?p=x264.git;a=commit;h=52a4fbe142926435571fbf2fb3d535e8873627d3 https://web.archive.org/web/20101017210542/http://git.videol...
barbegal 6y ago
This isn't possible H.264 is required to start with a key frame. Having said that you can have an H.264 stream with a single key frame.
- mdonahoe 6y ago
  Implementations I’ve seen use a gray I-frame to bootstrap the stream, then the p-frames will completely replace it after a complete N frame cycle (typically N is 5) The frames are divided into slices where one strip of the frame contains complete blocks, but the rest is motion predicted.