5 ms·

“Compression is understanding”. As our understanding of the problem domain increases, so too will our ability to compress it. Video has three key things: the st

“Compression is understanding”. As our understanding of the problem domain increases, so too will our ability to compress it. Video has three key things: the still images that approximate the input, the relationship of one image to the prior and next and a human bean who is sensitive to some kinds of distortion and not others, depending on context :) To have effective and good compression, you have to understand all three. Note: while I work at Netflix, I do not work on anything related to video encoding, these are just my semi-informed understandings :)

sephamorr 6y ago
I would also add computational complexity. We're increasingly willing to throw more transistors at video decoding and encoding due to their reduced cost and increased performance, so the algorithms can become more complicated, have larger perform larger searches, etc. If you think about it from a power perspective, the power cost of a bit of data transmitted to a mobile device is pretty large. I would expect to see video compression formats increase in complexity until the power cost of compressing away one more bit approaches the cost of transmitting one more bit over the network.
- aaronblohowiak 6y ago
  Well with the compression vs transmission/decompression ratio of video for stuff in the Netflix catalog, I think the inflection point is different than you suggest (not sure about YouTube.)
  - sephamorr 6y ago
    You're right. I was thinking about the decoding side (where total power for displaying the video should be minimized) but was unclear above.
FullyFunctional 6y ago
"Understanding" is an interesting way of putting it. Traditionally "compression" is defined in terms of entropy and for lossy compression (eg. audio/video) entropy is defined in terms of human perception. I would generalize your statement to "relationship of the past and future of the video stream" The main enabler of more advanced compression strategies is the higher performance available to the encoder. Decoders are essentially deterministic state machines. Encoders however have to search a large space of transformations to find ones that captures the entropy for a given situation. In the development of AV1 they called these transformation "tools" and the research encoder experimented with lots of them and only the most profitable ones made it into the standard. The compression can thus still evolve even with the AV1 standard frozen, just like how x264 and LAME have gotten so much better over their lifetime.
The_Colonel 6y ago
> Compression is understanding That's quite thought provoking. What do you see when the current generation codec is heavily bitrate constrained? First thing you'll notice are the ugly blocks with non-matching colors on the edges. Such primitive expression of lack of bitrate is showing quite clearly these codecs have very little understanding. With a codec of the (probably far) future where both encoder and decoder have good "understanding", you wouldn't see any blocks, lower color depth, smudged details or so. If some detail is missing because of constrained bitrate, decoder will just fill it in intelligently. It can go crazy far - when constrained with bit rate, the encoder can just send sort of a movie script - "Man stands on the beach and looks into the distance" and the decoder will "understand" it and render it in awesome detail including adding non-mentioned splashing waves, man's thinking expression and perhaps sunset (decoder effectively becomes little automated film director who takes missing information in movie script as artistic license). It will most probably look very different from the original movie, but it will be believable - if you didn't see the original then you won't really have a reason to suspect it's fake. With more bitrate available you can add more detail to the movie script - the man is black and old, he wears this and that which the decoder would take into account when reconstructing the video. At some point you're always constrained on the bitrate, but "understanding decoder" can always go deeper and fill in more realistic details when needed.
- aaronblohowiak 6y ago
  Exactly. What is the information that you need to preserve. When you think of this, the notion of lossy compression becomes much more understandable from a human+computer perspective. Another way to view the problem of compression is a quest to lower to kolmogrov complexity of the program that needs to run to generate the output that “represents” the original work “faithfully”. It is true that for lossless compression there are constraints on the size of the input and the size of the compressed artifact+decoder, but I believe for lossy compression, the fourth element is the perceptive capabilities (or preferences) of human minds.

sephamorr 6y ago

I would also add computational complexity. We're increasingly willing to throw more transistors at video decoding and encoding due to their reduced cost and increased performance, so the algorithms can become more complicated, have larger perform larger searches, etc. If you think about it from a power perspective, the power cost of a bit of data transmitted to a mobile device is pretty large. I would expect to see video compression formats increase in complexity until the power cost of compressing away one more bit approaches the cost of transmitting one more bit over the network.

aaronblohowiak 6y ago
Well with the compression vs transmission/decompression ratio of video for stuff in the Netflix catalog, I think the inflection point is different than you suggest (not sure about YouTube.)
- sephamorr 6y ago
  You're right. I was thinking about the decoding side (where total power for displaying the video should be minimized) but was unclear above.

FullyFunctional 6y ago

"Understanding" is an interesting way of putting it. Traditionally "compression" is defined in terms of entropy and for lossy compression (eg. audio/video) entropy is defined in terms of human perception. I would generalize your statement to "relationship of the past and future of the video stream" The main enabler of more advanced compression strategies is the higher performance available to the encoder. Decoders are essentially deterministic state machines. Encoders however have to search a large space of transformations to find ones that captures the entropy for a given situation. In the development of AV1 they called these transformation "tools" and the research encoder experimented with lots of them and only the most profitable ones made it into the standard. The compression can thus still evolve even with the AV1 standard frozen, just like how x264 and LAME have gotten so much better over their lifetime.

The_Colonel 6y ago

> Compression is understanding That's quite thought provoking. What do you see when the current generation codec is heavily bitrate constrained? First thing you'll notice are the ugly blocks with non-matching colors on the edges. Such primitive expression of lack of bitrate is showing quite clearly these codecs have very little understanding. With a codec of the (probably far) future where both encoder and decoder have good "understanding", you wouldn't see any blocks, lower color depth, smudged details or so. If some detail is missing because of constrained bitrate, decoder will just fill it in intelligently. It can go crazy far - when constrained with bit rate, the encoder can just send sort of a movie script - "Man stands on the beach and looks into the distance" and the decoder will "understand" it and render it in awesome detail including adding non-mentioned splashing waves, man's thinking expression and perhaps sunset (decoder effectively becomes little automated film director who takes missing information in movie script as artistic license). It will most probably look very different from the original movie, but it will be believable - if you didn't see the original then you won't really have a reason to suspect it's fake. With more bitrate available you can add more detail to the movie script - the man is black and old, he wears this and that which the decoder would take into account when reconstructing the video. At some point you're always constrained on the bitrate, but "understanding decoder" can always go deeper and fill in more realistic details when needed.

aaronblohowiak 6y ago
Exactly. What is the information that you need to preserve. When you think of this, the notion of lossy compression becomes much more understandable from a human+computer perspective. Another way to view the problem of compression is a quest to lower to kolmogrov complexity of the program that needs to run to generate the output that “represents” the original work “faithfully”. It is true that for lossless compression there are constraints on the size of the input and the size of the compressed artifact+decoder, but I believe for lossy compression, the fourth element is the perceptive capabilities (or preferences) of human minds.