4 ms·

> Yet, despite a decade-long effort, full neural-video compression—using deep learning—has not beat the best configurations of conventional codec standards in n

> Yet, despite a decade-long effort, full neural-video compression—using deep learning—has not beat the best configurations of conventional codec standards in normal conditions. Reviews from third parties show that when benchmarked with conventional distortion metrics as well as human opinion scores, conventional video encoders still outperform neural-network compression, especially when conventional encoders are enhanced with AI tools. I don't know how much better AI can do in practice. If you sit down and really think through how an intraframe compression technique works - something like JPEG or MPEG - You may come to the conclusion that we are already playing with the fringes of our reality. Consider The DCT and quantization. When a piece of information is transformed by The DCT, we talk about "energy compaction". That is to say, we took a block of information and determined how to represent it as a perfect (lossless) arrangement of amplitudes at given frequencies. This results in most of the important information (in human terms) condensing around the DC component in the top-left corner. We then use something explicitly engineered (i.e. 'trained') on the human visual system to quantize these figures into levels the computer can tolerate (incurring loss). Zig-zag encoding is practically an emergent mathematical mandate at this point... Then, take a long look at how we are doing interframe compression. Forward & backward difference frames, frames that are differences of differences, predicted vectors of information, etc. A lot of this is very obvious, but there are tradeoffs to be made that perhaps a human should be in charge of - I.e. subjective measures of latency in video streaming relative to group-of-picture size. Consider this - How could an AI system improve on techniques that optimize for the Human Visual System? A very large part of our multimedia compression today comes from strategically sacrificing information that is subjectively worthless to humans. All that said, I am sure you can find some low-hanging fruit. For example, if you go and ask ChatGPT to make JPEG work better, it may helpfully suggest swapping Huffman for arithmetic coding. Not exactly a groundbreaking discovery in my book, but it would make a big difference.

andyjohnson0 4y ago
DCT = Discrete Cosine Transform https://en.wikipedia.org/wiki/Discrete_cosine_transform https://en.wikipedia.org/wiki/Discrete_cosine_transform
ZeroGravitas 4y ago
Kind of tangential but: JPEG has always had arithmetic encoding in the spec. No one implemented it due to patents (even after the patents expired about a decade ago). Roughly 10% improvement I believe.
pixelchai 4y ago
> How could an AI system improve on techniques that optimize for the Human Visual System One way it may believed that AI may improve upon the traditional methods you mention is the assumption that AI can learn a tighter optimisation for the human system (than humans can handcraft)
CorrectHorseBat 4y ago
>Consider this - How could an AI system improve on techniques that optimize for the Human Visual System? A very large part of our multimedia compression today comes from strategically sacrificing information that is subjectively worthless to humans. Exactly that. AI could recognize what is important in a frame and use less compression there, while using pattern generation for high entropy parts which are not important.
- whitemary 4y ago
  We truly don't need AI to tell us what is important or not. We already know this, objectively. It's mathematically defined as the visible spectrum of the a available information, adjusted with relative high/low thresholds for compression level.
  - CuriouslyC 4y ago
    I think when they said important, they meant distinct or unique. For example, Vin Diesel driving muscle cars is the "important" part of the fast and furious movies, the city/plot/most of the supporting characters are details that could be simplified and reconstructed. An algorithm that generated FF movies could mostly hallucinate silly reasons for Vin Diesel to drive cars and shoot guns, with little real coherency or consistency and I bet the fanbase would love it. In the context of video, that means not spending bits on things like wall textures, plants, buildings, crowds, details of misc objects, etc.
  - CorrectHorseBat 4y ago
    Mathematical information density is not the same as important. A picture of white noise is mathematically very dense in information, but only median an standard deviation are important to us.
dylan604 4y ago
>I don't know how much better AI can do in practice. Coming soon, you show AI the script and the poster of the movie you want to compress, and then it just makes shit up on the fly when you hit play. There is no compression! The movie will be unique to each viewer as the algo will take a seed from each playback device and create the music, images, dialog all on the fly. A 2 hour movie in <1MB of storage

andyjohnson0 4y ago

DCT = Discrete Cosine Transform https://en.wikipedia.org/wiki/Discrete_cosine_transform https://en.wikipedia.org/wiki/Discrete_cosine_transform

ZeroGravitas 4y ago

Kind of tangential but: JPEG has always had arithmetic encoding in the spec. No one implemented it due to patents (even after the patents expired about a decade ago). Roughly 10% improvement I believe.

pixelchai 4y ago

> How could an AI system improve on techniques that optimize for the Human Visual System One way it may believed that AI may improve upon the traditional methods you mention is the assumption that AI can learn a tighter optimisation for the human system (than humans can handcraft)

CorrectHorseBat 4y ago

>Consider this - How could an AI system improve on techniques that optimize for the Human Visual System? A very large part of our multimedia compression today comes from strategically sacrificing information that is subjectively worthless to humans. Exactly that. AI could recognize what is important in a frame and use less compression there, while using pattern generation for high entropy parts which are not important.

whitemary 4y ago
We truly don't need AI to tell us what is important or not. We already know this, objectively. It's mathematically defined as the visible spectrum of the a available information, adjusted with relative high/low thresholds for compression level.
- CuriouslyC 4y ago
  I think when they said important, they meant distinct or unique. For example, Vin Diesel driving muscle cars is the "important" part of the fast and furious movies, the city/plot/most of the supporting characters are details that could be simplified and reconstructed. An algorithm that generated FF movies could mostly hallucinate silly reasons for Vin Diesel to drive cars and shoot guns, with little real coherency or consistency and I bet the fanbase would love it. In the context of video, that means not spending bits on things like wall textures, plants, buildings, crowds, details of misc objects, etc.
- CorrectHorseBat 4y ago
  Mathematical information density is not the same as important. A picture of white noise is mathematically very dense in information, but only median an standard deviation are important to us.

dylan604 4y ago

>I don't know how much better AI can do in practice. Coming soon, you show AI the script and the poster of the movie you want to compress, and then it just makes shit up on the fly when you hit play. There is no compression! The movie will be unique to each viewer as the algo will take a seed from each playback device and create the music, images, dialog all on the fly. A 2 hour movie in <1MB of storage