4 ms·

Not trolling, but I'd bet something that's augmented with generative AI. Not to the level of describing scenes with words, but context-aware interpolation.

by dehrmann 10mo ago

randall 10mo ago
for sure. macroblock hinting seems like a good place for research.
km3r 10mo ago
https://blogs.nvidia.com/blog/rtx-video-super-resolution/ https://blogs.nvidia.com/blog/rtx-video-super-resolution/ We already have some of the stepping stones for this. But honestly much better for upscaling poor quality streams vs just gives things a weird feeling when it is a better quality stream.
afiori 10mo ago
AI embeddings can be seen as a very advanced form of lossy compression
mort96 10mo ago
I don't want my video decoder inventing details which aren't there. I much rather want obvious compression artifacts than a codec where the "compression artifacts" look like perfectly realistic, high-quality hallucinated details.
- cubefox 10mo ago
  In case of many textures (grass, sand, hair, skin etc) it makes little difference whether the high frequency details are reproduced exactly or hallucinated. E.g. it doesn't matter whether the 1262nd blade of grass from the left side is bending to the left or to the right.
  - mort96 10mo ago
    And in the case of many others, it makes a very significant difference. And a codec doesn't have enough information to know. Imagine a criminal investigation. A witness happened to take a video as the perpetrator did the crime. In the video, you can clearly see a recognizable detail on the perpetrator's body in high quality; a birthmark perhaps. This rules out the main suspect -- but can we trust that the birthmark actually exists and isn't hallucinated? Would a non-AI codec have just showed a clearly compression-artifact-looking blob of pixels which can't be determined one way or the other? Or would a non-AI codec have contained actual image data of the birth mark in sufficient detail? Using AI to introduce realistic-looking details where there was none before (which is what your proposed AI codec inherently does) should never happen automatically.
    - cubefox 10mo ago
      Maybe there could be a "hallucination rate" parameter in the encoder: More hallucination would enable higher subjective image quality without increased accuracy. It could be used for Netflix streaming, where birthmarks and other forensic details don't matter because it's all just entertainment. Of course the hallucination parameter needs to be hard coded somehow in the output in order to determine its reliability.
    - beala 10mo ago
      There’s an infamous case of xerox photocopiers substituting in incorrect characters due to a poorly tuned compression algorithm. No AI necessary. https://en.wikipedia.org/wiki/JBIG2#:~:text=Character%20substitution%20errors%20in%20scanned%20documents https://en.wikipedia.org/wiki/JBIG2#:~:text=Character%20subs...
      - mort96 10mo ago
        Yeah, I had that case in mind actually. It's a perfect illustration of why compression artifacts should be obvious and not just realistic-looking hallucinations.
    - mapt 10mo ago
      > a codec doesn't have enough information to know. The material belief is that modern trained neural network methods that improve on ten generations of variations of the discrete cosine transform and wavelets, can bring a codec from "1% of knowing" to "5% of knowing". This is broadly useful. The level of abstraction does not need to be "The AI told the decoder to put a finger here", it may be "The AI told the decoder how to terminate the wrinkle on a finger here". An AI detail overlay. As we go from 1080p to 4K to 8K and beyond we care less and less about individual small-scale details being 100% correct, and there are representative elements that existing techniques are just really bad at squeezing into higher compression ratios. I don't claim that it's ideal, and the initial results left a lot to be desired in gaming (where latency and prediction is a Hard Problem), but AI upscaling is already routinely used for scene rips of older videos (from the VHS Age or the DVD Age), and it's clearly going to happen inside of a codec sooner or later.
- doopp 10mo ago
  A codec that uses AI isn't necessarily going to be using it to synthesize content. It could use it for things like improving rate-distortion optimizations and early-skip heuristics. Modern video codecs are complex beasts. It's not as simple as "take a macroblock and find some motion vectors that minimize residual below a certain threshold, else code as intra-block". They have hundreds of mutually-exclusive techniques to compress a certain area of the frame. Determining which technique will require the smallest residual is done with fast early-skip heuristics that often make the wrong decision. The official manual for x265, the H.265/HEVC encoder (https://x265.readthedocs.io/en/master/cli.html https://x265.readthedocs.io/en/master/cli.html), has literally hundreds of options, almost all about tuning these myriad of heuristics for your particular input. AI can be used to enhance things like early-skip heuristics. "This block looks like it'll benefit from a really-detailed motion search in this particular area" or "we'll save bits if we bypass the DCT step and quantize the block directly" or "this frame should definitely be a B-frame". Encoders already use heuristics to do this (brute forcing all possible decisions to find which is optimal is too slow), but they don't always make the best decision. An AI could be used to improve that. Now when I say AI, I'm not talking about massive, multi-billion weight monstrosities that synthesize nonsense, but extremely simple neural networks with a few thousand weights. The popular Opus codec uses a simple NN to estimate whether or not a frame of audio is speech or music, and uses that determination to decide whether to use their speech-optimized algorithm (SILK) or their music-optimized algorithm (CELT) to encode that particular frame. It's a short read but a very good one: https://jmvalin.ca/opus/opus-1.3/ https://jmvalin.ca/opus/opus-1.3/ This could be extended to video encoders without being used for interpolation where they would be liable to synthesize things that aren't there, DLSS-style.
cubefox 10mo ago
Neural codecs are indeed the future of audio and video compression. A lot of people / organizations are working on them and they are close to being practical. E.g. https://arxiv.org/abs/2502.20762 https://arxiv.org/abs/2502.20762

Not trolling, but I'd bet something that's augmented with generative AI. Not to the level of describing scenes with words, but context-aware interpolation.

randall 10mo ago

for sure. macroblock hinting seems like a good place for research.

km3r 10mo ago

https://blogs.nvidia.com/blog/rtx-video-super-resolution/ https://blogs.nvidia.com/blog/rtx-video-super-resolution/ We already have some of the stepping stones for this. But honestly much better for upscaling poor quality streams vs just gives things a weird feeling when it is a better quality stream.

afiori 10mo ago

AI embeddings can be seen as a very advanced form of lossy compression

mort96 10mo ago

I don't want my video decoder inventing details which aren't there. I much rather want obvious compression artifacts than a codec where the "compression artifacts" look like perfectly realistic, high-quality hallucinated details.

cubefox 10mo ago
In case of many textures (grass, sand, hair, skin etc) it makes little difference whether the high frequency details are reproduced exactly or hallucinated. E.g. it doesn't matter whether the 1262nd blade of grass from the left side is bending to the left or to the right.
- mort96 10mo ago
  And in the case of many others, it makes a very significant difference. And a codec doesn't have enough information to know. Imagine a criminal investigation. A witness happened to take a video as the perpetrator did the crime. In the video, you can clearly see a recognizable detail on the perpetrator's body in high quality; a birthmark perhaps. This rules out the main suspect -- but can we trust that the birthmark actually exists and isn't hallucinated? Would a non-AI codec have just showed a clearly compression-artifact-looking blob of pixels which can't be determined one way or the other? Or would a non-AI codec have contained actual image data of the birth mark in sufficient detail? Using AI to introduce realistic-looking details where there was none before (which is what your proposed AI codec inherently does) should never happen automatically.
  - cubefox 10mo ago
    Maybe there could be a "hallucination rate" parameter in the encoder: More hallucination would enable higher subjective image quality without increased accuracy. It could be used for Netflix streaming, where birthmarks and other forensic details don't matter because it's all just entertainment. Of course the hallucination parameter needs to be hard coded somehow in the output in order to determine its reliability.
  - beala 10mo ago
    There’s an infamous case of xerox photocopiers substituting in incorrect characters due to a poorly tuned compression algorithm. No AI necessary. https://en.wikipedia.org/wiki/JBIG2#:~:text=Character%20substitution%20errors%20in%20scanned%20documents https://en.wikipedia.org/wiki/JBIG2#:~:text=Character%20subs...
    - mort96 10mo ago
      Yeah, I had that case in mind actually. It's a perfect illustration of why compression artifacts should be obvious and not just realistic-looking hallucinations.
  - mapt 10mo ago
    > a codec doesn't have enough information to know. The material belief is that modern trained neural network methods that improve on ten generations of variations of the discrete cosine transform and wavelets, can bring a codec from "1% of knowing" to "5% of knowing". This is broadly useful. The level of abstraction does not need to be "The AI told the decoder to put a finger here", it may be "The AI told the decoder how to terminate the wrinkle on a finger here". An AI detail overlay. As we go from 1080p to 4K to 8K and beyond we care less and less about individual small-scale details being 100% correct, and there are representative elements that existing techniques are just really bad at squeezing into higher compression ratios. I don't claim that it's ideal, and the initial results left a lot to be desired in gaming (where latency and prediction is a Hard Problem), but AI upscaling is already routinely used for scene rips of older videos (from the VHS Age or the DVD Age), and it's clearly going to happen inside of a codec sooner or later.
doopp 10mo ago
A codec that uses AI isn't necessarily going to be using it to synthesize content. It could use it for things like improving rate-distortion optimizations and early-skip heuristics. Modern video codecs are complex beasts. It's not as simple as "take a macroblock and find some motion vectors that minimize residual below a certain threshold, else code as intra-block". They have hundreds of mutually-exclusive techniques to compress a certain area of the frame. Determining which technique will require the smallest residual is done with fast early-skip heuristics that often make the wrong decision. The official manual for x265, the H.265/HEVC encoder (https://x265.readthedocs.io/en/master/cli.html https://x265.readthedocs.io/en/master/cli.html), has literally hundreds of options, almost all about tuning these myriad of heuristics for your particular input. AI can be used to enhance things like early-skip heuristics. "This block looks like it'll benefit from a really-detailed motion search in this particular area" or "we'll save bits if we bypass the DCT step and quantize the block directly" or "this frame should definitely be a B-frame". Encoders already use heuristics to do this (brute forcing all possible decisions to find which is optimal is too slow), but they don't always make the best decision. An AI could be used to improve that. Now when I say AI, I'm not talking about massive, multi-billion weight monstrosities that synthesize nonsense, but extremely simple neural networks with a few thousand weights. The popular Opus codec uses a simple NN to estimate whether or not a frame of audio is speech or music, and uses that determination to decide whether to use their speech-optimized algorithm (SILK) or their music-optimized algorithm (CELT) to encode that particular frame. It's a short read but a very good one: https://jmvalin.ca/opus/opus-1.3/ https://jmvalin.ca/opus/opus-1.3/ This could be extended to video encoders without being used for interpolation where they would be liable to synthesize things that aren't there, DLSS-style.

cubefox 10mo ago

Neural codecs are indeed the future of audio and video compression. A lot of people / organizations are working on them and they are close to being practical. E.g. https://arxiv.org/abs/2502.20762 https://arxiv.org/abs/2502.20762