3 ms·

We must be reaching the limit at which video codecs can only achieve better quality by synthesizing details. That's already pretty prevalent in still images - p

We must be reaching the limit at which video codecs can only achieve better quality by synthesizing details. That's already pretty prevalent in still images - phone cameras do it, and there are lots of AI resizing algorithms that do it. It doesn't look like AV2 does any of that yet though fortunately (except film grain synthesis but I think that's fine).

meindnoch 1y ago
I think Fabrice Bellard have been experimenting with neural codecs like that.
- magicalhippo 1y ago
  The one I recall done by him was NNCP[1], a lossless compressor. [1]: https://bellard.org/nncp/ https://bellard.org/nncp/
  - throwaway48476 1y ago
    Computings own Da Vinci.
ksec 1y ago
>We must be reaching the limit at which video codecs can only achieve better quality by synthesizing details. Not quite yet as shown in H.267. But at some point the computational requirement vs bandwidth saving benefits would no longer make sense.
nasretdinov 1y ago
Arguably that's already happening with film grain — you have to extrapolate _what the original probably was_, encode it because it's smaller, then add the noise back to be more faithful to the original despite your image being better. I imagine e.g. a picture of an 8x8 circle actually takes more bits to encode than a mathematical description of the same circle
- rasz 1y ago
  >I imagine e.g. a picture of an 8x8 circle actually takes more bits to encode than a mathematical description of the same circle I wonder if there are codecs with provisions for storing common shapes. Text comes to mind - I imagine having a bank of 10 most popular fonts an encoding just the difference between source and text + distortion could save quite a lot of data on text heavy material. Add circles, lines, basic face shapes.
_kb 1y ago
Outside of AV1/2 (and linear media in general) that's already well and truly developed tech. Nvidia DLSS, AMD FSR and Intel XeSS all provide spatial/temporal super sampling to process lower fidelity base renders [0]. There also seems to be a fair bit of attention on that problem space from the real-time comms vendors with Cisco [1], Microsoft [2] and Google [3] already leaning on model based audio codecs. With the advantages that provides both around packet loss mitigation and shifting costs to end user (aka free) compute and away from central infra I can't see that not extending to the video channel too. [0]: https://mtisoftware.com/understanding-ai-upscaling-how-dlss-fsr-xess-boost-gaming-fps/ https://mtisoftware.com/understanding-ai-upscaling-how-dlss-... [1]: https://www.webex.com/gp/webex-ai-codec.html https://www.webex.com/gp/webex-ai-codec.html [2]: https://techcommunity.microsoft.com/blog/microsoftteamsblog/satin-microsoft%E2%80%99s-latest-ai-powered-audio-codec-for-real-time-communications/2141382 https://techcommunity.microsoft.com/blog/microsoftteamsblog/... [3]: https://research.google/blog/lyra-a-new-very-low-bitrate-codec-for-speech-compression/ https://research.google/blog/lyra-a-new-very-low-bitrate-cod...
illiac786 1y ago
Please no. This is what jbig2 does for images and it’s a nightmare in my view, you can’t trust the result is not something totally different from the original [1] [1]https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres...

meindnoch 1y ago

I think Fabrice Bellard have been experimenting with neural codecs like that.

magicalhippo 1y ago
The one I recall done by him was NNCP[1], a lossless compressor. [1]: https://bellard.org/nncp/ https://bellard.org/nncp/
- throwaway48476 1y ago
  Computings own Da Vinci.

ksec 1y ago

>We must be reaching the limit at which video codecs can only achieve better quality by synthesizing details. Not quite yet as shown in H.267. But at some point the computational requirement vs bandwidth saving benefits would no longer make sense.

nasretdinov 1y ago

Arguably that's already happening with film grain — you have to extrapolate _what the original probably was_, encode it because it's smaller, then add the noise back to be more faithful to the original despite your image being better. I imagine e.g. a picture of an 8x8 circle actually takes more bits to encode than a mathematical description of the same circle

rasz 1y ago
>I imagine e.g. a picture of an 8x8 circle actually takes more bits to encode than a mathematical description of the same circle I wonder if there are codecs with provisions for storing common shapes. Text comes to mind - I imagine having a bank of 10 most popular fonts an encoding just the difference between source and text + distortion could save quite a lot of data on text heavy material. Add circles, lines, basic face shapes.

_kb 1y ago

Outside of AV1/2 (and linear media in general) that's already well and truly developed tech. Nvidia DLSS, AMD FSR and Intel XeSS all provide spatial/temporal super sampling to process lower fidelity base renders [0]. There also seems to be a fair bit of attention on that problem space from the real-time comms vendors with Cisco [1], Microsoft [2] and Google [3] already leaning on model based audio codecs. With the advantages that provides both around packet loss mitigation and shifting costs to end user (aka free) compute and away from central infra I can't see that not extending to the video channel too. [0]: https://mtisoftware.com/understanding-ai-upscaling-how-dlss-fsr-xess-boost-gaming-fps/ https://mtisoftware.com/understanding-ai-upscaling-how-dlss-... [1]: https://www.webex.com/gp/webex-ai-codec.html https://www.webex.com/gp/webex-ai-codec.html [2]: https://techcommunity.microsoft.com/blog/microsoftteamsblog/satin-microsoft%E2%80%99s-latest-ai-powered-audio-codec-for-real-time-communications/2141382 https://techcommunity.microsoft.com/blog/microsoftteamsblog/... [3]: https://research.google/blog/lyra-a-new-very-low-bitrate-codec-for-speech-compression/ https://research.google/blog/lyra-a-new-very-low-bitrate-cod...

illiac786 1y ago

Please no. This is what jbig2 does for images and it’s a nightmare in my view, you can’t trust the result is not something totally different from the original [1] [1]https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres...