4 ms·

At a very (very!) abstract level AI is compression and compression is AI. There's been rumblings about how Google's AI edge would lead to jumps in video and im

At a very (very!) abstract level AI is compression and compression is AI. There's been rumblings about how Google's AI edge would lead to jumps in video and image coding but nothing big seems to have happened yet, though lots of micro-optimisations appear to be ML based. Some suggested that the AV1 throw a bunch of tools at it spec design was built for AI to use. Possibly VMAF is the biggest ML impact on codecs so far.

bick_nyers 4y ago
A full AI video codec redesign could yield some powerful benefits (especially on the generative side), but there are solid gains even on the existing codecs. Shot or scene based detection can be used to place your I, P, and B frames in better locations for example. For those that don't know, an I frame is a keyframe, a still image. P and B frame are predictions off of other frames (such as a keyframe). Frames within the same shot or scene are likely to be similar to one another, so having awareness of that contextual information can help you place for example a keyframe at precisely the start of that shot or scene, and predict everything in that shot or scene off of that.
- astrange 4y ago
  I-frame placement isn't hard; it's very obvious when a scene changes or when an I-frame would be profitable. You generally need to have them far more often than you'd like anyway, as otherwise you can't seek in videos. The main advantage of ML for video compression is that people accept giant ML models, so you can hide a lot of common "features" in there. The disadvantage is that classical codecs are macroblock-based for a reason; you can't just process a whole frame at a time, nobody has the memory bandwidth for that.
echelon 4y ago
> At a very (very!) abstract level AI is compression and compression is AI. I think it's not so abstract at all. The relationship feels fundamental. Encoder, decoder, codec. Compression, entropy, lossiness, ...
- CuriouslyC 4y ago
  They share find low bit count representations of information. The difference is everything else.
retrac 4y ago
> nothing big seems to have happened yet There's been some cool stuff in the speech area: https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-codec-for.html https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-... Also from Microsoft: https://techcommunity.microsoft.com/t5/microsoft-teams-blog/satin-microsoft-s-latest-ai-powered-audio-codec-for-real-time/ba-p/2141382 https://techcommunity.microsoft.com/t5/microsoft-teams-blog/...
lrog 4y ago
> At a very (very!) abstract level AI is compression and compression is AI. There's actually a pretty nice read from one of the key figures in data compression field - Matt Mahoney: "Text Compression as a Test for Artificial Intelligence" [1]. Some interesting reads are also on his personal website, though a bit outdated now: https://mattmahoney.net/dc/rationale.html https://mattmahoney.net/dc/rationale.html [1] https://static.aminer.org/pdf/PDF/000/014/009/text_compression_as_a_test_for_artificial_intelligence.pdf https://static.aminer.org/pdf/PDF/000/014/009/text_compressi...
3rd3 4y ago
AI = compression + goal. Though strictly speaking I think compression is only required in partially-observable and/or stochastic worlds where the information content of the world (far) exceeds the actor's model capacity.

bick_nyers 4y ago

A full AI video codec redesign could yield some powerful benefits (especially on the generative side), but there are solid gains even on the existing codecs. Shot or scene based detection can be used to place your I, P, and B frames in better locations for example. For those that don't know, an I frame is a keyframe, a still image. P and B frame are predictions off of other frames (such as a keyframe). Frames within the same shot or scene are likely to be similar to one another, so having awareness of that contextual information can help you place for example a keyframe at precisely the start of that shot or scene, and predict everything in that shot or scene off of that.

astrange 4y ago
I-frame placement isn't hard; it's very obvious when a scene changes or when an I-frame would be profitable. You generally need to have them far more often than you'd like anyway, as otherwise you can't seek in videos. The main advantage of ML for video compression is that people accept giant ML models, so you can hide a lot of common "features" in there. The disadvantage is that classical codecs are macroblock-based for a reason; you can't just process a whole frame at a time, nobody has the memory bandwidth for that.

echelon 4y ago

> At a very (very!) abstract level AI is compression and compression is AI. I think it's not so abstract at all. The relationship feels fundamental. Encoder, decoder, codec. Compression, entropy, lossiness, ...

CuriouslyC 4y ago
They share find low bit count representations of information. The difference is everything else.

retrac 4y ago

> nothing big seems to have happened yet There's been some cool stuff in the speech area: https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-codec-for.html https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-... Also from Microsoft: https://techcommunity.microsoft.com/t5/microsoft-teams-blog/satin-microsoft-s-latest-ai-powered-audio-codec-for-real-time/ba-p/2141382 https://techcommunity.microsoft.com/t5/microsoft-teams-blog/...

lrog 4y ago

> At a very (very!) abstract level AI is compression and compression is AI. There's actually a pretty nice read from one of the key figures in data compression field - Matt Mahoney: "Text Compression as a Test for Artificial Intelligence" [1]. Some interesting reads are also on his personal website, though a bit outdated now: https://mattmahoney.net/dc/rationale.html https://mattmahoney.net/dc/rationale.html [1] https://static.aminer.org/pdf/PDF/000/014/009/text_compression_as_a_test_for_artificial_intelligence.pdf https://static.aminer.org/pdf/PDF/000/014/009/text_compressi...

3rd3 4y ago

AI = compression + goal. Though strictly speaking I think compression is only required in partially-observable and/or stochastic worlds where the information content of the world (far) exceeds the actor's model capacity.