3 ms·

I'm a little surprised that we haven't seen something more AI driven yet. (yes yes buzzword I know) i.e. slice it not just into scenes but also into objects an

by Havoc 2y ago

I'm a little surprised that we haven't seen something more AI driven yet. (yes yes buzzword I know) i.e. slice it not just into scenes but also into objects and do bitrate on that level. i.e. Face and objects in foreground get more. It seems we now have pretty small models that can do that sort of stuff (see Apple & MS ones recently) so should be feasible at scale. I'd imagine you can also train an LLM on patterns that encoders choke on...chequered patterns etc.

cornstalks 2y ago
> i.e. Face and objects in foreground get more. Encoders have been doing that for decades at this point. AI-related models are actively being researched for multiple applications in video encoding. From generating predicted frames, to generating fine details, choosing tools and parameters for each frame, narrowing the search space, etc.
- Havoc 2y ago
  I know they do all sorts of clever predictive sorcery but I meant specifically what this image shows: https://deeplobe.ai/exploring-object-detection-applications-and-benefits/ https://deeplobe.ai/exploring-object-detection-applications-... Literally getting closer to understanding the context of the scene and what matters to a human rather than understanding the pixels
BoredPositron 2y ago
You mean transformer/diffuser based when you are talking about AI in this context? Because there are a lot of ml papers out there for encoding video. Stadia (RIP), Parsecs, Epic (Unreal Pixel streaming), META, Apple and Nvidia doing amazing stuff for their streaming services/VR/AR plays.
- Havoc 2y ago
  Yeah. This essentially https://deeplobe.ai/exploring-object-detection-applications-and-benefits/ https://deeplobe.ai/exploring-object-detection-applications-... If the model can figure out the key objects in the scene on a bit more deeper level then that can drive what parts the encoder needs to spend more bits on
kmeisthax 2y ago
After he got kicked out of MPEG, Leonardo Chiariglione opened his own standards organization[0] called MPAI, which is focused primarily on... standardizing machine learning techniques for video coding tools. I'm not sure if they've actually shipped anything yet, though. [0] With blackjack, and hookers: https://blog.chiariglione.org/the-mpai-framework-licence-approach-to-standard-essential-patent-sep-licensing/ https://blog.chiariglione.org/the-mpai-framework-licence-app...
coldsmoke 2y ago
We've experimented with that at SVT (Sweden's public service broadcaster). Or rather, the video team has, I'm just a web dev, so I don't know the details other than what's in this blog post: From the eyes of the viewer: Attention-aware video compression for improved low bitrate video https://medium.com/the-svt-tech-blog/from-the-eyes-of-the-viewer-attention-aware-video-compression-for-improved-low-bitrate-video-3375055e2d90 https://medium.com/the-svt-tech-blog/from-the-eyes-of-the-vi...
- sovereign_bits 2y ago
  Had to create an account just to ask more about this! Interesting approach, but the link is from last year and only talks about a research project. Did you ever test this on a larger scale? And do you know if it is only applicable to one encoder?
karmakaze 2y ago
Then we can have many 6-fingered characters and other hallucinations. It's all fiction anyways so what's one for a other? I hope it either gets great fast or phases out as a fad.
- Havoc 2y ago
  No not model to generate it. Model to steer the encoder and tell it what parts to invest more Bitrate into
  - karmakaze 2y ago
    That makes sense and shouldn't have been that difficult anyway. For the barbie glitter scene there's obviously more chroma 'texture' to apply variable bitrates. Flat colored scenes could continue using lower bitrates. Where more complex models could be useful is avoiding entropy coding of useless details etc (e.g. if those details were artifacts of processing in the first place).
- rowanG077 2y ago
  This is supremely easy to avoid. I'm not even sure how you arrived at thinking this will be a concern.
  - deleted 2y ago
    [deleted]
  - karmakaze 2y ago
    I didn't mean this one thing specifically. I was thinking more along the lines of Xerox copiers 'hallucinating' (randomly altering) digits in reports.
FrenchDevRemote 2y ago
The algorithms exists, but it requires way too much processing power and you also need to have a model already downloaded.

by Havoc 2y ago

cornstalks 2y ago

> i.e. Face and objects in foreground get more. Encoders have been doing that for decades at this point. AI-related models are actively being researched for multiple applications in video encoding. From generating predicted frames, to generating fine details, choosing tools and parameters for each frame, narrowing the search space, etc.

Havoc 2y ago
I know they do all sorts of clever predictive sorcery but I meant specifically what this image shows: https://deeplobe.ai/exploring-object-detection-applications-and-benefits/ https://deeplobe.ai/exploring-object-detection-applications-... Literally getting closer to understanding the context of the scene and what matters to a human rather than understanding the pixels

BoredPositron 2y ago

You mean transformer/diffuser based when you are talking about AI in this context? Because there are a lot of ml papers out there for encoding video. Stadia (RIP), Parsecs, Epic (Unreal Pixel streaming), META, Apple and Nvidia doing amazing stuff for their streaming services/VR/AR plays.

Havoc 2y ago
Yeah. This essentially https://deeplobe.ai/exploring-object-detection-applications-and-benefits/ https://deeplobe.ai/exploring-object-detection-applications-... If the model can figure out the key objects in the scene on a bit more deeper level then that can drive what parts the encoder needs to spend more bits on

kmeisthax 2y ago

After he got kicked out of MPEG, Leonardo Chiariglione opened his own standards organization[0] called MPAI, which is focused primarily on... standardizing machine learning techniques for video coding tools. I'm not sure if they've actually shipped anything yet, though. [0] With blackjack, and hookers: https://blog.chiariglione.org/the-mpai-framework-licence-approach-to-standard-essential-patent-sep-licensing/ https://blog.chiariglione.org/the-mpai-framework-licence-app...

coldsmoke 2y ago

We've experimented with that at SVT (Sweden's public service broadcaster). Or rather, the video team has, I'm just a web dev, so I don't know the details other than what's in this blog post: From the eyes of the viewer: Attention-aware video compression for improved low bitrate video https://medium.com/the-svt-tech-blog/from-the-eyes-of-the-viewer-attention-aware-video-compression-for-improved-low-bitrate-video-3375055e2d90 https://medium.com/the-svt-tech-blog/from-the-eyes-of-the-vi...

sovereign_bits 2y ago
Had to create an account just to ask more about this! Interesting approach, but the link is from last year and only talks about a research project. Did you ever test this on a larger scale? And do you know if it is only applicable to one encoder?

karmakaze 2y ago

Then we can have many 6-fingered characters and other hallucinations. It's all fiction anyways so what's one for a other? I hope it either gets great fast or phases out as a fad.

Havoc 2y ago
No not model to generate it. Model to steer the encoder and tell it what parts to invest more Bitrate into
- karmakaze 2y ago
  That makes sense and shouldn't have been that difficult anyway. For the barbie glitter scene there's obviously more chroma 'texture' to apply variable bitrates. Flat colored scenes could continue using lower bitrates. Where more complex models could be useful is avoiding entropy coding of useless details etc (e.g. if those details were artifacts of processing in the first place).
rowanG077 2y ago
This is supremely easy to avoid. I'm not even sure how you arrived at thinking this will be a concern.
- deleted 2y ago
  [deleted]
- karmakaze 2y ago
  I didn't mean this one thing specifically. I was thinking more along the lines of Xerox copiers 'hallucinating' (randomly altering) digits in reports.

FrenchDevRemote 2y ago

The algorithms exists, but it requires way too much processing power and you also need to have a model already downloaded.