4 ms·

I wonder if instead of heuristics, machine learning could be used to figure out the best parameters.

DavidVoid 6y ago
That seems likely at least. You could actually use ML for all of the video decoding, but that research is still in it's early stages. It has been done rather well with still images [1], so I'm sure it'll eventually be done with video too. Those ML techniques are still a little slow and require large networks (the one in [1] decodes to PNG at 0.7 megapixels/s and its network is 726MB) so more optimizations will be needed before they can see any real-world use. [1] https://hific.github.io/ https://hific.github.io/ HN thread: https://news.ycombinator.com/item?id=23652753 https://news.ycombinator.com/item?id=23652753
giantrobot 6y ago
That's already done today. Most modern codecs support variable bitrate encoding so more of the data budget can be given to high complexity scenes. Source video can also be processed multiple times with varied parameters and that output then compared structurally to the source to find the parameters that best encode the scene. This is beyond the more typical multi pass encoding where a first pass over the source just provides some encoding hints for the second pass. It takes several times longer to encode (though is embarrassingly parallel) but the output ends up higher quality for a given data rate than a more naïve approach.
- social_quotient 6y ago
  Any apps you can recommend for giving this a spin?
  - sp332 6y ago
    x264 with 2-pass encoding? "Machine learning" doesn't only mean deep neural networks.
proverbialbunny 6y ago
I was hoping H266 was going to be a neural network based approach, but it looks like that might end up being H267. Right now neural networks allow for higher compression for tailored content, so you need to ship a decoder with the video, or have several categories of decoders. The future is untold and it might end up not being done this way.
Sohcahtoa82 6y ago
In a somewhat-related topic, you might be interested in DLSS [0], where machine learning is being used in graphics rendering in games to draw the games at a lower resolution, then upscale the image to the monitor's resolution using a neural network to fill in the missing data. I imagine a similar thing could be done with video rendering, though you'd need some crazy computing power to train the neural network for each video, just like DLSS requires training for each game. [0] https://en.wikipedia.org/wiki/Deep_learning_super_sampling https://en.wikipedia.org/wiki/Deep_learning_super_sampling
fwip 6y ago
Sure. Machine Learning is just "heuristics we don't understand."
- giantrobot 6y ago
  Or more typically Monte Carlo heuristics.
  - mratsim 6y ago
    Reinforcement learning uses Monte Carlo a lot but traditional machine learning or deep learning don't.
IfOnlyYouKnew 6y ago
In the limit case, compression and AI are identical. Once you get to an AI that has full comprehension of what humans perceive to be reality, you can just give them a rough outline of a story, add some information on casting, writers, and Spielberg's mood during production, and they'll fill in the (rather large) blanks. That's a bit exaggerated, but I remember reading about one such algorithm a few days ago (by Netflix, maybe?). It was image compression that had internal representations such as "there is an oak tree on the left". It would then run the "decompression", find the differences to the original, and add further hints where neccessary.

That seems likely at least. You could actually use ML for all of the video decoding, but that research is still in it's early stages. It has been done rather well with still images [1], so I'm sure it'll eventually be done with video too. Those ML techniques are still a little slow and require large networks (the one in [1] decodes to PNG at 0.7 megapixels/s and its network is 726MB) so more optimizations will be needed before they can see any real-world use. [1] https://hific.github.io/ https://hific.github.io/ HN thread: https://news.ycombinator.com/item?id=23652753 https://news.ycombinator.com/item?id=23652753

giantrobot 6y ago

That's already done today. Most modern codecs support variable bitrate encoding so more of the data budget can be given to high complexity scenes. Source video can also be processed multiple times with varied parameters and that output then compared structurally to the source to find the parameters that best encode the scene. This is beyond the more typical multi pass encoding where a first pass over the source just provides some encoding hints for the second pass. It takes several times longer to encode (though is embarrassingly parallel) but the output ends up higher quality for a given data rate than a more naïve approach.

social_quotient 6y ago
Any apps you can recommend for giving this a spin?
- sp332 6y ago
  x264 with 2-pass encoding? "Machine learning" doesn't only mean deep neural networks.

proverbialbunny 6y ago

I was hoping H266 was going to be a neural network based approach, but it looks like that might end up being H267. Right now neural networks allow for higher compression for tailored content, so you need to ship a decoder with the video, or have several categories of decoders. The future is untold and it might end up not being done this way.

Sohcahtoa82 6y ago

In a somewhat-related topic, you might be interested in DLSS [0], where machine learning is being used in graphics rendering in games to draw the games at a lower resolution, then upscale the image to the monitor's resolution using a neural network to fill in the missing data. I imagine a similar thing could be done with video rendering, though you'd need some crazy computing power to train the neural network for each video, just like DLSS requires training for each game. [0] https://en.wikipedia.org/wiki/Deep_learning_super_sampling https://en.wikipedia.org/wiki/Deep_learning_super_sampling

fwip 6y ago

Sure. Machine Learning is just "heuristics we don't understand."

giantrobot 6y ago
Or more typically Monte Carlo heuristics.
- mratsim 6y ago
  Reinforcement learning uses Monte Carlo a lot but traditional machine learning or deep learning don't.

IfOnlyYouKnew 6y ago

In the limit case, compression and AI are identical. Once you get to an AI that has full comprehension of what humans perceive to be reality, you can just give them a rough outline of a story, add some information on casting, writers, and Spielberg's mood during production, and they'll fill in the (rather large) blanks. That's a bit exaggerated, but I remember reading about one such algorithm a few days ago (by Netflix, maybe?). It was image compression that had internal representations such as "there is an oak tree on the left". It would then run the "decompression", find the differences to the original, and add further hints where neccessary.