4 ms·

Has anyone thought of trying to end-to-end train an h.265 decoder? The results might not be 100% perfect, but the resulting codec might bypass a ton of patents.

vletal 5y ago
Since the h265 relies heavily on operations which are not easily differentiable, such as translation of patches of images, together with a pretty complicated binary format, I'd be pretty amazed if the NN actually learned anything meaningful at all.
- svantana 5y ago
  Interpolated translation is continuous and easily differentiated. There's lots of work on machine-learned video codecs already, from Nvidia, Qualcomm and others.
  - vletal 5y ago
    It's not difficult to propagate gradients while translating an image. Learning "pick a 8x8 patch from (145,17) apply X to it and translate it by (4,-8)" from data is on different level, is not it? The premise was using e2e learning to avoid patent issues. I am sure that with some preprocessing you can plug a NN inside the deciding process and learn very meaningful stuff.
  - zinekeller 5y ago
    > There's lots of work on machine-learned video codecs already, from Nvidia, Qualcomm and others. I've tried them, the current state of the art means that they're only useful on relatively static things (some shaking etc) while spike up to AV1-level bitrate to reach perceptual similarity when the movement is too much. Maybe in the future (or whatever under the wraps concoction Nvidia, Qualcomm or another player have), ML-based video codecs will surpass handtuned codecs, but it's not (yet) the present state.
itsamesicario 5y ago
Aren't software implementations royalty free anyway? and even if they weren't, could they not classify a neural network as a software implementation? (from the patent enforcing point of view) Because if that is not the case, this idea would be applicable to a lot of things right? Seems like an easy hack
- runnerup 5y ago
  > Because if that is not the case, this idea would be applicable to a lot of things right? Seems like an easy hack Well that does seem to be Microsoft's position at least. (see: Copilot)

vletal 5y ago

Since the h265 relies heavily on operations which are not easily differentiable, such as translation of patches of images, together with a pretty complicated binary format, I'd be pretty amazed if the NN actually learned anything meaningful at all.

svantana 5y ago
Interpolated translation is continuous and easily differentiated. There's lots of work on machine-learned video codecs already, from Nvidia, Qualcomm and others.
- vletal 5y ago
  It's not difficult to propagate gradients while translating an image. Learning "pick a 8x8 patch from (145,17) apply X to it and translate it by (4,-8)" from data is on different level, is not it? The premise was using e2e learning to avoid patent issues. I am sure that with some preprocessing you can plug a NN inside the deciding process and learn very meaningful stuff.
- zinekeller 5y ago
  > There's lots of work on machine-learned video codecs already, from Nvidia, Qualcomm and others. I've tried them, the current state of the art means that they're only useful on relatively static things (some shaking etc) while spike up to AV1-level bitrate to reach perceptual similarity when the movement is too much. Maybe in the future (or whatever under the wraps concoction Nvidia, Qualcomm or another player have), ML-based video codecs will surpass handtuned codecs, but it's not (yet) the present state.

itsamesicario 5y ago

Aren't software implementations royalty free anyway? and even if they weren't, could they not classify a neural network as a software implementation? (from the patent enforcing point of view) Because if that is not the case, this idea would be applicable to a lot of things right? Seems like an easy hack

runnerup 5y ago
> Because if that is not the case, this idea would be applicable to a lot of things right? Seems like an easy hack Well that does seem to be Microsoft's position at least. (see: Copilot)