11 ms·

I had no idea AV1 or any other codec provided for this. Pretty interesting! What would undermine it is imperfect removal of real noise or grain from the source

I had no idea AV1 or any other codec provided for this. Pretty interesting! What would undermine it is imperfect removal of real noise or grain from the source material: Then you'd have some remnants of the original grain, plus the synthesized grain... an incorrect result. It seems that this scheme requires perfectly clean input images. I hope this idea is taken further, and it becomes the norm among codecs or wrappers to provide room for an author-defined post-processing shader to be applied after decompression.

adgjlsfhk1 4y ago
It really doesn't. The whole reason this technique works is that our brain generally processes effects like noise and grain in terms of macro properties, and even if none of the pixels are anywhere like the right value, it will still look "right" as long as the noise distribution is closish.
- NonNefarious 4y ago
  No it really doesn't what? I don't know what that comment refers to.
  - NonNefarious 4y ago
  - avhon1 4y ago
    It doesn't require perfect removal of the original film grain. A mishmash composed of remnants of the original grain combined with new synthetic grain can still look correct to our brains.
    - NonNefarious 4y ago
      According to this paper, they're going a lot of effort to reproduce the original grain very accurately. If they combine that with some half-assed residual grain from the original, why go to all the effort to faithfully reproduce it? Based on your assertion, it wasn't really necessary to begin with.
klodolph 4y ago
Various audio codecs have the same thing. They will synthesize background noise. The amount of background noise and the rough frequency spectrum are encoded in the stream. If you removed the noise, it would sound a bit weird. For telephone calls in particular, the background noise lets you know that the phone call is connected. If you filter it out, people think that the call was dropped.
- detaro 4y ago
  In the phone case it's even called "comfort noise"
asojfdowgh 4y ago
the synthesized noise only covers what is removed by the denoiser, so it all adds up to what the input is (assuming the left-in noise makes it through the encoder)
- NonNefarious 4y ago
  Do you have a citation for that? Not saying it's wrong, but I didn't see it in the article (which I admittedly skimmed pretty quickly).
  - deleted 4y ago
    [deleted]
  - asojfdowgh 4y ago
    From the flow graph, the film grain estimation is done on the difference between the de-noised video, and the original. So it only picks up the difference that the denoiser creates. > and estimating the film grain parameters from flat regions of the difference between the noisy and de-noised versions of the video sequence.
    - NonNefarious 4y ago
      OK then.
babypuncher 4y ago
I'm not sure I like the idea of a film's photography potentially looking different based on the decoder used. I would much rather dedicate additional bits to preserving the noise that was present in the original film stock or added during post-production. Storage and bandwidth are only getting cheaper.
- cogman10 4y ago
  Funnily, I also don't like these techniques but that's because I'd rather eliminate the noise myself and have a noiseless image when possible. Grain, to me, is simply annoying. Adding it and preserving it has little value.
  - adgjlsfhk1 4y ago
    in that case, this is still probably a good feature for you since you can make an encoder that just doesn't re-synthesize noise and you'll get a de-noised picture.
    - cogman10 4y ago
      Not really, because these encoders are (generally speaking) making pretty large compromises with their noise filters in order to be timely. I want to spend the extra time doing motion compensated denoising. These noise filter will sometimes be temporal, rarely will be motion compensating (because that's computationally expensive) and as a result can't get as good a result as I can.
  - ZeroGravitas 4y ago
    In subjective tests, normal people often report grain (real or reconstructed) as a defect so you're probably in the majority of viewers. Creators and codec nerds value them more than the audience. So these grain removal techniques are in many ways a political compromise between people who just want the smaller video, and the people who want the grain. I'd be fascinated to see how codecs with and without this feature diverge in terms of encoding changes. Presumably there were some big gains just sitting there but the minority that preferred grain stopped then being exploited. Much of the drama around encoder quality has came down to this dichotomy. Choosing to lose the grain/detail was attacked as plastic or fake by the people who care about it, while people who didn't could point to better test scores. Which then caused more arguments about the validity of the measures.
eternityforest 4y ago
I can imagine getting a large amount of flaming for this idea, but wouldn't the logical extreme just be to basically define a container for a JS canvas with embeddable resources? In a more restricted form of course, to be sure that it will always render the same, and all compliant computers will have enough power to render it. You could do things like 480p training videos with ultra-crisp text, and snow/fire/etc effects. You'd probably have to prevent it from having network access or else you'd just be asking for the videos to rot over time, but you could give them an embedding API or even access to a few small bits of data, so you could have a YouTube video that changes with seasons and location. If you were strict about the no network access thing you could give access to all sensors, since the data would never leave the device. YouTube's already got some limited interactivity features, DVDs had them, why not all videos? It would mostly be a novelty, but you could also do really cool stuff like use Bluetooth beacons to skip around to certain parts for an interactive tour, although you might have to enforce downloading the whole thing before playing if you wanted 100% no privacy leaks.
- csnover 4y ago
  It seems like you are describing BD-J[0], or a network-restricted version of Adobe Flash? [0] https://en.wikipedia.org/wiki/BD-J https://en.wikipedia.org/wiki/BD-J
- occamrazor 4y ago
  The biggest problem with that idea is that it wouldn’t be possible to have hw decoders as efficient as the current ones. But there is definitely a trend towards more customizable decoders, with a larger set of primitive operations (DCT, motion estimation/compensation, convolution filters) and parameters. The codec space is slowly moving towards a model where the decoders are VM highly optimized for specific tasks.
- matthberg 4y ago
  That's an interesting idea. I reckon the complexity added with rendering vector graphics (text in particular, font rendering is notoriously difficult) outweighs the bandwidth savings, but still it seems like an area ripe for exploration. Canvas-like APIs might be too complicated to encode/decode efficiently, yet I suspect something closer to a Web/OpenGL fragment shader would be much more manageable (yet likely much worse for interactivity unfortunately). While that wouldn't quite mesh with the idea of vector-like text you proposed (without a heavy library or two thrown in), I suspect the engineering put into the existing graphics pipeline would make it a more feasible approach to augmented video. Looking at the stuff on Shadertoy.com and Inigo Quilez's work shows the capabilities of fragment shader based graphics, yet I suspect all of the magic would be in the details of the file format and encoding strategy. If anyone pokes around or explores a video/shader hybrid format let me know and post it on HN, I bet a bunch of people would be interested.
brontosaurusrex 4y ago
> It seems that this scheme requires perfectly clean input images. It doesn't, the base encode comes out superclean/stable (from my limited tests). The problem is glacial encoding speed which makes this thing not so interesting for home use. Anyway the final result with synthetic grain is wonderful for the bitrate used (I was testing 2 - 4 MBps for HD).

adgjlsfhk1 4y ago

It really doesn't. The whole reason this technique works is that our brain generally processes effects like noise and grain in terms of macro properties, and even if none of the pixels are anywhere like the right value, it will still look "right" as long as the noise distribution is closish.

NonNefarious 4y ago
No it really doesn't what? I don't know what that comment refers to.
- NonNefarious 4y ago
- avhon1 4y ago
  It doesn't require perfect removal of the original film grain. A mishmash composed of remnants of the original grain combined with new synthetic grain can still look correct to our brains.
  - NonNefarious 4y ago
    According to this paper, they're going a lot of effort to reproduce the original grain very accurately. If they combine that with some half-assed residual grain from the original, why go to all the effort to faithfully reproduce it? Based on your assertion, it wasn't really necessary to begin with.

klodolph 4y ago

Various audio codecs have the same thing. They will synthesize background noise. The amount of background noise and the rough frequency spectrum are encoded in the stream. If you removed the noise, it would sound a bit weird. For telephone calls in particular, the background noise lets you know that the phone call is connected. If you filter it out, people think that the call was dropped.

detaro 4y ago
In the phone case it's even called "comfort noise"

asojfdowgh 4y ago

the synthesized noise only covers what is removed by the denoiser, so it all adds up to what the input is (assuming the left-in noise makes it through the encoder)

NonNefarious 4y ago
Do you have a citation for that? Not saying it's wrong, but I didn't see it in the article (which I admittedly skimmed pretty quickly).
- deleted 4y ago
  [deleted]
- asojfdowgh 4y ago
  From the flow graph, the film grain estimation is done on the difference between the de-noised video, and the original. So it only picks up the difference that the denoiser creates. > and estimating the film grain parameters from flat regions of the difference between the noisy and de-noised versions of the video sequence.
  - NonNefarious 4y ago
    OK then.

babypuncher 4y ago

I'm not sure I like the idea of a film's photography potentially looking different based on the decoder used. I would much rather dedicate additional bits to preserving the noise that was present in the original film stock or added during post-production. Storage and bandwidth are only getting cheaper.

cogman10 4y ago
Funnily, I also don't like these techniques but that's because I'd rather eliminate the noise myself and have a noiseless image when possible. Grain, to me, is simply annoying. Adding it and preserving it has little value.
- adgjlsfhk1 4y ago
  in that case, this is still probably a good feature for you since you can make an encoder that just doesn't re-synthesize noise and you'll get a de-noised picture.
  - cogman10 4y ago
    Not really, because these encoders are (generally speaking) making pretty large compromises with their noise filters in order to be timely. I want to spend the extra time doing motion compensated denoising. These noise filter will sometimes be temporal, rarely will be motion compensating (because that's computationally expensive) and as a result can't get as good a result as I can.
- ZeroGravitas 4y ago
  In subjective tests, normal people often report grain (real or reconstructed) as a defect so you're probably in the majority of viewers. Creators and codec nerds value them more than the audience. So these grain removal techniques are in many ways a political compromise between people who just want the smaller video, and the people who want the grain. I'd be fascinated to see how codecs with and without this feature diverge in terms of encoding changes. Presumably there were some big gains just sitting there but the minority that preferred grain stopped then being exploited. Much of the drama around encoder quality has came down to this dichotomy. Choosing to lose the grain/detail was attacked as plastic or fake by the people who care about it, while people who didn't could point to better test scores. Which then caused more arguments about the validity of the measures.

eternityforest 4y ago

I can imagine getting a large amount of flaming for this idea, but wouldn't the logical extreme just be to basically define a container for a JS canvas with embeddable resources? In a more restricted form of course, to be sure that it will always render the same, and all compliant computers will have enough power to render it. You could do things like 480p training videos with ultra-crisp text, and snow/fire/etc effects. You'd probably have to prevent it from having network access or else you'd just be asking for the videos to rot over time, but you could give them an embedding API or even access to a few small bits of data, so you could have a YouTube video that changes with seasons and location. If you were strict about the no network access thing you could give access to all sensors, since the data would never leave the device. YouTube's already got some limited interactivity features, DVDs had them, why not all videos? It would mostly be a novelty, but you could also do really cool stuff like use Bluetooth beacons to skip around to certain parts for an interactive tour, although you might have to enforce downloading the whole thing before playing if you wanted 100% no privacy leaks.

csnover 4y ago
It seems like you are describing BD-J[0], or a network-restricted version of Adobe Flash? [0] https://en.wikipedia.org/wiki/BD-J https://en.wikipedia.org/wiki/BD-J
occamrazor 4y ago
The biggest problem with that idea is that it wouldn’t be possible to have hw decoders as efficient as the current ones. But there is definitely a trend towards more customizable decoders, with a larger set of primitive operations (DCT, motion estimation/compensation, convolution filters) and parameters. The codec space is slowly moving towards a model where the decoders are VM highly optimized for specific tasks.
matthberg 4y ago
That's an interesting idea. I reckon the complexity added with rendering vector graphics (text in particular, font rendering is notoriously difficult) outweighs the bandwidth savings, but still it seems like an area ripe for exploration. Canvas-like APIs might be too complicated to encode/decode efficiently, yet I suspect something closer to a Web/OpenGL fragment shader would be much more manageable (yet likely much worse for interactivity unfortunately). While that wouldn't quite mesh with the idea of vector-like text you proposed (without a heavy library or two thrown in), I suspect the engineering put into the existing graphics pipeline would make it a more feasible approach to augmented video. Looking at the stuff on Shadertoy.com and Inigo Quilez's work shows the capabilities of fragment shader based graphics, yet I suspect all of the magic would be in the details of the file format and encoding strategy. If anyone pokes around or explores a video/shader hybrid format let me know and post it on HN, I bet a bunch of people would be interested.

brontosaurusrex 4y ago

> It seems that this scheme requires perfectly clean input images. It doesn't, the base encode comes out superclean/stable (from my limited tests). The problem is glacial encoding speed which makes this thing not so interesting for home use. Anyway the final result with synthetic grain is wonderful for the bitrate used (I was testing 2 - 4 MBps for HD).