7 ms·

I agree that it's not perfect, though it does appear to be SoTA. Eventually something like this will just be part of every video codec. You stream a 480p versio

I agree that it's not perfect, though it does appear to be SoTA. Eventually something like this will just be part of every video codec. You stream a 480p version and let the TV create the 4K detail.

ethbr1 3y ago
So, DLAA for video instead of games? https://en.m.wikipedia.org/wiki/Deep_learning_anti-aliasing https://en.m.wikipedia.org/wiki/Deep_learning_anti-aliasing
- jsheard 3y ago
  Not really, DLAA and the current incarnation of DLSS are temporal techniques, meaning all of the detail they add is pulled from past frames. That's an approach which only really makes sense in games where you can jitter the camera to continuously generate samples at different subpixel offsets with each frame. The OP has more in common with the defunct DLSS 1.0, which tried to infer extra detail out of thin air rather than from previous frames, without much success in practice. That was like 5 years ago though so maybe the idea is worth revisiting at some point.
constantcrying 3y ago
Why would you ever do that? If you have the high res data you can actually compress the details which are there and then recreate them. No need to have those be recreated, when you actually have them. Downscaling the images and then upscaling them is pure insanity when the high res images are available.
- turnsout 3y ago
  So streaming services can save money on bandwidth
  - yellow_postit 3y ago
    Or low connectivity scenarios that pushes more local processing. I think it a bit unimaginative to see no use cases for this.
    - constantcrying 3y ago
      There is no use case, because it is a stupid idea. Downscaling then reconstructing is a stupid idea for exactly the same reasons why downscaling for compression is a bad idea. The issue isn't NN reconstruction, but that you are reconstructing the wrong data.
      - adgjlsfhk1 3y ago
        if the nn is part of the codec, you can choose to only downscale the regions that get reconstructed correctly.
        constantcrying 3y ago
        Why would you not let the NN work on the compressed data? That is actually where the information is.
        adgjlsfhk1 3y ago
        that's like asking why you don't train a llm on gzipped text. the compressed data is much harder to reason about
  - prmoustache 3y ago
    Meh. I think upscaling framerate would be more useful.
    - turnsout 3y ago
      TVs already do this… and it's basically a bad thing
  - constantcrying 3y ago
    That's absurd. I think anybody is aware that it is far superior to e.g. compress in the frequency domain than to down sample your image. If you don't believe me just compare a JPEG compressed image with the same image of the same size compressed with down sampling. You will notice a literal night and day difference. Down sampling is a bad way to do compression. It makes no sense to do NN reconstruction on that if you could have compressed that image better and reconstructed from that data.
    - p1esk 3y ago
      Are you saying that when Netflix streams a 480p version of a 4k movie to my TV they do not perform downsampling?
      - constantcrying 3y ago
        Yes. Down sampling makes only sense if you store per pixel data, which is obviously a dumb idea. You get a stream for 480p which contains frames which were compressed from the source files, or the 4k version. At some point there might have been down sampling involved, but you never actually get any of that data, you get the compressed version of those.
        p1esk 3y ago
        Not sure if I’m being dumb, or if it’s you not explaining it clearly: if Neflix produced low resolution frames from high resolution (4k to 480p), and if these 480p frames are what my TV is receiving - are you saying it’s not downsampling, and my TV would not benefit from this new upsampling method?
        constantcrying 3y ago
        Your TV never receives per pixel data. Why would you use a NN to enhance the data which your TV has constructed instead of enhancing the data it actually receives?
- Jabrov 3y ago
  There's lots of videos where there isn't high res data available
  - constantcrying 3y ago
    Totally irrelevant to the discussion, which is explicitly about streaming services delivering in lower resolutions than they have available.
    - ALittleLight 3y ago
      Streaming services already deliver in lower resolution than they available based on network conditions. Good upscaling would let you save on bandwidth and deliver content easier to people in poor network conditions. The tradeoff would be that details in the image wouldn't be exactly the same as the original - but, presumably, nobody would notice this so it would be fine.
      - constantcrying 3y ago
        Why would you not enhance the compressed data with a neutral network? That is where the information actually is.
  - sharpshadow 3y ago
    Satellite data would benefit from that.
- web007 3y ago
  Why would someone ever take a 40Mbps (compressed) video and downsample it so it can be encoded at 400Kbps (compressed) but played back with nearly the same fidelity / with similar artifacts to the same process at 50x data volume? The world will never know. You're also ignoring the part where all lossy codecs throw away those same details and then fake-recreate them with enough fidelity that people are satisfied. Same concept, different mechanism. Look up what 4:2:0 means vs 4:4:4 in a video codec and tell me you still think it's "pure insanity" to rescale. Or, you know, maybe some people have reasons for doing things that aren't the same as the narrow scope of use-cases you considered, and this would work perfectly well for them.
  - constantcrying 3y ago
    >Why would someone ever take a 40Mbps (compressed) video and downsample it so it can be encoded at 400Kbps (compressed) but played back with nearly the same fidelity Because you can just not downscale them and compress them in the frequency domain and encode them in 200Kbps? This is pretty obvious, seriously do you not understand what JPEG does? And why it doesn't do down sampling? Do you seriously believe downscaling outperforms compressing in the frequency domain?
    - adgjlsfhk1 3y ago
      4:2:0 which is used in all common video codecs is down scaling the color data.
      - constantcrying 3y ago
        Scaling color data is a different technique than down sampling. Again, all I am saying is that for a very good reason you do not stream pixel data or compress movies by storing data that was down sampled.
    - web007 3y ago
      Yes, absolutely. Paychovisual encoding can only do so much within the constraints of H.264/265. Throwing away 3/4 (half res) or 15/16 (quarter res) of the data, encoding to X bitrate and then decoding+upscaling looks far better than encoding to the same X bitrate with full resolution. For high bitrate, native resolution will of course look better. For low bitrate, the way H.26? algorithms work end up turning high resolution into a blocky ringing mess to compensate, vs lower resolution where you can see the content, just fuzzily. Go get Tears of Steel raw 4K video (Y4M I think it's called). Scale it down 4x and encode it with ffmpeg HEVC veryslow at CRF 30. Figure out the bitrate, then cheat - use two-pass veryslow HEVC encoding to get the best possible quality native resolution at the same bitrate as your 4x downscaled version. You're aiming for two files that are about the same size. Somehow I couldn't convince the codec to go low enough to match, so I had the low-res version about 60% of the high-res version filesize. Now go and play them both back at 4K with just whatever your native upscale is - bilinear, bicubic, maybe NVIDIA Shield with it's AI Upscaling. Go do that, then tell me you honestly think the blocky, streaky, illegible 4K native looks better than the "soft" quarter-res version.
sharpshadow 3y ago
That’s a good idea it would save a lot of bandwidth and could be used to buffer drops while keeping the quality.
- constantcrying 3y ago
  A better idea would obviously be to enhance the compressed data.
itishappy 3y ago
Your video codec should never create a 480p version at all. Downsampling is incredibly lossy. Instead stream the internal state of your network directly, effectively using the network to decompress your video. Train a new network to generate this state, acting as a compressor. This is the principle of neural compression. This has two major benefits: 1. You cut out the low resolution half your network entirely. (Go check out the architecture diagram of the original post.) 2. Your encoder network now has access to the original HD video, so it can choose to encode the high-frequency details directly instead of generating them afterwards.

I agree that it's not perfect, though it does appear to be SoTA. Eventually something like this will just be part of every video codec. You stream a 480p version and let the TV create the 4K detail.

ethbr1 3y ago

So, DLAA for video instead of games? https://en.m.wikipedia.org/wiki/Deep_learning_anti-aliasing https://en.m.wikipedia.org/wiki/Deep_learning_anti-aliasing

jsheard 3y ago
Not really, DLAA and the current incarnation of DLSS are temporal techniques, meaning all of the detail they add is pulled from past frames. That's an approach which only really makes sense in games where you can jitter the camera to continuously generate samples at different subpixel offsets with each frame. The OP has more in common with the defunct DLSS 1.0, which tried to infer extra detail out of thin air rather than from previous frames, without much success in practice. That was like 5 years ago though so maybe the idea is worth revisiting at some point.

constantcrying 3y ago

Why would you ever do that? If you have the high res data you can actually compress the details which are there and then recreate them. No need to have those be recreated, when you actually have them. Downscaling the images and then upscaling them is pure insanity when the high res images are available.

turnsout 3y ago
So streaming services can save money on bandwidth
- yellow_postit 3y ago
  Or low connectivity scenarios that pushes more local processing. I think it a bit unimaginative to see no use cases for this.
  - constantcrying 3y ago
    There is no use case, because it is a stupid idea. Downscaling then reconstructing is a stupid idea for exactly the same reasons why downscaling for compression is a bad idea. The issue isn't NN reconstruction, but that you are reconstructing the wrong data.
    - adgjlsfhk1 3y ago
      if the nn is part of the codec, you can choose to only downscale the regions that get reconstructed correctly.
      - constantcrying 3y ago
        Why would you not let the NN work on the compressed data? That is actually where the information is.
        adgjlsfhk1 3y ago
        that's like asking why you don't train a llm on gzipped text. the compressed data is much harder to reason about
- prmoustache 3y ago
  Meh. I think upscaling framerate would be more useful.
  - turnsout 3y ago
    TVs already do this… and it's basically a bad thing
- constantcrying 3y ago
  That's absurd. I think anybody is aware that it is far superior to e.g. compress in the frequency domain than to down sample your image. If you don't believe me just compare a JPEG compressed image with the same image of the same size compressed with down sampling. You will notice a literal night and day difference. Down sampling is a bad way to do compression. It makes no sense to do NN reconstruction on that if you could have compressed that image better and reconstructed from that data.
  - p1esk 3y ago
    Are you saying that when Netflix streams a 480p version of a 4k movie to my TV they do not perform downsampling?
    - constantcrying 3y ago
      Yes. Down sampling makes only sense if you store per pixel data, which is obviously a dumb idea. You get a stream for 480p which contains frames which were compressed from the source files, or the 4k version. At some point there might have been down sampling involved, but you never actually get any of that data, you get the compressed version of those.
      - p1esk 3y ago
        Not sure if I’m being dumb, or if it’s you not explaining it clearly: if Neflix produced low resolution frames from high resolution (4k to 480p), and if these 480p frames are what my TV is receiving - are you saying it’s not downsampling, and my TV would not benefit from this new upsampling method?
        constantcrying 3y ago
        Your TV never receives per pixel data. Why would you use a NN to enhance the data which your TV has constructed instead of enhancing the data it actually receives?
Jabrov 3y ago
There's lots of videos where there isn't high res data available
- constantcrying 3y ago
  Totally irrelevant to the discussion, which is explicitly about streaming services delivering in lower resolutions than they have available.
  - ALittleLight 3y ago
    Streaming services already deliver in lower resolution than they available based on network conditions. Good upscaling would let you save on bandwidth and deliver content easier to people in poor network conditions. The tradeoff would be that details in the image wouldn't be exactly the same as the original - but, presumably, nobody would notice this so it would be fine.
    - constantcrying 3y ago
      Why would you not enhance the compressed data with a neutral network? That is where the information actually is.
- sharpshadow 3y ago
  Satellite data would benefit from that.
web007 3y ago
Why would someone ever take a 40Mbps (compressed) video and downsample it so it can be encoded at 400Kbps (compressed) but played back with nearly the same fidelity / with similar artifacts to the same process at 50x data volume? The world will never know. You're also ignoring the part where all lossy codecs throw away those same details and then fake-recreate them with enough fidelity that people are satisfied. Same concept, different mechanism. Look up what 4:2:0 means vs 4:4:4 in a video codec and tell me you still think it's "pure insanity" to rescale. Or, you know, maybe some people have reasons for doing things that aren't the same as the narrow scope of use-cases you considered, and this would work perfectly well for them.
- constantcrying 3y ago
  >Why would someone ever take a 40Mbps (compressed) video and downsample it so it can be encoded at 400Kbps (compressed) but played back with nearly the same fidelity Because you can just not downscale them and compress them in the frequency domain and encode them in 200Kbps? This is pretty obvious, seriously do you not understand what JPEG does? And why it doesn't do down sampling? Do you seriously believe downscaling outperforms compressing in the frequency domain?
  - adgjlsfhk1 3y ago
    4:2:0 which is used in all common video codecs is down scaling the color data.
    - constantcrying 3y ago
      Scaling color data is a different technique than down sampling. Again, all I am saying is that for a very good reason you do not stream pixel data or compress movies by storing data that was down sampled.
  - web007 3y ago
    Yes, absolutely. Paychovisual encoding can only do so much within the constraints of H.264/265. Throwing away 3/4 (half res) or 15/16 (quarter res) of the data, encoding to X bitrate and then decoding+upscaling looks far better than encoding to the same X bitrate with full resolution. For high bitrate, native resolution will of course look better. For low bitrate, the way H.26? algorithms work end up turning high resolution into a blocky ringing mess to compensate, vs lower resolution where you can see the content, just fuzzily. Go get Tears of Steel raw 4K video (Y4M I think it's called). Scale it down 4x and encode it with ffmpeg HEVC veryslow at CRF 30. Figure out the bitrate, then cheat - use two-pass veryslow HEVC encoding to get the best possible quality native resolution at the same bitrate as your 4x downscaled version. You're aiming for two files that are about the same size. Somehow I couldn't convince the codec to go low enough to match, so I had the low-res version about 60% of the high-res version filesize. Now go and play them both back at 4K with just whatever your native upscale is - bilinear, bicubic, maybe NVIDIA Shield with it's AI Upscaling. Go do that, then tell me you honestly think the blocky, streaky, illegible 4K native looks better than the "soft" quarter-res version.

sharpshadow 3y ago

That’s a good idea it would save a lot of bandwidth and could be used to buffer drops while keeping the quality.

constantcrying 3y ago
A better idea would obviously be to enhance the compressed data.

itishappy 3y ago

Your video codec should never create a 480p version at all. Downsampling is incredibly lossy. Instead stream the internal state of your network directly, effectively using the network to decompress your video. Train a new network to generate this state, acting as a compressor. This is the principle of neural compression. This has two major benefits: 1. You cut out the low resolution half your network entirely. (Go check out the architecture diagram of the original post.) 2. Your encoder network now has access to the original HD video, so it can choose to encode the high-frequency details directly instead of generating them afterwards.