4 ms·
The first demo on the page alone shows that it is a huge failure. It clearly changes the expression of the person. Yes, it is impressive, but it's not what you
by constantcrying 3y ago
The first demo on the page alone shows that it is a huge failure. It clearly changes the expression of the person.
Yes, it is impressive, but it's not what you want to actually "enhance" a movie.
- philipov 3y agoIt doesn't change the expression - the animated gifs are merely out of sync. This appears to happen because they begin animating as soon as they finish loading, which happens at different times for each side of the image.
- mlyle 3y agoReloading can get them in sync. But, it seems to stop playback of the "left" one if you drag the slider completely left, which makes it easy to get desynced again.
- turnsout 3y agoI agree that it's not perfect, though it does appear to be SoTA. Eventually something like this will just be part of every video codec. You stream a 480p version and let the TV create the 4K detail.
- ethbr1 3y agoSo, DLAA for video instead of games? https://en.m.wikipedia.org/wiki/Deep_learning_anti-aliasing https://en.m.wikipedia.org/wiki/Deep_learning_anti-aliasing
- jsheard 3y agoNot really, DLAA and the current incarnation of DLSS are temporal techniques, meaning all of the detail they add is pulled from past frames. That's an approach which only really makes sense in games where you can jitter the camera to continuously generate samples at different subpixel offsets with each frame. The OP has more in common with the defunct DLSS 1.0, which tried to infer extra detail out of thin air rather than from previous frames, without much success in practice. That was like 5 years ago though so maybe the idea is worth revisiting at some point.
- constantcrying 3y agoWhy would you ever do that? If you have the high res data you can actually compress the details which are there and then recreate them. No need to have those be recreated, when you actually have them. Downscaling the images and then upscaling them is pure insanity when the high res images are available.
- turnsout 3y agoSo streaming services can save money on bandwidth
- yellow_postit 3y agoOr low connectivity scenarios that pushes more local processing. I think it a bit unimaginative to see no use cases for this.
- constantcrying 3y agoThere is no use case, because it is a stupid idea. Downscaling then reconstructing is a stupid idea for exactly the same reasons why downscaling for compression is a bad idea. The issue isn't NN reconstruction, but that you are reconstructing the wrong data.
- adgjlsfhk1 3y agoif the nn is part of the codec, you can choose to only downscale the regions that get reconstructed correctly.
- constantcrying 3y agoWhy would you not let the NN work on the compressed data? That is actually where the information is.
- adgjlsfhk1 3y ago
- sharpshadow 3y agoThat’s a good idea it would save a lot of bandwidth and could be used to buffer drops while keeping the quality.
- constantcrying 3y agoA better idea would obviously be to enhance the compressed data.
- itishappy 3y agoYour video codec should never create a 480p version at all. Downsampling is incredibly lossy. Instead stream the internal state of your network directly, effectively using the network to decompress your video. Train a new network to generate this state, acting as a compressor. This is the principle of neural compression. This has two major benefits: 1. You cut out the low resolution half your network entirely. (Go check out the architecture diagram of the original post.) 2. Your encoder network now has access to the original HD video, so it can choose to encode the high-frequency details directly instead of generating them afterwards.