4 ms·

Amazing how this is all done in real-time, and incredible stability of image. Further, I’d love to also see a tech demo using a Californian dataset as the inpu

Amazing how this is all done in real-time, and incredible stability of image. Further, I’d love to also see a tech demo using a Californian dataset as the input: To my originally European eyes, it’s simply making GTA look more like video shot in Germany vs more realistic (there’s a difference!), though I get that any data set can be fed into this :)

mosselman 5y ago
Is it real-time rather than post processed?
- nikodunk 5y ago
  Great point I don’t know
- thih9 5y ago
  From the "Enhancing Photorealism Enhancement" video [1]: "It is a convolutional network which produces images frame by frame and can be run at interactive rates.". [1]: https://www.youtube.com/watch?v=P1IcaBn3ej0 https://www.youtube.com/watch?v=P1IcaBn3ej0 (0:19)
  - dandellion 5y ago
    From the paper: "Inference with our approach in its current unoptimized implementation takes half a second on a Geforce RTX 3090 GPU." That means 1 or 2 frames per second, so interactive but pretty much unplayable, which is to be expected as it's just research, but considering that they have DLSS working in realtime with tensor cores perhaps something like this will also be achievable very soon. Edit: Out of curiosity I looked up how long DLSS takes to process and it's less than 2ms, so you'd probably need to speed this up by two orders of magnitude to run it a game.
L_ 5y ago
There's also a version trained on Mapillary Vistas, which has images from all around the world. Results from that are at the bottom of the page and at the end of the video ( https://youtu.be/P1IcaBn3ej0?t=462 https://youtu.be/P1IcaBn3ej0?t=462 )
- janekm 5y ago
  To me that version is much more reflective of the actual potential of this method than "GTA Düsseldorf dash cam". When watching the images produced with that one I get the sense that the sense of realism is enhanced by the dashcam-style image degradations and could be largely emulated in-engine (green LUT, blur + glow filters to emulate lens issues).

mosselman 5y ago

Is it real-time rather than post processed?

nikodunk 5y ago
Great point I don’t know
thih9 5y ago
From the "Enhancing Photorealism Enhancement" video [1]: "It is a convolutional network which produces images frame by frame and can be run at interactive rates.". [1]: https://www.youtube.com/watch?v=P1IcaBn3ej0 https://www.youtube.com/watch?v=P1IcaBn3ej0 (0:19)
- dandellion 5y ago
  From the paper: "Inference with our approach in its current unoptimized implementation takes half a second on a Geforce RTX 3090 GPU." That means 1 or 2 frames per second, so interactive but pretty much unplayable, which is to be expected as it's just research, but considering that they have DLSS working in realtime with tensor cores perhaps something like this will also be achievable very soon. Edit: Out of curiosity I looked up how long DLSS takes to process and it's less than 2ms, so you'd probably need to speed this up by two orders of magnitude to run it a game.

L_ 5y ago

There's also a version trained on Mapillary Vistas, which has images from all around the world. Results from that are at the bottom of the page and at the end of the video ( https://youtu.be/P1IcaBn3ej0?t=462 https://youtu.be/P1IcaBn3ej0?t=462 )

janekm 5y ago
To me that version is much more reflective of the actual potential of this method than "GTA Düsseldorf dash cam". When watching the images produced with that one I get the sense that the sense of realism is enhanced by the dashcam-style image degradations and could be largely emulated in-engine (green LUT, blur + glow filters to emulate lens issues).