6 ms·

At first I was impressed, but then I realize they also work on (raw) G-buffers, depth maps, normals, etc. grabbed directly from the engine. I mean if we're alre

by dvt 5y ago

At first I was impressed, but then I realize they also work on (raw) G-buffers, depth maps, normals, etc. grabbed directly from the engine. I mean if we're already doing all of that, why not just pass it through 3DSMax and get an actual photorealistic rendering? Ray tracing is already starting to be a thing in consumer products, so we're bound to get photorealism at interactive framerates in the next decade or so. No neural networks needed. There might also be a misconception that GTAV couldn't make their game more murky/realistic-looking, but more often than not, saturation and contrast is purposefully cranked up in games (particularly driving ones). I will say that the lush background mountains look cool though.

dsign 5y ago
I watched their video, and I tend to agree. With the amount of computing they use, ray-tracing is probably going to be just as good and probably higher frame-rate. But maybe this technique, in due time, will result in an alternative class of rendering pipelines with less focus in shading and materials.
ma2rten 5y ago
I think it's worth exploring multiple approaches to a problem. Maybe this neural network approach can be improved upon and it will also generate photorealistic looking images but have other advantages such as less effort on behalf of the game developer or less compute.
arketyp 5y ago
Good point. But I wouldn't rule out neural networks. They could still be used to derive efficient shader models that can be run in real time. https://arxiv.org/abs/1603.06078 https://arxiv.org/abs/1603.06078
HellDunkel 5y ago
You no longer need 3dsmax for raytracing. But what could be really interesting is to train your ml model against path traced images and apply the outputs to selective areas of your screem where the g buffer provives insufficient information density (i. i.e.foliage)
- T-A 5y ago
  https://en.wikipedia.org/wiki/Deep_learning_super_sampling https://en.wikipedia.org/wiki/Deep_learning_super_sampling
dkarras 5y ago
Realistic renders with raytracing assumes accurate models. In the scene, the grass can be a rectangular block with a grass texture applied to it for instance, NN will transform it into an actual grass patch (see example images) similar to how it would look like in real life (not constrained by your 3D geometry / scene setup). So realistic lighting is just one part of the problem space here. A road with an asphalt texture applied on it will need a lot of manual tweaks / material design to look realistic under raytracing regime, NN can look at it and implicitly say "hey, this is a road, and I know what a road looks like under this lighting, so I'm going to render it like this" and it automatically becomes realistic.
- lloeki 5y ago
  Almost sounds like one could grossly placeholder-mark-paint surfaces/areas by type and the NN stuff will paint over it proper content and consistent details by dreaming them from these semantic hints.
  - drxzcl 5y ago
    You can probably have the game output custom-made data to serve as an input for the subsequent NN and skip a whole bunch of lighting calculations.
phromo 5y ago
For an example of photorealistic rendering approach coupled to a scenario generator, see https://synscapes.on.liu.se/index.html https://synscapes.on.liu.se/index.html
anticristi 5y ago
I share your disappointment. There was recently a thread on HN on how good metadata always beats machine learning. I wonder how much complexity could be saved by letting the application properly annotate objects: "This is a tree located at coordinates X,Y,Z in the scene. Please NN take over and project a tree appropriate for the light conditions and geographical location."
virtualritz 5y ago
> [...] Ray tracing is already starting to be a thing in consumer products, so we're bound to get photorealism at interactive framerates in the next decade or so. No neural networks needed. Define 'photorealism'. You limit yourself to the rendering aspect and ignore the content/asset problem and thereby economic factors. This technique is probably one or two orders of magnitude cheaper than generating geometry and shaders that have the required level of detail in any traditional way. Either by an artist or programmatic/procedural (Someone has to write that code too/set up that node graph in Houdini or the like). Yes, you can also just 3D scan stuff (see Quixel, etc.) but that has limits too. More specifically for achieving photorealism by 'traditional means': consider scales. An asset, however produced in finite time, will only hold up to some range of scales. The technique in the paper allows to get as close or far away as you want and everything will still look real. No level of detail work needed, just a few more training data. Traditionally, if you use textures, you will reach the limit of the texture resolution. Even if you use procedural techniques – not everything is a fractal. A close up of a brick looks very different from a wall of bricks etc. Again, who will write that code? My guess is rather that we will see a hybrid of more photorealism by traditional means and more 'icing on the cake' by methods like this.
dirtyid 5y ago
Good ray tracing is expensive and still requires a lot of asset preparation. Difference between what RTX can do realtime right now and production quality final renders is massive. I think there will ultimately be quality wall for realtime tracing that target ms/frame versus mins/frame for HQ renders. After which hacks like this must be used to further close distance by adding layers of additional details that really sell photorealism. TBH I doubt RTX could reproduce even this quality for many years, if ever. It's not just about murkiness and lighting quality, this is akin to adding the last 10% of polish that usually takes 90% of the time. It's also the difference between amateur vs much greater proficiency level. Having access to the assets of scene in 3D max, someone with a basic training can replicate base GTA scene, the enhancements shown here takes years more experience. The amount accessibility and saved labour is tremendous. It's why even VHS video is still more photo realistic than 4K RTX. Real life has a lot of minutiae and interacting details beyond resolution and good lighting models.
wffurr 5y ago
The Cityscapes dataset is a bit murky looking; try scrolling down to the Mapillary Vista dataset instead. Much better colors. They should have led with that one instead.
- mrec 5y ago
  Not disagreeing, but the Cityscapes samples looked 100% photorealistic to me, whereas the Mapillary samples just looked different. Disclaimer: in the UK, so grey and murky == realistic.
  - anthk 5y ago
    >Disclaimer: in the UK, so grey and murky == realistic. Ditto in Northern Spain. The sunny dataset is more akin to Mediterranean landscapes. To me the shinny/sunny/reflective stuff it looked unreal.
- notacoward 5y ago
  The MV examples generally look better, but seem to have one characteristic weakness: street signs. Many of them seem unrealistically saturated and glow-y with blue fringes around letters/symbols. Kind of odd, considering that everything else looks so good.
calaphos 5y ago
I don't think the goal of this is to make games look better. Especially the GTAV dataset is often used as a synthetic dataset for research. The appeal is that you can extract normal/material and other semantic information from the game engine for a driving scene. Basically avoid expensive segmentation labels on real data by working the other way around. Things like this synthetic to real translation can then be used for other downstream tasks (semantic segmentation, distance/normal prediction).

by dvt 5y ago

dsign 5y ago

I watched their video, and I tend to agree. With the amount of computing they use, ray-tracing is probably going to be just as good and probably higher frame-rate. But maybe this technique, in due time, will result in an alternative class of rendering pipelines with less focus in shading and materials.

ma2rten 5y ago

I think it's worth exploring multiple approaches to a problem. Maybe this neural network approach can be improved upon and it will also generate photorealistic looking images but have other advantages such as less effort on behalf of the game developer or less compute.

arketyp 5y ago

Good point. But I wouldn't rule out neural networks. They could still be used to derive efficient shader models that can be run in real time. https://arxiv.org/abs/1603.06078 https://arxiv.org/abs/1603.06078

HellDunkel 5y ago

You no longer need 3dsmax for raytracing. But what could be really interesting is to train your ml model against path traced images and apply the outputs to selective areas of your screem where the g buffer provives insufficient information density (i. i.e.foliage)

T-A 5y ago
https://en.wikipedia.org/wiki/Deep_learning_super_sampling https://en.wikipedia.org/wiki/Deep_learning_super_sampling

dkarras 5y ago

Realistic renders with raytracing assumes accurate models. In the scene, the grass can be a rectangular block with a grass texture applied to it for instance, NN will transform it into an actual grass patch (see example images) similar to how it would look like in real life (not constrained by your 3D geometry / scene setup). So realistic lighting is just one part of the problem space here. A road with an asphalt texture applied on it will need a lot of manual tweaks / material design to look realistic under raytracing regime, NN can look at it and implicitly say "hey, this is a road, and I know what a road looks like under this lighting, so I'm going to render it like this" and it automatically becomes realistic.

lloeki 5y ago
Almost sounds like one could grossly placeholder-mark-paint surfaces/areas by type and the NN stuff will paint over it proper content and consistent details by dreaming them from these semantic hints.
- drxzcl 5y ago
  You can probably have the game output custom-made data to serve as an input for the subsequent NN and skip a whole bunch of lighting calculations.

phromo 5y ago

For an example of photorealistic rendering approach coupled to a scenario generator, see https://synscapes.on.liu.se/index.html https://synscapes.on.liu.se/index.html

anticristi 5y ago

I share your disappointment. There was recently a thread on HN on how good metadata always beats machine learning. I wonder how much complexity could be saved by letting the application properly annotate objects: "This is a tree located at coordinates X,Y,Z in the scene. Please NN take over and project a tree appropriate for the light conditions and geographical location."

virtualritz 5y ago

> [...] Ray tracing is already starting to be a thing in consumer products, so we're bound to get photorealism at interactive framerates in the next decade or so. No neural networks needed. Define 'photorealism'. You limit yourself to the rendering aspect and ignore the content/asset problem and thereby economic factors. This technique is probably one or two orders of magnitude cheaper than generating geometry and shaders that have the required level of detail in any traditional way. Either by an artist or programmatic/procedural (Someone has to write that code too/set up that node graph in Houdini or the like). Yes, you can also just 3D scan stuff (see Quixel, etc.) but that has limits too. More specifically for achieving photorealism by 'traditional means': consider scales. An asset, however produced in finite time, will only hold up to some range of scales. The technique in the paper allows to get as close or far away as you want and everything will still look real. No level of detail work needed, just a few more training data. Traditionally, if you use textures, you will reach the limit of the texture resolution. Even if you use procedural techniques – not everything is a fractal. A close up of a brick looks very different from a wall of bricks etc. Again, who will write that code? My guess is rather that we will see a hybrid of more photorealism by traditional means and more 'icing on the cake' by methods like this.

dirtyid 5y ago

Good ray tracing is expensive and still requires a lot of asset preparation. Difference between what RTX can do realtime right now and production quality final renders is massive. I think there will ultimately be quality wall for realtime tracing that target ms/frame versus mins/frame for HQ renders. After which hacks like this must be used to further close distance by adding layers of additional details that really sell photorealism. TBH I doubt RTX could reproduce even this quality for many years, if ever. It's not just about murkiness and lighting quality, this is akin to adding the last 10% of polish that usually takes 90% of the time. It's also the difference between amateur vs much greater proficiency level. Having access to the assets of scene in 3D max, someone with a basic training can replicate base GTA scene, the enhancements shown here takes years more experience. The amount accessibility and saved labour is tremendous. It's why even VHS video is still more photo realistic than 4K RTX. Real life has a lot of minutiae and interacting details beyond resolution and good lighting models.

wffurr 5y ago

The Cityscapes dataset is a bit murky looking; try scrolling down to the Mapillary Vista dataset instead. Much better colors. They should have led with that one instead.

mrec 5y ago
Not disagreeing, but the Cityscapes samples looked 100% photorealistic to me, whereas the Mapillary samples just looked different. Disclaimer: in the UK, so grey and murky == realistic.
- anthk 5y ago
  >Disclaimer: in the UK, so grey and murky == realistic. Ditto in Northern Spain. The sunny dataset is more akin to Mediterranean landscapes. To me the shinny/sunny/reflective stuff it looked unreal.
notacoward 5y ago
The MV examples generally look better, but seem to have one characteristic weakness: street signs. Many of them seem unrealistically saturated and glow-y with blue fringes around letters/symbols. Kind of odd, considering that everything else looks so good.

calaphos 5y ago

I don't think the goal of this is to make games look better. Especially the GTAV dataset is often used as a synthetic dataset for research. The appeal is that you can extract normal/material and other semantic information from the game engine for a driving scene. Basically avoid expensive segmentation labels on real data by working the other way around. Things like this synthetic to real translation can then be used for other downstream tasks (semantic segmentation, distance/normal prediction).