4 ms·

Why not let it output the gbuffer images instead, or perhaps even let it operate on the artwork instead? That way you could bake the changes into the game witho

Why not let it output the gbuffer images instead, or perhaps even let it operate on the artwork instead? That way you could bake the changes into the game without having to run the NN in real time.

derefr 5y ago
From what I can tell, the whole reason the road "looks like a video game" is that it's using a source texture and then stretching it along the road's geometry, "diluting" the level of detail as seen from the camera viewpoint. The image enhancement here is happening "in camera", like a fragment shader; but it's using a lot more context than the sort of local/cellular kernels that fragment shaders usually are. Look at e.g. the gutters on the roads, where there are leaves/dirt — they get filled in with tons of extra perceptual texture (texture that exists in the projection plane, rather than texture that exists "on" the road), bringing them up to the same LoD as the rest of the road. If that texture was "pushed back" to the origin texture, said texture would have to exist at some sort of ridiculous resolution that modern graphics cards wouldn't have a hope of rendering.
- gmueckl 5y ago
  The part about ridiculous resolutions for texture detail like leaves isn't necessarily true. There are very old techniques for scattering small decals densely across surfaces. The basic trick is to use coarse textures that encode the positioning of decals within each texel footprint.
  - derefr 5y ago
    Sure, but no matter what the geometry/bump maps/etc. are on the road, they're not going to render out as anything but an aliased, badly-lit mess unless you're heavily supersampling them (and then doing ray-tracing to the supersampled image.) We're talking about "objects" where the crucial detail-features of their meshes for getting correct color math out (e.g. the normal+reflectance maps) would be, on average, scaled to be smaller than a pixel from the viewing distance+angle, with potentially many objects contributing to the final color of each pixel. So, rather than rendering the road at 8x-16x and then downsampling it, in order to get a pile of leaves to appear to have ~1.5x LoD at the projection plane, you can just achieve that 1.5x LoD pixel by pixel at the projection plane. Or, to put that another way: when a digital artist is painting a landscape, they don't have to zoom in by 8x, paint every individual blade of grass, and then zoom out. They can just use an artistic technique at 1x zoom that approximates the texture that would have been created by downsampling "photographic" detail painted in at 8x to your screen resolution. Here, the ML model is playing the role of the digital artist, "painting" on the projection plane.
  - enriquto 5y ago
    The problem is not texture antialiasing (which is essentially solved). The problem is shape antialiasing, whether many distant small shapes are so close together that become a sort of translucent buzz, which is very computationally heavy to compute.
    - gmueckl 5y ago
      REYES renderers had a pretty nifty solution for shape antialiasing with their slicing and dicing of geometry into sub-pixel sized primitives. That was in the 80s, just for reference. Unfortunately, this method has a pretty complex implementation and just doesn't play all that well with hardware rasterization or path tracing and has since fallen out of favor.
- Gauge_Irrahphe 5y ago
  I don't think so. I think it's because the texture is already photo-like, and then there is a normal map applied on top of that. It basically creates two layers of shading that don't fit together. Few games manage to avoid that problem.
  - derefr 5y ago
    Interesting hypothesis. I think this would predict that you’d get more realistic results if you were using “bare, unlit” textures (e.g. texture images + normal maps created by using 3D scanning or even computational tomography, as you would if you wanted to 3D-print a copy of the object) and then applying regular normal maps to them. Anyone have screenshots of 3D game objects that were explicitly mentioned as being authored this way?
dahart 5y ago
This is a good question - it certainly is possible to run a NN on the artwork or pre-g-buffer, and you could imagine having something bake the changes. To do that, you’d need access to the game’s source code and assets, and you would need to rewrite portions of the rendering engine. Training might be incredibly slow if it needs to iterate on builds of the game, or that problem could be worked around but might need considerable engineering to support the training iterations (e.g. data representation and a pipeline with the ability to render changes to the artwork and renderer on the fly without needing a game rebuild). So I’d speculate the answer to why not is that it’s just not what the authors set out to do, that they don’t have access to the game source nor the time or resources needed to refactor the renderer. Part of the magic of neural networks is their black box properties - they can do what they do without needing to understand 3d geometry or integrate with a render engine. Throw an image in, and a new image pops out, without the neural network needing to understand what it’s doing or why. That said, I would guess that coming down the road is examples of exactly what you’re suggesting - games that will use neural networks to drive realism in the assets and shaders and renderer. To some degree there are already tools starting to do this.

Why not let it output the gbuffer images instead, or perhaps even let it operate on the artwork instead? That way you could bake the changes into the game without having to run the NN in real time.

derefr 5y ago

From what I can tell, the whole reason the road "looks like a video game" is that it's using a source texture and then stretching it along the road's geometry, "diluting" the level of detail as seen from the camera viewpoint. The image enhancement here is happening "in camera", like a fragment shader; but it's using a lot more context than the sort of local/cellular kernels that fragment shaders usually are. Look at e.g. the gutters on the roads, where there are leaves/dirt — they get filled in with tons of extra perceptual texture (texture that exists in the projection plane, rather than texture that exists "on" the road), bringing them up to the same LoD as the rest of the road. If that texture was "pushed back" to the origin texture, said texture would have to exist at some sort of ridiculous resolution that modern graphics cards wouldn't have a hope of rendering.

gmueckl 5y ago
The part about ridiculous resolutions for texture detail like leaves isn't necessarily true. There are very old techniques for scattering small decals densely across surfaces. The basic trick is to use coarse textures that encode the positioning of decals within each texel footprint.
- derefr 5y ago
  Sure, but no matter what the geometry/bump maps/etc. are on the road, they're not going to render out as anything but an aliased, badly-lit mess unless you're heavily supersampling them (and then doing ray-tracing to the supersampled image.) We're talking about "objects" where the crucial detail-features of their meshes for getting correct color math out (e.g. the normal+reflectance maps) would be, on average, scaled to be smaller than a pixel from the viewing distance+angle, with potentially many objects contributing to the final color of each pixel. So, rather than rendering the road at 8x-16x and then downsampling it, in order to get a pile of leaves to appear to have ~1.5x LoD at the projection plane, you can just achieve that 1.5x LoD pixel by pixel at the projection plane. Or, to put that another way: when a digital artist is painting a landscape, they don't have to zoom in by 8x, paint every individual blade of grass, and then zoom out. They can just use an artistic technique at 1x zoom that approximates the texture that would have been created by downsampling "photographic" detail painted in at 8x to your screen resolution. Here, the ML model is playing the role of the digital artist, "painting" on the projection plane.
- enriquto 5y ago
  The problem is not texture antialiasing (which is essentially solved). The problem is shape antialiasing, whether many distant small shapes are so close together that become a sort of translucent buzz, which is very computationally heavy to compute.
  - gmueckl 5y ago
    REYES renderers had a pretty nifty solution for shape antialiasing with their slicing and dicing of geometry into sub-pixel sized primitives. That was in the 80s, just for reference. Unfortunately, this method has a pretty complex implementation and just doesn't play all that well with hardware rasterization or path tracing and has since fallen out of favor.
Gauge_Irrahphe 5y ago
I don't think so. I think it's because the texture is already photo-like, and then there is a normal map applied on top of that. It basically creates two layers of shading that don't fit together. Few games manage to avoid that problem.
- derefr 5y ago
  Interesting hypothesis. I think this would predict that you’d get more realistic results if you were using “bare, unlit” textures (e.g. texture images + normal maps created by using 3D scanning or even computational tomography, as you would if you wanted to 3D-print a copy of the object) and then applying regular normal maps to them. Anyone have screenshots of 3D game objects that were explicitly mentioned as being authored this way?

dahart 5y ago

This is a good question - it certainly is possible to run a NN on the artwork or pre-g-buffer, and you could imagine having something bake the changes. To do that, you’d need access to the game’s source code and assets, and you would need to rewrite portions of the rendering engine. Training might be incredibly slow if it needs to iterate on builds of the game, or that problem could be worked around but might need considerable engineering to support the training iterations (e.g. data representation and a pipeline with the ability to render changes to the artwork and renderer on the fly without needing a game rebuild). So I’d speculate the answer to why not is that it’s just not what the authors set out to do, that they don’t have access to the game source nor the time or resources needed to refactor the renderer. Part of the magic of neural networks is their black box properties - they can do what they do without needing to understand 3d geometry or integrate with a render engine. Throw an image in, and a new image pops out, without the neural network needing to understand what it’s doing or why. That said, I would guess that coming down the road is examples of exactly what you’re suggesting - games that will use neural networks to drive realism in the assets and shaders and renderer. To some degree there are already tools starting to do this.