11 ms·

Enhancing Photorealism Enhancement

522 points by lnyan 5y ago

serf 5y ago
is the road-smoothing an artifact from the fact that the camera input sources lack the ability to resolve a detailed picture of the road at-speed?
- 317070 5y ago
  Probably yes. Alternatively, it shows that the road textures used in GTA V were not very apt for their job. I do think the smoother ones look considerably more realistic.
Gauge_Irrahphe 5y ago
Why not let it output the gbuffer images instead, or perhaps even let it operate on the artwork instead? That way you could bake the changes into the game without having to run the NN in real time.
- derefr 5y ago
  From what I can tell, the whole reason the road "looks like a video game" is that it's using a source texture and then stretching it along the road's geometry, "diluting" the level of detail as seen from the camera viewpoint. The image enhancement here is happening "in camera", like a fragment shader; but it's using a lot more context than the sort of local/cellular kernels that fragment shaders usually are. Look at e.g. the gutters on the roads, where there are leaves/dirt — they get filled in with tons of extra perceptual texture (texture that exists in the projection plane, rather than texture that exists "on" the road), bringing them up to the same LoD as the rest of the road. If that texture was "pushed back" to the origin texture, said texture would have to exist at some sort of ridiculous resolution that modern graphics cards wouldn't have a hope of rendering.
  - gmueckl 5y ago
    The part about ridiculous resolutions for texture detail like leaves isn't necessarily true. There are very old techniques for scattering small decals densely across surfaces. The basic trick is to use coarse textures that encode the positioning of decals within each texel footprint.
    - derefr 5y ago
      Sure, but no matter what the geometry/bump maps/etc. are on the road, they're not going to render out as anything but an aliased, badly-lit mess unless you're heavily supersampling them (and then doing ray-tracing to the supersampled image.) We're talking about "objects" where the crucial detail-features of their meshes for getting correct color math out (e.g. the normal+reflectance maps) would be, on average, scaled to be smaller than a pixel from the viewing distance+angle, with potentially many objects contributing to the final color of each pixel. So, rather than rendering the road at 8x-16x and then downsampling it, in order to get a pile of leaves to appear to have ~1.5x LoD at the projection plane, you can just achieve that 1.5x LoD pixel by pixel at the projection plane. Or, to put that another way: when a digital artist is painting a landscape, they don't have to zoom in by 8x, paint every individual blade of grass, and then zoom out. They can just use an artistic technique at 1x zoom that approximates the texture that would have been created by downsampling "photographic" detail painted in at 8x to your screen resolution. Here, the ML model is playing the role of the digital artist, "painting" on the projection plane.
    - enriquto 5y ago
      The problem is not texture antialiasing (which is essentially solved). The problem is shape antialiasing, whether many distant small shapes are so close together that become a sort of translucent buzz, which is very computationally heavy to compute.
      - gmueckl 5y ago
        REYES renderers had a pretty nifty solution for shape antialiasing with their slicing and dicing of geometry into sub-pixel sized primitives. That was in the 80s, just for reference. Unfortunately, this method has a pretty complex implementation and just doesn't play all that well with hardware rasterization or path tracing and has since fallen out of favor.
- dahart 5y ago
  This is a good question - it certainly is possible to run a NN on the artwork or pre-g-buffer, and you could imagine having something bake the changes. To do that, you’d need access to the game’s source code and assets, and you would need to rewrite portions of the rendering engine. Training might be incredibly slow if it needs to iterate on builds of the game, or that problem could be worked around but might need considerable engineering to support the training iterations (e.g. data representation and a pipeline with the ability to render changes to the artwork and renderer on the fly without needing a game rebuild). So I’d speculate the answer to why not is that it’s just not what the authors set out to do, that they don’t have access to the game source nor the time or resources needed to refactor the renderer. Part of the magic of neural networks is their black box properties - they can do what they do without needing to understand 3d geometry or integrate with a render engine. Throw an image in, and a new image pops out, without the neural network needing to understand what it’s doing or why. That said, I would guess that coming down the road is examples of exactly what you’re suggesting - games that will use neural networks to drive realism in the assets and shaders and renderer. To some degree there are already tools starting to do this.
anigbrowl 5y ago
I mean it just looks like they're simulating the windscreen glass. I guess it's more realistic but I somewhat prefer the original.
- StavrosK 5y ago
  Is the windshield glass really the only change you see, when grass becomes green, reflections are added, the entire photo has a higher color temperature, etc?
  - derefr 5y ago
    Looking at the last comparison on the page, the grass isn't just becoming green, it's being "puffed" out from its original geometry, as if the grass geometry is being vertex-shaded with small spiky-ball-like objects. (A bit like the look of "static grass" in model making.)
  - caslon 5y ago
    Most of it is making light more intense, and most of everything else could be emulated with a blur filter and a green tint. It's impressive for what it represents technically, but it's visually subtle.
  - anigbrowl 5y ago
    I went through and looked at each of the examples and gave my impression. That's why I say 'looks like...' as opposed to doing a critique of their method.
- rzzzt 5y ago
  I think the Cityscapes dataset pictures were taken from the car's hood; the Mercedes emblem is up close. The camera position in game also seems to match this.
nikodunk 5y ago
Amazing how this is all done in real-time, and incredible stability of image. Further, I’d love to also see a tech demo using a Californian dataset as the input: To my originally European eyes, it’s simply making GTA look more like video shot in Germany vs more realistic (there’s a difference!), though I get that any data set can be fed into this :)
- mosselman 5y ago
  Is it real-time rather than post processed?
  - nikodunk 5y ago
    Great point I don’t know
  - thih9 5y ago
    From the "Enhancing Photorealism Enhancement" video [1]: "It is a convolutional network which produces images frame by frame and can be run at interactive rates.". [1]: https://www.youtube.com/watch?v=P1IcaBn3ej0 https://www.youtube.com/watch?v=P1IcaBn3ej0 (0:19)
    - dandellion 5y ago
      From the paper: "Inference with our approach in its current unoptimized implementation takes half a second on a Geforce RTX 3090 GPU." That means 1 or 2 frames per second, so interactive but pretty much unplayable, which is to be expected as it's just research, but considering that they have DLSS working in realtime with tensor cores perhaps something like this will also be achievable very soon. Edit: Out of curiosity I looked up how long DLSS takes to process and it's less than 2ms, so you'd probably need to speed this up by two orders of magnitude to run it a game.
- L_ 5y ago
  There's also a version trained on Mapillary Vistas, which has images from all around the world. Results from that are at the bottom of the page and at the end of the video ( https://youtu.be/P1IcaBn3ej0?t=462 https://youtu.be/P1IcaBn3ej0?t=462 )
wefhi 5y ago
"return of GTA IV"
Toutouxc 5y ago
So, can we gradually get to a point where the engine itself only renders simplified abstractions (i.e. not the current state-of-the-art graphics) and feeds them to a neural network that produces the output?
- arketyp 5y ago
  This is what has been called Deep Shading: https://arxiv.org/abs/1603.06078 https://arxiv.org/abs/1603.06078
- deleted 5y ago
  [deleted]
dvt 5y ago
At first I was impressed, but then I realize they also work on (raw) G-buffers, depth maps, normals, etc. grabbed directly from the engine. I mean if we're already doing all of that, why not just pass it through 3DSMax and get an actual photorealistic rendering? Ray tracing is already starting to be a thing in consumer products, so we're bound to get photorealism at interactive framerates in the next decade or so. No neural networks needed. There might also be a misconception that GTAV couldn't make their game more murky/realistic-looking, but more often than not, saturation and contrast is purposefully cranked up in games (particularly driving ones). I will say that the lush background mountains look cool though.
- dsign 5y ago
  I watched their video, and I tend to agree. With the amount of computing they use, ray-tracing is probably going to be just as good and probably higher frame-rate. But maybe this technique, in due time, will result in an alternative class of rendering pipelines with less focus in shading and materials.
- ma2rten 5y ago
  I think it's worth exploring multiple approaches to a problem. Maybe this neural network approach can be improved upon and it will also generate photorealistic looking images but have other advantages such as less effort on behalf of the game developer or less compute.
- arketyp 5y ago
  Good point. But I wouldn't rule out neural networks. They could still be used to derive efficient shader models that can be run in real time. https://arxiv.org/abs/1603.06078 https://arxiv.org/abs/1603.06078
- HellDunkel 5y ago
  You no longer need 3dsmax for raytracing. But what could be really interesting is to train your ml model against path traced images and apply the outputs to selective areas of your screem where the g buffer provives insufficient information density (i. i.e.foliage)
larschdk 5y ago
Impressive, but I think that photorealism is not necessarily what you would want to strive for in games, as it is not true realism. It emulates the deficiencies of the camera that we perceive it as realistic, because we are trained on thousands of photo images and video footage. I prefer visionrealism instead of photorealism in my games and don't really find that lens effects like flares, chromatic aberration, and depth of field enhances the experience of realism. It feels like I'm looking through a camera rather than being there myself. For me it breaks immersion instead of enhancing it.
- avereveard 5y ago
  yeah, games visuals are more about art direction than search of realism this would work great in a flight sim tho, albeit blur and out of focus effects are terrible for visual clarity however, there's one small bit that has not been given enough attention imho, which is on the latter set, under "removes distant haze": this would be a great technique to complement flat LOD reduction of open world games while keeping up the crispness and contrast of distant items.
  - simooooo 5y ago
    Would be interesting to see what it can do for LODs similar to how you see imposters used today
- ginko 5y ago
  That's mostly a problem with the input data. A gamestudio of the size of rockstar could afford to acquire their own custom fit dataset. Get some good cameras and lenses and capture footage of wherever they want to set their next game. Then they can apply some motion picture color grading to that and use that as the input data for the NN.
- dwd 5y ago
  I found their approach to be more like the difference between a digitally filmed movie (that can look too crisp) and one shot on cellulose that has a kind of softness to it. Where it noticeably fell short was displaying red lights (tail lights, traffic lights, etc) that lost their brightness.
ohlookabird 5y ago
Looks really nice. It is however funny to see what it does to road signs (black arrows suddenly turn blue or the orange "don't walk" hand of pedestrian traffic lights seems to disappear). I wonder if this has to do that they use image data from Germany (having slightly different signs than the US).
- zuzun 5y ago
  It's also giving the license plates the blue stripe.
oceanofsolaris 5y ago
I am especially impressed by how much more realistic the lighting ends up looking despite the network probably having less information to work with than the engine! Though it ends up turning the lighting very flat. I guess that is an effect of the source images they were using. As others have said, using this directly in the engine (tuned to work with the intended art style etc) could probably produce almost miraculous results, if it can be made to work at a reasonable frame rate. That would also allow the developers to use high-quality rendered images instead of these green-tinted, low-contrast "automotive grade" camera images as source of truth (there are good reasons these images look like that ... but they don't look pleasant).
- deleted 5y ago
  [deleted]
orbital-decay 5y ago
They mention temporal stability across the footage, but what about the entire game? It would be unfortunate to have the same car to turn from Audi to BMW under entirely different lighting and angles, or a palm tree into a pine. (the difference might be much more subtle but annoying nonetheless)
- dahart 5y ago
  Part of the contribution of this paper is about preventing that specific problem. “We also seek to eliminate artifacts that can be seen in the results of prior deep-learning approaches, which often hallucinate objects. To this end, we analyze the datasets that are commonly used for photorealism enhancement. Our analysis reveals that their scene layouts differ in ways that can explain artifacts commonly seen in prior work. To better align the datasets and alleviate the artifacts, we propose a new strategy for sampling image patches during training. We further design a new adversarial training objective that facilitates enhancements that are geometrically and semantically consistent with the content of the input image.”
xpuente 5y ago
Why is the demo at 720p? Does not look good to do something like that.
- rasz 5y ago
  There is a long an rich tradition to release your DNN/FX work in a post stamp sized presentation.
bawolff 5y ago
The results dont look that great to me. The original looks better for a game. The "enhanced" version is so unsaturated it could be bordering on black & white.
- eutectic 5y ago
  The model trained on TSIT gives nicer results.
- svantana 5y ago
  That's because of the dataset. Later in the video they show results using another dataset with more vibrant colour, that looks a lot better IMO.
  - bawolff 5y ago
    I just watched the video to the end - definitely an improvement, but it still looks "faker" than the original to me. Maybe some uncanny valley effect is going on with me.
    - jiofih 5y ago
      You’re just used to the style. Putting a photo of a road next to it will show you how far from reality GTA rendering is.
    - 317070 5y ago
      Out of curiosity, did you play the game? To me, it is hard to imagine someone would find the game to look less fake.
      - bawolff 5y ago
        I did not. I'm not saying the game looks real, just that it looks less "wrong" somehow, to me.
- ObserverNeutral 5y ago
  germany is unsaturated
playpause 5y ago
Uninformed take: this looks to me like an example of ‘style transfer’. If so, it doesn’t seem as compelling as other examples I’ve seen before, eg making a photo look like a Van Gogh using a single painting as input, which has been around for a few years. Can someone with better knowledge in this area explain if I’m missing something?
- L_ 5y ago
  Style transfer commonly hallucinates objects or adds artifacts. Making something look like a painting is actually much easier than making it look realistic. In paintings, artifacts are often tolerated as 'artistic'. In a photo, it just pops out. Also, style transfer approaches are not temporally stable most of the time. Have a look at the comparisons with state-of-the-art image-to-image translation (CUT, TSIT) and photo style transfer (WCT2). In the case of GTA/Cityscapes shown here, most of these methods put trees in the sky and/or flicker. Also the use of G-buffers allows a much deeper and more robust translation than just using images as style-transfer approaches.
  - playpause 5y ago
    Thank you, this is really interesting.
- deleted 5y ago
  [deleted]
gonzo41 5y ago
This is pretty impressive, I wonder if you could build really cheaply textured maps and then upgrade them with this method as a post production touch up. Essentially removing most of the human work from texture libraries.
plutonorm 5y ago
The number of comments here that are negative is just staggering to me. These images are within a hair's breadth of being indistinguishable from reality. It's an enormous leap. An enormous leap from the previous techniques and an enormous leap towards photorealistic realtime graphics. You don't want your games to look realistic? Go play pacman or outrun. You want to use path tracing instead? You have no idea how much cheaper this technique is. And on top of that the neural network is also fixing up the unrealistic texture work on all kinds of things in the scene. The first games to use this are going to be from small game studios with less investment in the status quo and they are going to blow the competition out of the water.
- HellDunkel 5y ago
  Maching learning is often oversold as a shortcut solution for some very hard problems and this is a perfect example. What you predict about small studios seems unlikely as ML is a beast to tame, needs highly specialized engineers and huge high qualty datasets for training.
  - plutonorm 5y ago
    So were ajax crud web apps, 25 years ago.
    - zokier 5y ago
      And they still suck, 25 years later
- bayindirh 5y ago
  Actually, I'm pretty amazed by the results. This is indeed a giant leap. Also it's coming from Intel instead of AMD or Nvidia, which makes me hopeful about its wide availability. The only thing I didn't agree is that Rockstar is incapable of improving their engine further. Since GTA-III, this is somewhat a deliberate stylistic choice up to a certain point, like how Source engine is also opinionated about how its games look. Also, we need to keep in mind that this method is way more surgical than a simple post-processor. They've probably grafted a lot of tapping points to video driver to be able to implement this.
alkonaut 5y ago
Is there a cert issue? I can't access the page. A cache anywhere, or youtube link perhaps?
himinlomax 5y ago
It seems to be completely removing the atmospheric haze, unfortunately.
dirtyid 5y ago
Impressively more google streetview / Russian dashboard camera like. Not a knock, quality improvement noticeable. Base GTA feels like intern 3D rendering work, after looks produced by someone fairly talented / competent. I can see this saving and incredible amount of labour.
- jiofih 5y ago
  > GTA feels like intern 3D rendering work Lol, I guess you’re new? Or maybe you don’t know the difference between 3D rendering software and game engines? GTA V came out in 2013 and the fact that it is still used as a reference for graphics quality tells you everything you need to know.
  - dirtyid 5y ago
    My old dayjob was photorealistic architectural renderings. The point is GTAV, which was/is exemplar for game engine graphics, and even current gen RTX game engine rendering all look "bad" due to constraints and budgets of real time rendering. Bad in the sense that they're necessarily rudimentary - actual photo realism requires many layers of expensive details. This technique fills in many of those kind of details. The improvement in photo realism demonstrated here is significant and a huge leap in terms of image quality.
max_ 5y ago
How does this compare to Physical Based Rendering [0]? [0]: https://en.m.wikipedia.org/wiki/Physically_based_rendering https://en.m.wikipedia.org/wiki/Physically_based_rendering
- 317070 5y ago
  For one, it will be a lot faster to compute. For two, it is a lot less laborious work to set up, which does not scale with the size of the world like PBR, but only with performance. So where previously, you needed a lot of designers to make a big world look realistic with PBR, now you need to collect a big dataset with the style you want to have, and apply it to your game. I suspect that will scale differently.
- dahart 5y ago
  FWIW, I think it doesn’t compare to PBR - these are two completely different things, and there’s no reason you can’t do both. Maybe as a simple example of that, it’s common today to run a NN based denoiser on top of a physically based path traced rendering. This neural network operates on images - input is image and output is image. Note that it cannot synthesize an image from scratch, it can only take an image and make it look more like the lighting in it’s training set. PBR is a technique for taking a 3d scene description and synthesizing an image, so for PBR you need models and lighting and materials, and even with physically based techniques it might still come out not completely realistic looking. There’s always room to emulate realistic subtleties of texture, lighting, camera lenses, video/film response, outdoor colors, atmosphere, etc.
BiteCode_dev 5y ago
Very impressive. It will hard to balance realism and readability though. GTA uses some surrealist contrasts because it makes the games easier to play, especially for long hours, or when you go fast. But the tech is so new I'm not going to complain, it's already very good, especially given what it eats in resources.
anentropic 5y ago
It makes it clear there's way too much contrast in the road textures in the actual game.
micheljansen 5y ago
Really impressive and striking to see the stylistic effects of different training sets (Cityscapes vs Mapillary Vistas). Obviously the GTA V designers applied a particular style and colour scheme which gets lost, but I'm sure something like StyleGAN could address that.
jeisc 5y ago
Does any one here know about some existing self-driving(self-crashing) technology that uses two cameras in unison on the sides of the car like the eyes of a equide providing a nearly 360 angle image only readable by the computer due to 2D perspective constraints.
amelius 5y ago
I noticed some strange (unrealistic) effects on the back of the red car at the 8:19 mark in this video: https://youtu.be/P1IcaBn3ej0?t=499 https://youtu.be/P1IcaBn3ej0?t=499
- TGRHavoc 5y ago
  You mean the shadow of the tree?
stanfordkid 5y ago
I would love to see this on NBA2K21 ... seems like you have an ideal training set with recorded basketball games.
cousin_it 5y ago
Nice! So a next gen game could be made with a simple renderer, painting areas of the screen with a tag saying "grass" or "window", and a neural network trained on high quality renders in the desired style (not necessarily photorealistic). The same technique would probably work for character animation as well.
neonological 5y ago
One day there's going to be a gan where the input is a random number and the output is the source code of an entire triple A game.
wodenokoto 5y ago
Wow, that is pretty cool. Maybe GTA 6 or 7 will include no textures and lower poly models, and instead spend a large budget driving around LA and just film stuff to train neural render.
deleted 5y ago
[deleted]
lobocinza 5y ago
Photo-realistic burgers http://youbentmywookie.com/wookie/gallery/0910_wtf/fast_food_ad_reality.jpg http://youbentmywookie.com/wookie/gallery/0910_wtf/fast_food... They made the game look bad with shitty fps. If GTA developers wanted they could make the game look photo-realistic but that was never the intention.
PedroBatista 5y ago
Kinda looks like Google StreetView. Looks great in overcast settings but has some difficulties with sunny whether. Pretty impressive.
aspaceman 5y ago
Oh this is really impressive. I've always wondered how games were going to solve the last mile of photorealism, and an additional color-grading filter seems like the perfect step. Really impressive work. I think titling it "photorealistic enhancement" is a bit of a whiff, but "photorealistic real-time color grading / filtering" is bad-ass.
bshanks 5y ago
Many of the GTA V pictures look more realistic to me. I think some of this is due to location; in the GTA V pics, the brownish hue, the haze and washed-out colors, the patchy grass, and the grainy roads seem to me to really capture the actual look of typical southern California urban sprawl. The enhancement makes things greener, less hazy, more grassy, and smooths roads, which is more typical for many areas of the United States, but not for southern California. In addition, the enhancement blurs a bit, which imo is not good; the blur adds to photorealism by hiding mistakes, but on the other hand it makes it more obvious that you are looking at a picture because you can't see details.
- yummypaint 5y ago
  For me the most unrealistic part of the GTA renders was the trees, and the post processing significantly improved that. Blur is also an important part of real photos, it's unusual for everything to be in perfect focus. I would argue that these kinds of renders are getting good enough that we should be more precise about what is meant by photorealism. Most pictures i take with my phone have alot of blur, don't capture colors well, and are frankly quite crappy. I feel like people expect photorealism to be equivalent to a really well-planned photo by a professional.
Balgair 5y ago
That's really cool! But why does every car look like it's been freshly waxed? The newspaper boxes have a bit of rust and grime on them. Those look really real.
tb0ne 5y ago
I love how the license plate in the first image gains a blue strip common in european license plates due to the dataset coming from germany.
dioporcus 5y ago
Che matta di coglioni.. Se la tecnica e vera e la svolta già con quixell c'è stato e ci sarà un balzo avanti.. E mostruoso vedere GTa reale saranno renderizzate cmq frame per frame e il processo sarà lunghissimo... Nuove tecniche...cmq ci sono già tecnologie molto avanti la verità e che continuano a vendere a 350 euro merda come schede video...qualcosa di decente a prezzi esorbitanti perche? Perché sono delle merde a loro non costerebbe per dire nulla darci schede ottime a buon mercato esistono tecnologie che metterebbero per terra le case delle schede video e dei processori... La verità e che sviluppano game su merda di PC che già costano... Se potessero basarsi su una più vasta fascia di gente con PC molto più performanti avrebbero già reso i game 10 volte meglio. Ma fare girare titoli nuovi su una già GTX 1080 si fatica con settaggi alti...cosa devo spendere 2000 euro per poter godermi un game?

522 points by lnyan 5y ago

serf 5y ago

is the road-smoothing an artifact from the fact that the camera input sources lack the ability to resolve a detailed picture of the road at-speed?

317070 5y ago
Probably yes. Alternatively, it shows that the road textures used in GTA V were not very apt for their job. I do think the smoother ones look considerably more realistic.

Gauge_Irrahphe 5y ago

Why not let it output the gbuffer images instead, or perhaps even let it operate on the artwork instead? That way you could bake the changes into the game without having to run the NN in real time.

derefr 5y ago
From what I can tell, the whole reason the road "looks like a video game" is that it's using a source texture and then stretching it along the road's geometry, "diluting" the level of detail as seen from the camera viewpoint. The image enhancement here is happening "in camera", like a fragment shader; but it's using a lot more context than the sort of local/cellular kernels that fragment shaders usually are. Look at e.g. the gutters on the roads, where there are leaves/dirt — they get filled in with tons of extra perceptual texture (texture that exists in the projection plane, rather than texture that exists "on" the road), bringing them up to the same LoD as the rest of the road. If that texture was "pushed back" to the origin texture, said texture would have to exist at some sort of ridiculous resolution that modern graphics cards wouldn't have a hope of rendering.
- gmueckl 5y ago
  The part about ridiculous resolutions for texture detail like leaves isn't necessarily true. There are very old techniques for scattering small decals densely across surfaces. The basic trick is to use coarse textures that encode the positioning of decals within each texel footprint.
  - derefr 5y ago
    Sure, but no matter what the geometry/bump maps/etc. are on the road, they're not going to render out as anything but an aliased, badly-lit mess unless you're heavily supersampling them (and then doing ray-tracing to the supersampled image.) We're talking about "objects" where the crucial detail-features of their meshes for getting correct color math out (e.g. the normal+reflectance maps) would be, on average, scaled to be smaller than a pixel from the viewing distance+angle, with potentially many objects contributing to the final color of each pixel. So, rather than rendering the road at 8x-16x and then downsampling it, in order to get a pile of leaves to appear to have ~1.5x LoD at the projection plane, you can just achieve that 1.5x LoD pixel by pixel at the projection plane. Or, to put that another way: when a digital artist is painting a landscape, they don't have to zoom in by 8x, paint every individual blade of grass, and then zoom out. They can just use an artistic technique at 1x zoom that approximates the texture that would have been created by downsampling "photographic" detail painted in at 8x to your screen resolution. Here, the ML model is playing the role of the digital artist, "painting" on the projection plane.
  - enriquto 5y ago
    The problem is not texture antialiasing (which is essentially solved). The problem is shape antialiasing, whether many distant small shapes are so close together that become a sort of translucent buzz, which is very computationally heavy to compute.
    - gmueckl 5y ago
      REYES renderers had a pretty nifty solution for shape antialiasing with their slicing and dicing of geometry into sub-pixel sized primitives. That was in the 80s, just for reference. Unfortunately, this method has a pretty complex implementation and just doesn't play all that well with hardware rasterization or path tracing and has since fallen out of favor.
dahart 5y ago
This is a good question - it certainly is possible to run a NN on the artwork or pre-g-buffer, and you could imagine having something bake the changes. To do that, you’d need access to the game’s source code and assets, and you would need to rewrite portions of the rendering engine. Training might be incredibly slow if it needs to iterate on builds of the game, or that problem could be worked around but might need considerable engineering to support the training iterations (e.g. data representation and a pipeline with the ability to render changes to the artwork and renderer on the fly without needing a game rebuild). So I’d speculate the answer to why not is that it’s just not what the authors set out to do, that they don’t have access to the game source nor the time or resources needed to refactor the renderer. Part of the magic of neural networks is their black box properties - they can do what they do without needing to understand 3d geometry or integrate with a render engine. Throw an image in, and a new image pops out, without the neural network needing to understand what it’s doing or why. That said, I would guess that coming down the road is examples of exactly what you’re suggesting - games that will use neural networks to drive realism in the assets and shaders and renderer. To some degree there are already tools starting to do this.

anigbrowl 5y ago

I mean it just looks like they're simulating the windscreen glass. I guess it's more realistic but I somewhat prefer the original.

StavrosK 5y ago
Is the windshield glass really the only change you see, when grass becomes green, reflections are added, the entire photo has a higher color temperature, etc?
- derefr 5y ago
  Looking at the last comparison on the page, the grass isn't just becoming green, it's being "puffed" out from its original geometry, as if the grass geometry is being vertex-shaded with small spiky-ball-like objects. (A bit like the look of "static grass" in model making.)
- caslon 5y ago
  Most of it is making light more intense, and most of everything else could be emulated with a blur filter and a green tint. It's impressive for what it represents technically, but it's visually subtle.
- anigbrowl 5y ago
  I went through and looked at each of the examples and gave my impression. That's why I say 'looks like...' as opposed to doing a critique of their method.
rzzzt 5y ago
I think the Cityscapes dataset pictures were taken from the car's hood; the Mercedes emblem is up close. The camera position in game also seems to match this.

nikodunk 5y ago

Amazing how this is all done in real-time, and incredible stability of image. Further, I’d love to also see a tech demo using a Californian dataset as the input: To my originally European eyes, it’s simply making GTA look more like video shot in Germany vs more realistic (there’s a difference!), though I get that any data set can be fed into this :)

mosselman 5y ago
Is it real-time rather than post processed?
- nikodunk 5y ago
  Great point I don’t know
- thih9 5y ago
  From the "Enhancing Photorealism Enhancement" video [1]: "It is a convolutional network which produces images frame by frame and can be run at interactive rates.". [1]: https://www.youtube.com/watch?v=P1IcaBn3ej0 https://www.youtube.com/watch?v=P1IcaBn3ej0 (0:19)
  - dandellion 5y ago
    From the paper: "Inference with our approach in its current unoptimized implementation takes half a second on a Geforce RTX 3090 GPU." That means 1 or 2 frames per second, so interactive but pretty much unplayable, which is to be expected as it's just research, but considering that they have DLSS working in realtime with tensor cores perhaps something like this will also be achievable very soon. Edit: Out of curiosity I looked up how long DLSS takes to process and it's less than 2ms, so you'd probably need to speed this up by two orders of magnitude to run it a game.
L_ 5y ago
There's also a version trained on Mapillary Vistas, which has images from all around the world. Results from that are at the bottom of the page and at the end of the video ( https://youtu.be/P1IcaBn3ej0?t=462 https://youtu.be/P1IcaBn3ej0?t=462 )

wefhi 5y ago

"return of GTA IV"

Toutouxc 5y ago

So, can we gradually get to a point where the engine itself only renders simplified abstractions (i.e. not the current state-of-the-art graphics) and feeds them to a neural network that produces the output?

arketyp 5y ago
This is what has been called Deep Shading: https://arxiv.org/abs/1603.06078 https://arxiv.org/abs/1603.06078
deleted 5y ago
[deleted]

dvt 5y ago

At first I was impressed, but then I realize they also work on (raw) G-buffers, depth maps, normals, etc. grabbed directly from the engine. I mean if we're already doing all of that, why not just pass it through 3DSMax and get an actual photorealistic rendering? Ray tracing is already starting to be a thing in consumer products, so we're bound to get photorealism at interactive framerates in the next decade or so. No neural networks needed. There might also be a misconception that GTAV couldn't make their game more murky/realistic-looking, but more often than not, saturation and contrast is purposefully cranked up in games (particularly driving ones). I will say that the lush background mountains look cool though.

dsign 5y ago
I watched their video, and I tend to agree. With the amount of computing they use, ray-tracing is probably going to be just as good and probably higher frame-rate. But maybe this technique, in due time, will result in an alternative class of rendering pipelines with less focus in shading and materials.
ma2rten 5y ago
I think it's worth exploring multiple approaches to a problem. Maybe this neural network approach can be improved upon and it will also generate photorealistic looking images but have other advantages such as less effort on behalf of the game developer or less compute.
arketyp 5y ago
Good point. But I wouldn't rule out neural networks. They could still be used to derive efficient shader models that can be run in real time. https://arxiv.org/abs/1603.06078 https://arxiv.org/abs/1603.06078
HellDunkel 5y ago
You no longer need 3dsmax for raytracing. But what could be really interesting is to train your ml model against path traced images and apply the outputs to selective areas of your screem where the g buffer provives insufficient information density (i. i.e.foliage)

larschdk 5y ago

Impressive, but I think that photorealism is not necessarily what you would want to strive for in games, as it is not true realism. It emulates the deficiencies of the camera that we perceive it as realistic, because we are trained on thousands of photo images and video footage. I prefer visionrealism instead of photorealism in my games and don't really find that lens effects like flares, chromatic aberration, and depth of field enhances the experience of realism. It feels like I'm looking through a camera rather than being there myself. For me it breaks immersion instead of enhancing it.

avereveard 5y ago
yeah, games visuals are more about art direction than search of realism this would work great in a flight sim tho, albeit blur and out of focus effects are terrible for visual clarity however, there's one small bit that has not been given enough attention imho, which is on the latter set, under "removes distant haze": this would be a great technique to complement flat LOD reduction of open world games while keeping up the crispness and contrast of distant items.
- simooooo 5y ago
  Would be interesting to see what it can do for LODs similar to how you see imposters used today
ginko 5y ago
That's mostly a problem with the input data. A gamestudio of the size of rockstar could afford to acquire their own custom fit dataset. Get some good cameras and lenses and capture footage of wherever they want to set their next game. Then they can apply some motion picture color grading to that and use that as the input data for the NN.
dwd 5y ago
I found their approach to be more like the difference between a digitally filmed movie (that can look too crisp) and one shot on cellulose that has a kind of softness to it. Where it noticeably fell short was displaying red lights (tail lights, traffic lights, etc) that lost their brightness.

ohlookabird 5y ago

Looks really nice. It is however funny to see what it does to road signs (black arrows suddenly turn blue or the orange "don't walk" hand of pedestrian traffic lights seems to disappear). I wonder if this has to do that they use image data from Germany (having slightly different signs than the US).

zuzun 5y ago
It's also giving the license plates the blue stripe.

oceanofsolaris 5y ago

I am especially impressed by how much more realistic the lighting ends up looking despite the network probably having less information to work with than the engine! Though it ends up turning the lighting very flat. I guess that is an effect of the source images they were using. As others have said, using this directly in the engine (tuned to work with the intended art style etc) could probably produce almost miraculous results, if it can be made to work at a reasonable frame rate. That would also allow the developers to use high-quality rendered images instead of these green-tinted, low-contrast "automotive grade" camera images as source of truth (there are good reasons these images look like that ... but they don't look pleasant).

deleted 5y ago
[deleted]

orbital-decay 5y ago

They mention temporal stability across the footage, but what about the entire game? It would be unfortunate to have the same car to turn from Audi to BMW under entirely different lighting and angles, or a palm tree into a pine. (the difference might be much more subtle but annoying nonetheless)

dahart 5y ago
Part of the contribution of this paper is about preventing that specific problem. “We also seek to eliminate artifacts that can be seen in the results of prior deep-learning approaches, which often hallucinate objects. To this end, we analyze the datasets that are commonly used for photorealism enhancement. Our analysis reveals that their scene layouts differ in ways that can explain artifacts commonly seen in prior work. To better align the datasets and alleviate the artifacts, we propose a new strategy for sampling image patches during training. We further design a new adversarial training objective that facilitates enhancements that are geometrically and semantically consistent with the content of the input image.”

xpuente 5y ago

Why is the demo at 720p? Does not look good to do something like that.

rasz 5y ago
There is a long an rich tradition to release your DNN/FX work in a post stamp sized presentation.

bawolff 5y ago

The results dont look that great to me. The original looks better for a game. The "enhanced" version is so unsaturated it could be bordering on black & white.

eutectic 5y ago
The model trained on TSIT gives nicer results.
svantana 5y ago
That's because of the dataset. Later in the video they show results using another dataset with more vibrant colour, that looks a lot better IMO.
- bawolff 5y ago
  I just watched the video to the end - definitely an improvement, but it still looks "faker" than the original to me. Maybe some uncanny valley effect is going on with me.
  - jiofih 5y ago
    You’re just used to the style. Putting a photo of a road next to it will show you how far from reality GTA rendering is.
  - 317070 5y ago
    Out of curiosity, did you play the game? To me, it is hard to imagine someone would find the game to look less fake.
    - bawolff 5y ago
      I did not. I'm not saying the game looks real, just that it looks less "wrong" somehow, to me.
ObserverNeutral 5y ago
germany is unsaturated

playpause 5y ago

Uninformed take: this looks to me like an example of ‘style transfer’. If so, it doesn’t seem as compelling as other examples I’ve seen before, eg making a photo look like a Van Gogh using a single painting as input, which has been around for a few years. Can someone with better knowledge in this area explain if I’m missing something?

L_ 5y ago
Style transfer commonly hallucinates objects or adds artifacts. Making something look like a painting is actually much easier than making it look realistic. In paintings, artifacts are often tolerated as 'artistic'. In a photo, it just pops out. Also, style transfer approaches are not temporally stable most of the time. Have a look at the comparisons with state-of-the-art image-to-image translation (CUT, TSIT) and photo style transfer (WCT2). In the case of GTA/Cityscapes shown here, most of these methods put trees in the sky and/or flicker. Also the use of G-buffers allows a much deeper and more robust translation than just using images as style-transfer approaches.
- playpause 5y ago
  Thank you, this is really interesting.
deleted 5y ago
[deleted]

gonzo41 5y ago

This is pretty impressive, I wonder if you could build really cheaply textured maps and then upgrade them with this method as a post production touch up. Essentially removing most of the human work from texture libraries.

plutonorm 5y ago

The number of comments here that are negative is just staggering to me. These images are within a hair's breadth of being indistinguishable from reality. It's an enormous leap. An enormous leap from the previous techniques and an enormous leap towards photorealistic realtime graphics. You don't want your games to look realistic? Go play pacman or outrun. You want to use path tracing instead? You have no idea how much cheaper this technique is. And on top of that the neural network is also fixing up the unrealistic texture work on all kinds of things in the scene. The first games to use this are going to be from small game studios with less investment in the status quo and they are going to blow the competition out of the water.

HellDunkel 5y ago
Maching learning is often oversold as a shortcut solution for some very hard problems and this is a perfect example. What you predict about small studios seems unlikely as ML is a beast to tame, needs highly specialized engineers and huge high qualty datasets for training.
- plutonorm 5y ago
  So were ajax crud web apps, 25 years ago.
  - zokier 5y ago
    And they still suck, 25 years later
bayindirh 5y ago
Actually, I'm pretty amazed by the results. This is indeed a giant leap. Also it's coming from Intel instead of AMD or Nvidia, which makes me hopeful about its wide availability. The only thing I didn't agree is that Rockstar is incapable of improving their engine further. Since GTA-III, this is somewhat a deliberate stylistic choice up to a certain point, like how Source engine is also opinionated about how its games look. Also, we need to keep in mind that this method is way more surgical than a simple post-processor. They've probably grafted a lot of tapping points to video driver to be able to implement this.

alkonaut 5y ago

Is there a cert issue? I can't access the page. A cache anywhere, or youtube link perhaps?

himinlomax 5y ago

It seems to be completely removing the atmospheric haze, unfortunately.

dirtyid 5y ago

Impressively more google streetview / Russian dashboard camera like. Not a knock, quality improvement noticeable. Base GTA feels like intern 3D rendering work, after looks produced by someone fairly talented / competent. I can see this saving and incredible amount of labour.

jiofih 5y ago
> GTA feels like intern 3D rendering work Lol, I guess you’re new? Or maybe you don’t know the difference between 3D rendering software and game engines? GTA V came out in 2013 and the fact that it is still used as a reference for graphics quality tells you everything you need to know.
- dirtyid 5y ago
  My old dayjob was photorealistic architectural renderings. The point is GTAV, which was/is exemplar for game engine graphics, and even current gen RTX game engine rendering all look "bad" due to constraints and budgets of real time rendering. Bad in the sense that they're necessarily rudimentary - actual photo realism requires many layers of expensive details. This technique fills in many of those kind of details. The improvement in photo realism demonstrated here is significant and a huge leap in terms of image quality.

max_ 5y ago

How does this compare to Physical Based Rendering [0]? [0]: https://en.m.wikipedia.org/wiki/Physically_based_rendering https://en.m.wikipedia.org/wiki/Physically_based_rendering

317070 5y ago
For one, it will be a lot faster to compute. For two, it is a lot less laborious work to set up, which does not scale with the size of the world like PBR, but only with performance. So where previously, you needed a lot of designers to make a big world look realistic with PBR, now you need to collect a big dataset with the style you want to have, and apply it to your game. I suspect that will scale differently.
dahart 5y ago
FWIW, I think it doesn’t compare to PBR - these are two completely different things, and there’s no reason you can’t do both. Maybe as a simple example of that, it’s common today to run a NN based denoiser on top of a physically based path traced rendering. This neural network operates on images - input is image and output is image. Note that it cannot synthesize an image from scratch, it can only take an image and make it look more like the lighting in it’s training set. PBR is a technique for taking a 3d scene description and synthesizing an image, so for PBR you need models and lighting and materials, and even with physically based techniques it might still come out not completely realistic looking. There’s always room to emulate realistic subtleties of texture, lighting, camera lenses, video/film response, outdoor colors, atmosphere, etc.

BiteCode_dev 5y ago

Very impressive. It will hard to balance realism and readability though. GTA uses some surrealist contrasts because it makes the games easier to play, especially for long hours, or when you go fast. But the tech is so new I'm not going to complain, it's already very good, especially given what it eats in resources.

anentropic 5y ago

It makes it clear there's way too much contrast in the road textures in the actual game.

micheljansen 5y ago

Really impressive and striking to see the stylistic effects of different training sets (Cityscapes vs Mapillary Vistas). Obviously the GTA V designers applied a particular style and colour scheme which gets lost, but I'm sure something like StyleGAN could address that.

jeisc 5y ago

Does any one here know about some existing self-driving(self-crashing) technology that uses two cameras in unison on the sides of the car like the eyes of a equide providing a nearly 360 angle image only readable by the computer due to 2D perspective constraints.

amelius 5y ago

I noticed some strange (unrealistic) effects on the back of the red car at the 8:19 mark in this video: https://youtu.be/P1IcaBn3ej0?t=499 https://youtu.be/P1IcaBn3ej0?t=499

TGRHavoc 5y ago
You mean the shadow of the tree?

stanfordkid 5y ago

I would love to see this on NBA2K21 ... seems like you have an ideal training set with recorded basketball games.

cousin_it 5y ago

Nice! So a next gen game could be made with a simple renderer, painting areas of the screen with a tag saying "grass" or "window", and a neural network trained on high quality renders in the desired style (not necessarily photorealistic). The same technique would probably work for character animation as well.

neonological 5y ago

One day there's going to be a gan where the input is a random number and the output is the source code of an entire triple A game.

wodenokoto 5y ago

Wow, that is pretty cool. Maybe GTA 6 or 7 will include no textures and lower poly models, and instead spend a large budget driving around LA and just film stuff to train neural render.

deleted 5y ago

[deleted]

lobocinza 5y ago

Photo-realistic burgers http://youbentmywookie.com/wookie/gallery/0910_wtf/fast_food_ad_reality.jpg http://youbentmywookie.com/wookie/gallery/0910_wtf/fast_food... They made the game look bad with shitty fps. If GTA developers wanted they could make the game look photo-realistic but that was never the intention.

PedroBatista 5y ago

Kinda looks like Google StreetView. Looks great in overcast settings but has some difficulties with sunny whether. Pretty impressive.

aspaceman 5y ago

Oh this is really impressive. I've always wondered how games were going to solve the last mile of photorealism, and an additional color-grading filter seems like the perfect step. Really impressive work. I think titling it "photorealistic enhancement" is a bit of a whiff, but "photorealistic real-time color grading / filtering" is bad-ass.

bshanks 5y ago

Many of the GTA V pictures look more realistic to me. I think some of this is due to location; in the GTA V pics, the brownish hue, the haze and washed-out colors, the patchy grass, and the grainy roads seem to me to really capture the actual look of typical southern California urban sprawl. The enhancement makes things greener, less hazy, more grassy, and smooths roads, which is more typical for many areas of the United States, but not for southern California. In addition, the enhancement blurs a bit, which imo is not good; the blur adds to photorealism by hiding mistakes, but on the other hand it makes it more obvious that you are looking at a picture because you can't see details.

yummypaint 5y ago
For me the most unrealistic part of the GTA renders was the trees, and the post processing significantly improved that. Blur is also an important part of real photos, it's unusual for everything to be in perfect focus. I would argue that these kinds of renders are getting good enough that we should be more precise about what is meant by photorealism. Most pictures i take with my phone have alot of blur, don't capture colors well, and are frankly quite crappy. I feel like people expect photorealism to be equivalent to a really well-planned photo by a professional.

Balgair 5y ago

That's really cool! But why does every car look like it's been freshly waxed? The newspaper boxes have a bit of rust and grime on them. Those look really real.

tb0ne 5y ago

I love how the license plate in the first image gains a blue strip common in european license plates due to the dataset coming from germany.

dioporcus 5y ago

Che matta di coglioni.. Se la tecnica e vera e la svolta già con quixell c'è stato e ci sarà un balzo avanti.. E mostruoso vedere GTa reale saranno renderizzate cmq frame per frame e il processo sarà lunghissimo... Nuove tecniche...cmq ci sono già tecnologie molto avanti la verità e che continuano a vendere a 350 euro merda come schede video...qualcosa di decente a prezzi esorbitanti perche? Perché sono delle merde a loro non costerebbe per dire nulla darci schede ottime a buon mercato esistono tecnologie che metterebbero per terra le case delle schede video e dei processori... La verità e che sviluppano game su merda di PC che già costano... Se potessero basarsi su una più vasta fascia di gente con PC molto più performanti avrebbero già reso i game 10 volte meglio. Ma fare girare titoli nuovi su una già GTX 1080 si fatica con settaggi alti...cosa devo spendere 2000 euro per poter godermi un game?