11 ms·
Ray Tracing Essentials Part 7: Denoising for Ray Tracing
- simias 7y agoThe tech is cool but I still can't shake the feeling that from a practical perspective, in the context of videogame graphics, that's a lot of work for a meager difference in the end result compared to good old rasterizer shading. Sure, you get more accurate shadows and reflections... But I'd gladly trade that for more detailed models and larger, more complex environments for instance. Maybe it'll become the new standard for real time 3D graphics in the future but for the time being I file it next to HairWorks as an nvidia gimmick whose main purpose is to make the competition look worse in benchmarks because they don't implement that API when it practice the visual difference is fairly subtle.
- MH15 7y agoA lot of Nvidia's new RTX tech may be marketed towards gaming but it's actually having an impact in 3D rendering. Cycles supports RTX out of the box now and I'm sure other renderers will/do as well. And the denoising tech is amazing, while not up to the challenge on final renders it's perfect for getting a quick frame upstream.
- m12k 7y agoIf nothing else the tech allows 3D artists and level designers to get instant feedback while they are tweaking a scene, without having to wait for a full render/lightmap bake - having iteration time go from minutes to seconds is a game changer for creative work like that.
- fxtentacle 7y agoBut sadly, it doesn't work like that. Let's say I render a reflective vase in V-Ray and use NVIDIA OptiX denoising. The ray-tracer will cast a few specular rays so that some pixels will get super bright from having a direct reflection of the light source, while other pixels will not yet receive specular illumination. The OptiX denoising will then blur things out, so that the entire vase is equally brightened up. But that quick preview looks so significantly different from the final rendering result, that it is effectively useless and, of course, highly misleading. In the final result, only the parts of the surface where the curvature is "just right" will receive specular. In my preview, everything will.
- virtualritz 7y agoUsing GPUs for offline rendering to gain speed is a dangerous fallacy. Dangerous because the GPU has many restrictions which developers have to invest time in working around. I feel in recent years makers of offline renderers have spent more time dealing with these restrictions to get their renderers to work well with GPUs than actually improving their renderers. I.e. reduce level of noise, better algorithms for hair, volumes, etc. The renderers produce way too much noise (or have prohibitive render times w/o except for big studios). Solution? Denoising! Which is (mostly) using GPU-powered ML! The irony of this seems lost on many freelancers who buy expensive GPU rigs to be able to deliver their jobs on the likes of OTOY or Redshift. For this money you can buy a vastly superior CPU rig. If only your renderer could use it! Most people compare apples to oranges here. When I hear people rave about GPU offline renderers, I usually ask them: - Have you made time to first pixel part of your comparison (complex scenes can take dozens of minutes before they start rendering on GPU renderers). - Are you comparing images as they come out of renderer A before denoising to images of renderer B before denoising? Because anyone can pipe their shit through Intel Open Image Denoise after the fact. It's free even. - Have you actually used a machine that has more than eight cores personally in production? If all your money goes into buying expensive graphics cards your CPU is usually shite. A notable exception is 3Delight which doesn't even ship a denoiser. On simple scenes the renderer can't beat the GPUs but on anything a bit more complex -- the stuff you deal with in actual production -- it smokes the competition CPU & GPU alike. Not least in the amount of noise it produces using comparable sampling settings. Without using any GPU compute. When this GPU craze took over other vendors who's renderers were traditionally CPU bound too, in the last 5-6 years, 3Delight's developers have spent all their time finessing the performance and output quality w/o relying on GPUs. It shows.
- ben-schaaf 7y agoCycles does the same work regardless of whether it's running on the CPU or GPU. You can even run mixed where both the CPU and GPU do rendering. The amount of noise has nothing to do with it running on the GPU or not, that's simply how path-tracing works...
- 7y ago
- EvbodysNobs191 7y agojust wondering is it possible to implemented in FPGA for ioquake3? https://researchspace.auckland.ac.nz/handle/2292/36394 https://researchspace.auckland.ac.nz/handle/2292/36394
- speedgoose 7y agoIf you have static lights in a static world with pre-computed raytraced lights, yes it doesn't change that much. But once you have a dynamic world and dynamic lights, it's a game changer. Minecraft with raytracing is a great example. It's so much better with raytracing compared to simple shaders. I also heard that it will make game development a lot smoother and faster in the futur once most hardware will be powerful enough. No need to build lightmasses, no ambiant occlusion hacks and similar, no reflections with approximate results.
- sprash 7y agoThis[1] demo does pretty convincing dynamic lighting without the need for RTX relying just on regular shaders. To me nvidias RTX looks like a gimmick with the sole purpose of enforcing vendor lock-in. 1.: https://www.youtube.com/watch?v=GtU2194C-D4 https://www.youtube.com/watch?v=GtU2194C-D4
- DavidVoid 7y agoRTX is hardware acceleration for many ray tracing operations, how is that a gimmick?
- dahart 7y agoThat demo is a ray marching shader with hardly any geometry, and the geometry is procedural. You do understand this method can't be used to render video games or films? > RTX looks like a gimmick with the sole purpose of enforcing vendor lock-in. That's like saying a floating point unit, or a Google TPU, or even an Intel CPU is a gimmick to enforce vendor lock-in. If you want to do something faster in software, you can make hardware for it. Don't buy hardware if you don't need speed.
- sprash 7y agoThe most expensive step in ray marching is to find the distance which you will get for free after one rasterization pass via the z-buffer. With the z-buffer already present every effect used in this demo can be applied in the same way to purely rasterized graphics with even less cost. E.g. Doom Eternal has very elaborated dynamic lighting effects but uses 100% forward rendering. RTX is completely pointless.
- mschuetz 7y agoI think the most important thing to do with RTX will be indirect illumination. Maybe RTX will be capable of computing relatively high quality lightmaps on the fly, store the results for as long as necessary, and keep recomputing for new parts of the scene.
- deleted 7y ago[deleted]
- xbmcuser 7y agoRay tracing is not powerful enough nor are games built yet from the ground up with ray tracing in mind. With the newer generation consoles expected to arrive with ray tracing that will change then probably we will get to see something different. Personally I feel raytracing will make the most difference in vr rather than normal games.
- onion2k 7y agoI'd gladly trade that for more detailed models and larger, more complex environments for instance. No doubt nVidia could make a card that enabled games to support models with double the polycount of today's games, but if companies refuse to ship games with them because the art effort is too great then it's not going to get players to update their PCs. I suspect the push towards raytracing is a way of selling more graphics cards without radically increasing the amount of investment needed by games companies in the way that higher fidelity models or more complex environments would.
- virtualritz 7y ago> No doubt nVidia could make a card that enabled games to support models with double the polycount of today's games [...] The solution is to use tessellation on the GPU, on demand, based on projected size of features. In blockbuster VFX heavy models are a recent phenomenon because mircopolygon renderer cores were replace with ray-tracing ones. Models used to be much lighter, subdivided at render time and displacement, both textured as well as resolution unlimited procedural, added the final details. The same offline rendering techniques that enabled the VFX of the 90's and early 2000's could be utilized in GPUs today to push geometric detail to the next level in realtime graphics.
- gmueckl 7y agoMicropolygons work beautifully for local only shading, but that is a very limited model and a thing of the past. The milage that you can get out of this model that was designed for severely memory constrained systems (high resolution cinematic animations on computers with 4MB if RAM) is enormous. But once you bounce rays randomly through a geometrically complex scene, you either re-tesselate all possibly intersected geometry dynamically again and again with a comsiderable runtime overhead or you pre-tessellate the entire scene based on camera position and film resolution. The later approach is the basis for VFX at the moment, but some renderers like Renderman still can do dynamic on demand tessellation where required AFAIK.
- pjmlp 7y agoIt is already being used in movie studios for a better rendering experience. https://home.otoy.com/render/octane-render/ https://home.otoy.com/render/octane-render/ Fun fact, at GTC 2020 Otoy revealed that they are moving away from Vulkan and adopting CUDA instead (via Optix 7) due to better compute power and tooling for ray tracing algorithms. https://developer.nvidia.com/optix https://developer.nvidia.com/optix
- crote 7y agoIf you want more detailed models and more complicated scenes, you should really look into raytracing. The problem with raytracing is that it mostly depends on your resolution, not on your scene. Right now, we're just on the edge of being able to do raytracing at practical resolutions in realtime. But once that's possible, the rest is basically free. Rendering 10M polygons is almost as fast as rendering 1K polygons. And writing a path tracer isn't that hard - most of the math is quite trivial. You get all the magic shadow / reflection / caustic / lighting stuff for free. We're getting quite close to being able to do movie-level graphics on consumer machines in realtime, and all that without any of the tricks required for okay-ish looking rasterizer shading. Best of all, it's even easier to work with for artists as well!
- fxtentacle 7y ago"Rendering 10M polygons is almost as fast as rendering 1K polygons" That is correct for ray tracing. But modeling and animating 10M polygons requires a Hollywood movie budget, while I can model and convincingly animate 1K polygons myself.
- crote 7y agoOf course, but at least now you have the ability to render it. "Black Ops 2" made $1B in 15 days, so some studios already have this budget, so they should be able to make use of it. As for smaller players? Personally, I'd be interesting in procedural generation, or just modelling NURBS surfaces. And have you seen the demos of raytraced Minecraft / Doom? The highly advanced lighting already makes it look way more detailled, even with a low poly count. Even low budget games will be able to make good-looking games.
- fxtentacle 7y agoThe Doom+Raytracing pictures look great, but I believe they are fake https://www.dsogaming.com/news/doom-rtx-released-mod-bringing-real-time-ray-tracing-to-the-classic-doom-game-via-the-vulkan-api/ https://www.dsogaming.com/news/doom-rtx-released-mod-bringin... For Minecraft, the water is a bit nicer, but it still looks very unconvincing to me: https://www.youtube.com/watch?v=nXoLY9lF-NI https://www.youtube.com/watch?v=nXoLY9lF-NI
- fxtentacle 7y agoFunny that you mention HairWorks, because that one causes a lot of issues in practice because the hair is rendered in post-processing. Therefore, it is not present in the z-buffer so that alpha-blended transparent effects (e.g. smoke) may accidentally overlap with the hair. For an example of this issue, see the excellent Shadow of the Tomb Raider game.
- alkonaut 7y agoSame. You can always spend those transistors/cycles/watts on something else that is more compelling (more frames, better AA or higher rennder resolution). For most games doing 4K/120 without raytracing would probably be a better experience than 4K/60 with raytracing.
- xioxox 7y agoHave a look at the game Control. I think this is a game where I feel they've used the technology really well. Maybe the office building location is a good one for raytracing, as it's full of reflective surfaces, but the graphics feel substantially better with it switched on. Remedy Games also managed to maintain a reasonable framerate with lower end RTX hardware. Other games look less impressive, however, such as minor changes in the shadows. It obviously requires some thought before it can be used.
- gmueckl 7y agoControl has fantastic raytraced reflections. I don't want to go back to faked reflections based on screen space hacks and cube maps. The game gave me a few accidental jump scares because I repeatedly registered movements in glass windows that was really just a vage reflection of something else in the scene, e.g. the face of the player character that you usually see only from behind.
- ekianjo 7y agoCharacter animation is still a field that is lacking, compared to otherwise more and more realistic environments. So Raytracing may help make interiors and exteriors really impressive, but you'll be stuck with characters that have the same looping animation when they run and move around.
- DavidVoid 7y agoA lot of computational work yes (with the currently available hardware at least), but many visual effects are much much easier to implement with ray tracing than with rasterization techniques. Dynamic off-screen reflections, soft shadows, ambient occlusion, dynamic global illumination, caustics etc. can all be implemented more easily and realistically with ray tracing than with previous techniques.
- nilkn 7y agoI think Minecraft RTX pretty much seals the deal: ray tracing will definitely be the future of lighting in real-time computer graphics.
- tasty_freeze 7y agoIn addition to the examples other have given where it improves the visuals, raytracing can make developing a game easier. Currently, game developers spend huge amounts of time cooking up scene specific lighting hacks, things that make no physical sense but result in the visual appearance they are after. Another hack is dynamically computing an environment map for a shiny object (that is, rendering the scene from the vantage point of the object) so it can be used as a texture to model first order reflections. It is infeasible to compute that dynamic environment map for every object, so the art department picks and chooses which assets get reflections and which don't. Or maybe an environment map is computed from the vantage point of the centroid of a car, which is used for all reflections (windshield, bumpers, roof, hood), and that works OK for objects which are far away from the car. But if there are any objects near the car, the reflection angles are wrong in different ways on different parts of the car. With raytracing, a lot fewer hacks are needed. Yes, what nvidia is offering here is a hack in its own way (denoising, neural net upscaling too), but is one hack which applies to all cases, vs the old way of needing to create a different hack for each situation.
- _dps 7y ago> scene specific lighting hacks, things that make no physical sense but result in the visual appearance they are after. The flip side of this is that physically realistic shadows aren't always what you want. Any amateur photographer who has noticed just how awful hard shadows are on faces on clear sunny days has learned this lesson the hard way. To quote a graphics professor of mine (tongue in cheek), "Physically accurate rendering is a crutch for people who aren't smart enough to cheat their way to the look they want".
- deleted 7y ago[deleted]
- touchpadder 7y agoI watched some videos comparing ray traced and rasterised graphics and in real life the benefit seems negligible. On top of that rasterised graphics use all kinds of smart techniques to improve the performance. Ray tracing the whole scene is kind of brute forcing, brick and mortar method. It should be used selectively and not on each frame IMO
- mehmetoguzderin 7y agoAll kinds of smart techniques are emerging for ray tracing, too; you don't have to shoot the rays directly from the camera per se. Both rasterization and ray tracing will coexist to provide better graphics, and it was just that ray tracing didn't have the place it deserves on the chip.
- ihaveajob 7y agoFrom the fundamental point of view, the cost of raytracing grows proportional to the number of pixels rendered times log(scene size), whereas rasterization grows linearly with the scene size. So raytracing enables much more complex scenes at a fraction of the cost of rasterizing, which explains why it's being used more and more in real time rendering.
- ccmonnett 7y agoFor others interested in why ray tracing scales O(log N), this is covered in an earlier video in the NVIDIA "Ray Tracing Essentials" series, specifically this one: https://youtu.be/ynCxnR1i0QY?t=173 https://youtu.be/ynCxnR1i0QY?t=173 It's timestamped to the discussion of why this is true, but the whole video (like the series, IMO) is very informative, this one focused on "Rasterization vs Ray Tracing".
- sudosysgen 7y agoFor someone that doesn't want to watch, the essence of it is that space is world-space is divided in a tree-like structure, which makes traversing the scene as costly as traversing the tree, thus log(n) operations.
- deleted 7y ago[deleted]
- fxtentacle 7y agoWhile this works OK-ish for static images, it is almost unusable for animations. But for static images, blur+sharpen in Photoshop has worked OK-ish, too, for many years. So the practical benefit of this is negligible. What we need is a denoising technique where the denoising artifacts move convincingly with the features of the scene, so that you can use it for movies and games. Oh and for games, as long as this is NVIDIA-exclusive, developers have to treat it as an optional add-on. For multiplayer games, that implies that Ray Tracing may never show details (such as a reflection of an enemy) that would give a strategical advantage. Plus the real issue with contemporary game development is that consoles make a majority of the revenue (due to less piracy) but they choke when you have 50k+ polygons on an animated character. And you'll be limited to 2 GB GPU RAM on 30% of your PC player base, because they use laptop GPUs. In the end, then, you usually don't have enough detail to make ray-tracing look good. It looks amazing for high-poly curved surfaces, such as those used for offline-rendered cinema movies. But on a blocky realtime game model, ray-tracing may also highlight artifacts. Here's a ray-traced low poly bunny: https://i.imgur.com/MGotRC7.png https://i.imgur.com/MGotRC7.png Notice how clearly you can see that it is low poly. In a rasterization engine, one would "fix" this by blurring the edges with shaders and bending the corners with normal maps. So in a sense, ray-tracing is too honest to work well with current video game models.
- FreeFull 7y agoThere's no reason why ray-tracing can't work with bent normals. That's how Blender's "smooth shading" works, and Blender uses raytracing for rendering.
- fxtentacle 7y agoThat is correct for Cycles, but the kind of raytracing that is possible with hardware-acceleration inside a GPU is severely restricted in comparison to a general-purpose CPU raytracer like Blender's Cycles. I believe Eevee, the Blender GPU ray-tracer, does also currently not support having reflection rays collide with surface generated by normal/displacement operations, but instead they will approximate flight path by using the unmodified geometry.
- swalsh 7y agoThis might be a stupid question, because I don't really know what I'm talking about.. But is there a way this could be used to improve physics engines? My understanding is that a physics engine generally uses continuous collision detection. I don't know if noise plays a roll in casting for collisions, but if you could cast less rays, and fill in the holes in the same way... that seems like it could improve performance there too?
- OneGuy123 7y agoThe problem is that you might get a lot of instability. The most obvious case: stack 20 boxes one slightly above other and let them fall. If you will have random noise in there the simulation can become unstable/explode.
- Swiffy0 7y agoAren't the collision models already pretty simple as is? Simple shapes like boxes, spheres and lines, for which the collisions are fast to calculate. Don't know how it works if you have full blown meshes colliding though...
- Out_of_Characte 7y agoI dont know, it depends on implementation. Collision detection doesn't involve any graphics, it might not paralellise well. so it might still be easier on the CPU.
- screye 7y agoI wouldn't be surprised to see something like that. Remember that Krishevsky's software implementation for Alexnet of making neural networks work with GPUs, wasn't initially intended by Nvidia either. But, look how that worked out.
- aspaceman 7y agoYes. I think I’ve seen some preliminary papers with related ideas (because that seems like where ML is at) but I don’t remember anything substantial. It could be an interesting area of research. Start branching out from ray tracing to other integration problems.
- dmitshur 7y agoWhat APIs are being used to perform ray tracing in these new games that support it? Edit: Found a starting point of an answer, covering Nvidia’s hardware at least, at https://developer.nvidia.com/rtx https://developer.nvidia.com/rtx: > Ray tracing acceleration is leveraged by developers through NVIDIA OptiX, Microsoft DXR enhanced with NVIDIA ray tracing libraries, and the upcoming Vulkan ray tracing API.
- seanalltogether 7y agoI wonder if the same concept of running the output image through a fast neural network could be applied to typical rasterized game engines. For instance, applying antialiasing or anisotripic filtering or shadow blurring?
- ATsch 7y agoThis is more or less what DLSS is