7 ms·

Mitsuba 3 Physically Based Renderer

219 points by oumua_don17 4y ago

solarmist 4y ago
Anyone know how this relates to GPU ray tracing or if it does or why it is exciting?
- keyle 4y ago
  I know nothing of Mitsuba, but one of the optional requirements is nvidia drivers for gpu compute. So it looks like it does.
  - mattkrause 4y ago
    There are “variants” that use simple scalar floats, CUDA, and an intermediate mode that does CPU-based vectorization. The same folks have made some vectorization libraries: enoki (used in v2) and a newer one called Dr. Jit. More about it here and why it’s different from (e.g.,) PyTorch https://drjit.readthedocs.io/en/latest/intro.html https://drjit.readthedocs.io/en/latest/intro.html
- erichocean 4y ago
  It does use the GPU, but this isn't really for production use (at all). It's a research renderer and its key functionality is reversible rendering—going from an image to the scene parameters that would create that image. The purpose of this kind of renderer is to develop research ideas that will eventually make it into a production renderer someday…
  - solarmist 4y ago
    Thank you. I didn’t have a clue why I should care about this.
  - IIAOPSW 4y ago
    Unless it simply encodes scene information as imperceptible color gradients, this is impossible as a render necessarily projects from 3d to 2d.
    - throw149102 4y ago
      It's not possible to do uniquely, i.e. producing the unique scene that produced the image, but it's possible to produce some scene that would produce that image.
      - yababa_y 4y ago
        If it's possible to model those diffractive materials they use for those diffractive neural networks, it would be entertaining to do an inverse rendering on a digit recognition and see what if anything it can do.
      - IIAOPSW 4y ago
        It is possible to produce infinite scenes that would produce the image, including the scene which contains the render itself as a flat object viewed through a camera. Sampling from this infinite set of possibilities isn't useful unless you introduce a ton of fragile assumptions about what you could be looking at to begin with, in which case you kind of already knew what was in the scene.
        erichocean 4y ago
        > in which case you kind of already knew what was in the scene It's almost like I wrote "scene parameters" for a reason. More info here: https://blog.qarnot.com/an-overview-of-differentiable-rendering/ https://blog.qarnot.com/an-overview-of-differentiable-render... > While these techniques came closer and closer to photorealism, another question arose: what if instead of going from a 3D scene to a 2D image (rendering), we went from a 2D image to a 3D scene? As you may imagine, reconstructing 3D scenes from 2D information is quite complex, but there have been many advances in the last few years. This area of study is called inverse graphics.
      - taylorius 4y ago
        This is true, though machine learning models will learn typical scene structures, alongside the rendering parameters. It is also common to use more than one source image (from a different viewpoint, a bit like photogrammetry).
- dahart 4y ago
  There’s a page with a paper and video about to their Just-In-Time compilation engine: http://rgl.epfl.ch/publications/Jakob2022DrJit http://rgl.epfl.ch/publications/Jakob2022DrJit It relates to GPU ray tracing because the engine can compile either CPU or GPU specific code, specialized to the available hardware and tailored to the specific scene data; makes it so you can render faster than otherwise.
dang 4y ago
Related: Mitsuba – A physically based renderer - https://news.ycombinator.com/item?id=10096737 https://news.ycombinator.com/item?id=10096737 - Aug 2015 (8 comments)
kristianov 4y ago
The tagged line is really weird. Physically Based? Not really based indeed.
- mattkrause 4y ago
  How so? Raytracing tries to create images by simulating how light bounces around an environment. It’s physically-based in that it uses laws of physics/optics to do that (e.g., Snell’s Law when light passes through different media), rather than some method that just happens to look cool.
jiggawatts 4y ago
We're at a cusp in technology where "reverse rendering" is rapidly progressing from a research curiosity to a mainstream product. As soon as high-quality model generation becomes possible on a typical workstation, there will be a dramatic shift in how computer game art assets are produced. Many artists will be replaced by this mechanical, automated method of 3D model production. No longer will you need a team of 100 artists for every 2-3 developers to manually model and texture objects, instead you'll have 10-20 photographers running around taking pictures of real-life objects and scenes. These will then be reverse-rendered into models, which can then be directly placed into the game world by just a handful of map designer artists. You can see this in Unreal Engine 5, where they combined their "infinite detail" polygon engine Nanite with their acquisition of Quixel's Megascans library of models captured via photogrammetry. However, photogrammetry is not the same as reverse rendering: it's only really suitable for diffuse objects captured in good conditions (foggy or overcast days). It can't handle transparency, reflections, or complex/dynamic self-shadowing. Currently it's mainly used for backdrops like rocks, logs, and road surfaces. Reverse rendering can in principle capture anything a renderer can do, including specular materials. It can "undo" self-shadowing and produce objects that can be re-lit in different scenes. For gamers, these are going to be exciting times! For graphic artists... perhaps also exciting, but not necessarily in a good way.
- paisawalla 4y ago
  It seems like this means artists would mostly be needed for creative art, e.g. spaceships, monsters, etc. That is, things that don't exist, so can't be photographed. But there are already commercial and free libraries of commonplace items like furniture, cars, and buildings. So isn't what I described pretty close to the same situation today already?
  - taylorius 4y ago
    But spaceships and monsters do exist. There are thousands of them, just look on google images. An AI inverse rendering system, could just as easily ingest a ton of space sci-fi images, as real world photos.
- 4y ago
alleycat5000 4y ago
This is a nice tutorial on differentiable and inverse rendering (one of the Mitsuba authors also gives the tutorial): https://shuangz.com/courses/pbdr-course-sg20/ https://shuangz.com/courses/pbdr-course-sg20/ Inverse rendering is such a cool idea... solving for the inputs (3D models and materials) to a renderer such that they match real world inputs (photographs) by computing gradients automatically.
airstrike 4y ago
Couple this with VR glasses and you can relive your favorite memories in first person rather than through a photograph. I've been thinking about this for probably a decade now, and can't wait for it to become reality.
- taylorius 4y ago
  I'd be careful with that. My suspicion is that our memories would transform themselves into memories of the simulation, rather than the original event.
  - airstrike 4y ago
    I’d watch that sci-fi film
ykl 4y ago
As with almost everything that comes out of Wenzel Jakob's lab at EPFL, the documentation for Mitsuba 3 is _incredible_. Check out this writeup of how polarization is handled in Mitsuba 3 (complete with animated diagrams): https://mitsuba.readthedocs.io/en/latest/src/key_topics/polarization.html#validation-against-measurements https://mitsuba.readthedocs.io/en/latest/src/key_topics/pola... That writeup is PhD Thesis material in terms of scope and detail, but the student that wrote it, Tizian Zeltner, actually wrote his PhD on another topic entirely. This is just the level of stuff that routinely comes out of that lab.
- throwaway17_17 4y ago
  Agree on the quality. I have several documents from the Mitsuba 2 era that are of the absolute highest quality. And for anyone wondering, this counts both the quality of content and the actual success of communicating complex and varied ideas in commendable fashion.
- bayindirh 4y ago
  This is a result of a group in harmony with people who are in love with what they are doing. When you like the subject, and pour what you have into it, this is what comes out. I'm sure this wasn't tiring for them. Make no mistake, this is a very high achievement, and is very rare in both private companies and research labs. It's a very hard point to reach if you don't like or care about what you are doing. Kudos to them.
AdamH12113 4y ago
What is a physically based renderer? It sounds like ray tracing that can be done in reverse on a final image?
- cornstalks 4y ago
  I’ve always seen “physically based” to mean the renderer does a physical simulation (photons bouncing, refracting, having wavelengths stretched, subsurface scattering, BSDF calculations, etc.). So much in computer graphics are just (clever) hacks upon hacks to get something that looks “good enough” but isn’t really simulating physics in any meaningful way (like SSAO, texture baking, bump mapping, etc.). These hacks are much, much faster than simulating the physical process of photons interacting with the world. I don’t know where “physically based” originated but my first introduction to it was pbrt, which I suspect popularized “physically based” naming. Differentiable rendering is the name of going from the final image and “reverse rendering” it to reconstruct a 3D model of it.
  - fluxem 4y ago
    Some clarification regarding the differentiable rendering vs inverse rendering. Differentiable rendering is what it says: differentiating the rendering process. Imagine that x is the scene, then f(x) is the function that renders the image. Then, Differentiable rendering is simply taking the derivatives: f'(x). Inverse rendering is a process of finding scene parameters x, such that f(x) produces the given image y. This is often achieved by using differentiable rendering together with an optimization algorithm (like SGD or Adam). However, due to the nature of rendering, it's easy to get stuck in a local minimum. Therefore, even a perfect differentiable rendering engine is not sufficient for the inverse rendering.
  - ohgodplsno 4y ago
    Mind you, physically based renderers also use a bunch of clever hacks upon hacks to be able to run in realtime. Nobody is simulating photons bouncing in the nanostructure of a material. We just have formulas thay say "for this roughness and metallicness, your material will look like that and it'll be good enough". We still abuse the hell out of UV maps to fake things, still include ambient occlusion (as a stylistic choice most of these days). It's a different workflow that gives you consistency once you know what you're doing, as opposed to the days of Phong shading.
nl 4y ago
The inverse rendering (Radiance Field Reconstruction (NeRF-like)) is very impressive. It can reconstruct physical models from multiple photos. See the final results at https://mitsuba.readthedocs.io/en/latest/src/inverse_rendering/radiance_field_reconstruction.html#Results https://mitsuba.readthedocs.io/en/latest/src/inverse_renderi... (scroll down a bit to see the reconstructed image vs the original)
junon 4y ago
This is missing examples of the outputs. I don't see any actual renders of note. Am I missing something?
- moonchrome 4y ago
  From what I see the point of this renderer is that it can invert the rendering to come up with a 3d scene that matches the input render image, so I guess outputs aren't as important ?
deleted 4y ago
[deleted]
andybak 4y ago
IMHO it's "differentiable" that's the most interesting part of this. There's tons of physically based renderers. It's odd that that word doesn't appear on http://www.mitsuba-renderer.org/ http://www.mitsuba-renderer.org/ - unless "retargetable" carries a similar meaning?
- doctorhandshake 4y ago
  This was my reaction - I clicked for PBR and was surprised to find the much more interesting lede buried
naillo 4y ago
Just waiting for someone to apply CLIP guidence onto this (so I don't have to).
majoe 4y ago
As someone coming from the physics side and not so much from the Computer graphics side of physically based rendering, I have high hopes for differentiable rendering as a general tool to extract knowledge from light-based measurements. Most differentiable renderers stop at the object surface, but differentiable volumetric light transport is what would allow us to use the renderer to optimise a measurement device or for medical diagnostics or just to estimate the optical properties of a material (that is absorption and scattering coefficent and scattering phase function), which is a pretty hard problem in itself. From a quick look, Mitsuba 3 seems to have everything in place to be used in such a setting and I'm really excited to see some use cases outside of Computer graphics.
picsao 4y ago

219 points by oumua_don17 4y ago

solarmist 4y ago

Anyone know how this relates to GPU ray tracing or if it does or why it is exciting?

keyle 4y ago
I know nothing of Mitsuba, but one of the optional requirements is nvidia drivers for gpu compute. So it looks like it does.
- mattkrause 4y ago
  There are “variants” that use simple scalar floats, CUDA, and an intermediate mode that does CPU-based vectorization. The same folks have made some vectorization libraries: enoki (used in v2) and a newer one called Dr. Jit. More about it here and why it’s different from (e.g.,) PyTorch https://drjit.readthedocs.io/en/latest/intro.html https://drjit.readthedocs.io/en/latest/intro.html
erichocean 4y ago
It does use the GPU, but this isn't really for production use (at all). It's a research renderer and its key functionality is reversible rendering—going from an image to the scene parameters that would create that image. The purpose of this kind of renderer is to develop research ideas that will eventually make it into a production renderer someday…
- solarmist 4y ago
  Thank you. I didn’t have a clue why I should care about this.
- IIAOPSW 4y ago
  Unless it simply encodes scene information as imperceptible color gradients, this is impossible as a render necessarily projects from 3d to 2d.
  - throw149102 4y ago
    It's not possible to do uniquely, i.e. producing the unique scene that produced the image, but it's possible to produce some scene that would produce that image.
    - yababa_y 4y ago
      If it's possible to model those diffractive materials they use for those diffractive neural networks, it would be entertaining to do an inverse rendering on a digit recognition and see what if anything it can do.
    - IIAOPSW 4y ago
      It is possible to produce infinite scenes that would produce the image, including the scene which contains the render itself as a flat object viewed through a camera. Sampling from this infinite set of possibilities isn't useful unless you introduce a ton of fragile assumptions about what you could be looking at to begin with, in which case you kind of already knew what was in the scene.
      - erichocean 4y ago
        > in which case you kind of already knew what was in the scene It's almost like I wrote "scene parameters" for a reason. More info here: https://blog.qarnot.com/an-overview-of-differentiable-rendering/ https://blog.qarnot.com/an-overview-of-differentiable-render... > While these techniques came closer and closer to photorealism, another question arose: what if instead of going from a 3D scene to a 2D image (rendering), we went from a 2D image to a 3D scene? As you may imagine, reconstructing 3D scenes from 2D information is quite complex, but there have been many advances in the last few years. This area of study is called inverse graphics.
    - taylorius 4y ago
      This is true, though machine learning models will learn typical scene structures, alongside the rendering parameters. It is also common to use more than one source image (from a different viewpoint, a bit like photogrammetry).
dahart 4y ago
There’s a page with a paper and video about to their Just-In-Time compilation engine: http://rgl.epfl.ch/publications/Jakob2022DrJit http://rgl.epfl.ch/publications/Jakob2022DrJit It relates to GPU ray tracing because the engine can compile either CPU or GPU specific code, specialized to the available hardware and tailored to the specific scene data; makes it so you can render faster than otherwise.

dang 4y ago

Related: Mitsuba – A physically based renderer - https://news.ycombinator.com/item?id=10096737 https://news.ycombinator.com/item?id=10096737 - Aug 2015 (8 comments)

kristianov 4y ago

The tagged line is really weird. Physically Based? Not really based indeed.

mattkrause 4y ago
How so? Raytracing tries to create images by simulating how light bounces around an environment. It’s physically-based in that it uses laws of physics/optics to do that (e.g., Snell’s Law when light passes through different media), rather than some method that just happens to look cool.

jiggawatts 4y ago

We're at a cusp in technology where "reverse rendering" is rapidly progressing from a research curiosity to a mainstream product. As soon as high-quality model generation becomes possible on a typical workstation, there will be a dramatic shift in how computer game art assets are produced. Many artists will be replaced by this mechanical, automated method of 3D model production. No longer will you need a team of 100 artists for every 2-3 developers to manually model and texture objects, instead you'll have 10-20 photographers running around taking pictures of real-life objects and scenes. These will then be reverse-rendered into models, which can then be directly placed into the game world by just a handful of map designer artists. You can see this in Unreal Engine 5, where they combined their "infinite detail" polygon engine Nanite with their acquisition of Quixel's Megascans library of models captured via photogrammetry. However, photogrammetry is not the same as reverse rendering: it's only really suitable for diffuse objects captured in good conditions (foggy or overcast days). It can't handle transparency, reflections, or complex/dynamic self-shadowing. Currently it's mainly used for backdrops like rocks, logs, and road surfaces. Reverse rendering can in principle capture anything a renderer can do, including specular materials. It can "undo" self-shadowing and produce objects that can be re-lit in different scenes. For gamers, these are going to be exciting times! For graphic artists... perhaps also exciting, but not necessarily in a good way.

paisawalla 4y ago
It seems like this means artists would mostly be needed for creative art, e.g. spaceships, monsters, etc. That is, things that don't exist, so can't be photographed. But there are already commercial and free libraries of commonplace items like furniture, cars, and buildings. So isn't what I described pretty close to the same situation today already?
- taylorius 4y ago
  But spaceships and monsters do exist. There are thousands of them, just look on google images. An AI inverse rendering system, could just as easily ingest a ton of space sci-fi images, as real world photos.
4y ago

alleycat5000 4y ago

This is a nice tutorial on differentiable and inverse rendering (one of the Mitsuba authors also gives the tutorial): https://shuangz.com/courses/pbdr-course-sg20/ https://shuangz.com/courses/pbdr-course-sg20/ Inverse rendering is such a cool idea... solving for the inputs (3D models and materials) to a renderer such that they match real world inputs (photographs) by computing gradients automatically.

airstrike 4y ago

Couple this with VR glasses and you can relive your favorite memories in first person rather than through a photograph. I've been thinking about this for probably a decade now, and can't wait for it to become reality.

taylorius 4y ago
I'd be careful with that. My suspicion is that our memories would transform themselves into memories of the simulation, rather than the original event.
- airstrike 4y ago
  I’d watch that sci-fi film

ykl 4y ago

As with almost everything that comes out of Wenzel Jakob's lab at EPFL, the documentation for Mitsuba 3 is _incredible_. Check out this writeup of how polarization is handled in Mitsuba 3 (complete with animated diagrams): https://mitsuba.readthedocs.io/en/latest/src/key_topics/polarization.html#validation-against-measurements https://mitsuba.readthedocs.io/en/latest/src/key_topics/pola... That writeup is PhD Thesis material in terms of scope and detail, but the student that wrote it, Tizian Zeltner, actually wrote his PhD on another topic entirely. This is just the level of stuff that routinely comes out of that lab.

throwaway17_17 4y ago
Agree on the quality. I have several documents from the Mitsuba 2 era that are of the absolute highest quality. And for anyone wondering, this counts both the quality of content and the actual success of communicating complex and varied ideas in commendable fashion.
bayindirh 4y ago
This is a result of a group in harmony with people who are in love with what they are doing. When you like the subject, and pour what you have into it, this is what comes out. I'm sure this wasn't tiring for them. Make no mistake, this is a very high achievement, and is very rare in both private companies and research labs. It's a very hard point to reach if you don't like or care about what you are doing. Kudos to them.

AdamH12113 4y ago

What is a physically based renderer? It sounds like ray tracing that can be done in reverse on a final image?

cornstalks 4y ago
I’ve always seen “physically based” to mean the renderer does a physical simulation (photons bouncing, refracting, having wavelengths stretched, subsurface scattering, BSDF calculations, etc.). So much in computer graphics are just (clever) hacks upon hacks to get something that looks “good enough” but isn’t really simulating physics in any meaningful way (like SSAO, texture baking, bump mapping, etc.). These hacks are much, much faster than simulating the physical process of photons interacting with the world. I don’t know where “physically based” originated but my first introduction to it was pbrt, which I suspect popularized “physically based” naming. Differentiable rendering is the name of going from the final image and “reverse rendering” it to reconstruct a 3D model of it.
- fluxem 4y ago
  Some clarification regarding the differentiable rendering vs inverse rendering. Differentiable rendering is what it says: differentiating the rendering process. Imagine that x is the scene, then f(x) is the function that renders the image. Then, Differentiable rendering is simply taking the derivatives: f'(x). Inverse rendering is a process of finding scene parameters x, such that f(x) produces the given image y. This is often achieved by using differentiable rendering together with an optimization algorithm (like SGD or Adam). However, due to the nature of rendering, it's easy to get stuck in a local minimum. Therefore, even a perfect differentiable rendering engine is not sufficient for the inverse rendering.
- ohgodplsno 4y ago
  Mind you, physically based renderers also use a bunch of clever hacks upon hacks to be able to run in realtime. Nobody is simulating photons bouncing in the nanostructure of a material. We just have formulas thay say "for this roughness and metallicness, your material will look like that and it'll be good enough". We still abuse the hell out of UV maps to fake things, still include ambient occlusion (as a stylistic choice most of these days). It's a different workflow that gives you consistency once you know what you're doing, as opposed to the days of Phong shading.

nl 4y ago

The inverse rendering (Radiance Field Reconstruction (NeRF-like)) is very impressive. It can reconstruct physical models from multiple photos. See the final results at https://mitsuba.readthedocs.io/en/latest/src/inverse_rendering/radiance_field_reconstruction.html#Results https://mitsuba.readthedocs.io/en/latest/src/inverse_renderi... (scroll down a bit to see the reconstructed image vs the original)

junon 4y ago

This is missing examples of the outputs. I don't see any actual renders of note. Am I missing something?

moonchrome 4y ago
From what I see the point of this renderer is that it can invert the rendering to come up with a 3d scene that matches the input render image, so I guess outputs aren't as important ?

deleted 4y ago

[deleted]

andybak 4y ago

IMHO it's "differentiable" that's the most interesting part of this. There's tons of physically based renderers. It's odd that that word doesn't appear on http://www.mitsuba-renderer.org/ http://www.mitsuba-renderer.org/ - unless "retargetable" carries a similar meaning?

doctorhandshake 4y ago
This was my reaction - I clicked for PBR and was surprised to find the much more interesting lede buried

naillo 4y ago

Just waiting for someone to apply CLIP guidence onto this (so I don't have to).

majoe 4y ago

As someone coming from the physics side and not so much from the Computer graphics side of physically based rendering, I have high hopes for differentiable rendering as a general tool to extract knowledge from light-based measurements. Most differentiable renderers stop at the object surface, but differentiable volumetric light transport is what would allow us to use the renderer to optimise a measurement device or for medical diagnostics or just to estimate the optical properties of a material (that is absorption and scattering coefficent and scattering phase function), which is a pretty hard problem in itself. From a quick look, Mitsuba 3 seems to have everything in place to be used in such a setting and I'm really excited to see some use cases outside of Computer graphics.

picsao 4y ago