6 ms·

The Apparent Simplicity of RGB Rendering

63 points by kelsolaar 3y ago

deleted 3y ago
[deleted]
alhirzel 3y ago
Those interested in this article may be interested in experimenting with the Mitsuba renderer [1]. This is one of the best open-source spectral renderers, and it includes support for RGB and spectral modes (in addition to propagating things like polarization), all in one unified rendering pipeline with CUDA and non-CUDA backends. [1]: https://www.mitsuba-renderer.org/ https://www.mitsuba-renderer.org/
- josephg 3y ago
  It seems strange that the website of a renderer wouldn’t have any rendered images. What does it look like? Do spectral renderers achieve a better result than traditional rendering methods? Show me! I want to see the difference!
  - zokier 3y ago
    https://mitsuba.readthedocs.io/en/stable/src/gallery.html https://mitsuba.readthedocs.io/en/stable/src/gallery.html But Mitsuba is not really complete system to make pretty pictures, it is more of a framework for researchers to prototype algorithms etc. So while many state of the art methods have been implemented on top of Mitsuba, not nearly all of them have been merged to the mainline. If you want comparisons then you probably need to look up specific papers to see comparisons for the specific problems they solve.
  - incrudible 3y ago
    Better is subjective, but the whole point of spectral rendering is to be more objective. The visual difference is minor and its relevance to film/games is put into question by the artistic post processing that inevitably distorts the final result. There are some effects that “just work” with spectral rendering though, so you might save some hackery there.
  - alhirzel 3y ago
    Another stand-out feature of Mitsuba is its focus on differentiability. Wenzel Jakob is an absolute rockstar in computer graphics, and his group creates informative demo and explanatory videos for their areas of work. These include: Differentiable rendering: http://rgl.epfl.ch/publications/Jakob2022DrJit http://rgl.epfl.ch/publications/Jakob2022DrJit Volumetric rendering: http://rgl.epfl.ch/publications/NimierDavid2022Unbiased http://rgl.epfl.ch/publications/NimierDavid2022Unbiased Optimization of light path choice via AD: http://rgl.epfl.ch/publications/Vicini2021PathReplay http://rgl.epfl.ch/publications/Vicini2021PathReplay
- dtf 3y ago
  Also worth mentioning Manuka, which is Weta's production spectral renderer. There's a nice overview of the architecture here, which includes a section on their motivations for, and implementation of, spectral light transport: https://jo.dreggn.org/home/2018_manuka.pdf https://jo.dreggn.org/home/2018_manuka.pdf
  - kelsolaar 3y ago
    OP author here. Thanks for mentioning it, I happen to work at Wētā FX and we are obviously very biased toward spectral rendering and the problems it solves. I often hear people saying that spectral rendering is complicated but it is actually the opposite, mostly everything becomes simplified with it. It, gives, at the very least an elegant first principle based framework to work from.
- wdfx 3y ago
  Another one like this is (was? not sure if it's maintained any more) Lux Render: https://luxcorerender.org/ https://luxcorerender.org/ I played my part in this back in the 2010s maintaining the blender integration, fun times :) But both the renderer and the integrations got pretty much entirely re-written in the move to GPU compute shortly after that time.
zokier 3y ago
It is baffling just how well RGB rendering works despite it being such a huge hack; many real-world scenes are practically indistinguishable between rgb and spectral rendering. But still I find the physically based rendering pretty disingenuous considering the huge gap still remaining between real-world and standard pbr although I accept that it is just a domain-specific term that shouldn't be taken too literally. Just to spark discussion, here is one example of research aiming to reduce that gap and bring more physicality: https://ssteinberg.xyz/2023/03/27/rtplt/ https://ssteinberg.xyz/2023/03/27/rtplt/
- esperent 3y ago
  > I find the physically based rendering pretty disingenuous considering the huge gap still remaining between real-world and standard pbr > ... I accept that it is just a domain-specific term that shouldn't be taken too literally. Actually it is supposed to be taken literally. That's why it's physically based rendering and not physically correct rendering. The goal is to create a system based on real world physics to get as close as possible within the constraints of current computer hardware. I believe this article gets it wrong: > The primary goal of physically-based rendering (PBR) is to create a simulation that accurately reproduces the imaging process of electro-magnetic spectrum radiation incident to an observer. This simulation should be indistinguishable from reality for a similar observer. I don't think the stated goal of PBR is to be indistinguishable from reality for a similar observer. At least, I have never seen this stated anywhere. However, it's worth noting that PBR is a loaded term and includes both physically based lighting, and physically based shading (surface properties). If we are purely talking about lighting, that is, modelling the way light bounces in a physical way, it might be possible to get an exact match to physical reality. Light bounces in a relatively simple way. The problem is that surfaces are not simple. They absorb, reflect, diffract, scatter, and so on. So even if the equations modelling the reaction of light are 100% physically correct, it's an entirely different story to model a complex surface like skin in a physically correct way. This article is about light, so it's talking about physically based lighting. In which case the phrase this simulation should be indistinguishable from reality for a similar observer is more reasonable, but I think it should still be qualified for clarity.
meindnoch 3y ago
Tristimulus values (X, Y, Z) are the dot product of the incoming spectral radiance (L) and the spectral sensitivity curves (x, y, z) of the three cone cell types: X = dot(L, x) Y = dot(L, y) Z = dot(L, z) The spectrum of the light reflected from a surface (Lᵣ) is the pointwise product of the incoming spectral radiance (L) and the spectral reflectance curve of the surface (R): Lᵣ = L * R What RGB rendering does, is it replaces this equation (for x, y, z): X = dot(Lᵣ, x) = dot(L * R, x) (eq I) with this (for x, y, z): X = dot(L, x) * dot(R, x) (eq II) Which are not equal in the general case. On the other hand, they are equal up to a scaling factor if L is a constant function, since in that case: dot(L * R, x) = C * dot(R, x) (from eq I) dot(L, x) * dot(R, x) = D * dot(R, x) (from eq II) where C and D are scaling factors that depend solely on L. The interpretation of this is that as long as the light source has a sufficiently flat spectrum (like sunlight!), the difference between spectral rendering and RGB rendering will be negligible.
- david-gpu 3y ago
  > as long as the light source has a sufficiently flat spectrum (like sunlight!), the difference between spectral rendering and RGB rendering will be negligible. (Great comment, nitpick ahead) ...to the human eye, that is. For industrial applications spectral rendering can still be useful.
- kevingadd 3y ago
  one caveat is that (AFAIK) if you are doing optical simulations in your rendering, like prisms or glass, the spectral rendering would produce more accurate caustics/reflections/refractions. But to be fair, most realtime rendering is not doing that. I could imagine it being important for stuff like CG films.
- kelsolaar 3y ago
  OP author here. This is correct, however, not everything is lit by sunlight, especially onset in the motion picture industry, where solid state lighting and its narrow band irradiance sources, e.g. led lights or led walls, are prevalent. Those emission sources induce a lot of unexpected appearance problems, e.g. metameric failures that RGB rendering cannot possibly solve. Our spectral renderer, i.e. Manuka [1], was specifically designed to address those issues whilst imaging with a virtual motion picture camera. One recent example I have in mind is The Batman (2022): There are sequences/shots lit with high-pressure sodium lights (HPS), those have a flat spectrum with spikes between 500 and 600nm, the appearance of objects lit under them is very specific and can only be accurately reproduced with a spectral renderer such as Manuka, Mitsuba [2] or Art [3]. - [1] https://www.wetafx.co.nz/research-and-tech/technology/manuka/ https://www.wetafx.co.nz/research-and-tech/technology/manuka... - [2] https://www.mitsuba-renderer.org/ https://www.mitsuba-renderer.org/ - [3] https://cgg.mff.cuni.cz/ART/about/ https://cgg.mff.cuni.cz/ART/about/
lsaferite 3y ago
I've been working with a hyperspectral camera (350-1000nm,4nm,12b,410x410,8Hz) lately and this article was quite interesting through that specific lens. I'm pretty much in love with hyperspectral imaging now and it's really opened my eyes to a whole new world of information and technology. Not super relevant, just thought I'd share. Also, this camera has really made me want a real-life tricorder.
- alhirzel 3y ago
  You might enjoy the ECOSTRESS Spectral Library [1]. It has downloadable reflectance spectra overlapping with your sensor for a variety of materials. These spectra are collected in highly controlled lab conditions for pure material samples. Could be useful for calibration or making your tricorder ;) I do consulting work in this area, in case you've ever got questions or could use a hand developing something. [1]: https://speclib.jpl.nasa.gov/ https://speclib.jpl.nasa.gov/
  - lsaferite 3y ago
    Awesome, thanks for the information! I'll certainly keep you in mind if I need/want some help in the future.

63 points by kelsolaar 3y ago

deleted 3y ago

[deleted]

Those interested in this article may be interested in experimenting with the Mitsuba renderer [1]. This is one of the best open-source spectral renderers, and it includes support for RGB and spectral modes (in addition to propagating things like polarization), all in one unified rendering pipeline with CUDA and non-CUDA backends. [1]: https://www.mitsuba-renderer.org/ https://www.mitsuba-renderer.org/

josephg 3y ago
It seems strange that the website of a renderer wouldn’t have any rendered images. What does it look like? Do spectral renderers achieve a better result than traditional rendering methods? Show me! I want to see the difference!
- zokier 3y ago
  https://mitsuba.readthedocs.io/en/stable/src/gallery.html https://mitsuba.readthedocs.io/en/stable/src/gallery.html But Mitsuba is not really complete system to make pretty pictures, it is more of a framework for researchers to prototype algorithms etc. So while many state of the art methods have been implemented on top of Mitsuba, not nearly all of them have been merged to the mainline. If you want comparisons then you probably need to look up specific papers to see comparisons for the specific problems they solve.
- incrudible 3y ago
  Better is subjective, but the whole point of spectral rendering is to be more objective. The visual difference is minor and its relevance to film/games is put into question by the artistic post processing that inevitably distorts the final result. There are some effects that “just work” with spectral rendering though, so you might save some hackery there.
- alhirzel 3y ago
  Another stand-out feature of Mitsuba is its focus on differentiability. Wenzel Jakob is an absolute rockstar in computer graphics, and his group creates informative demo and explanatory videos for their areas of work. These include: Differentiable rendering: http://rgl.epfl.ch/publications/Jakob2022DrJit http://rgl.epfl.ch/publications/Jakob2022DrJit Volumetric rendering: http://rgl.epfl.ch/publications/NimierDavid2022Unbiased http://rgl.epfl.ch/publications/NimierDavid2022Unbiased Optimization of light path choice via AD: http://rgl.epfl.ch/publications/Vicini2021PathReplay http://rgl.epfl.ch/publications/Vicini2021PathReplay
dtf 3y ago
Also worth mentioning Manuka, which is Weta's production spectral renderer. There's a nice overview of the architecture here, which includes a section on their motivations for, and implementation of, spectral light transport: https://jo.dreggn.org/home/2018_manuka.pdf https://jo.dreggn.org/home/2018_manuka.pdf
- kelsolaar 3y ago
  OP author here. Thanks for mentioning it, I happen to work at Wētā FX and we are obviously very biased toward spectral rendering and the problems it solves. I often hear people saying that spectral rendering is complicated but it is actually the opposite, mostly everything becomes simplified with it. It, gives, at the very least an elegant first principle based framework to work from.
wdfx 3y ago
Another one like this is (was? not sure if it's maintained any more) Lux Render: https://luxcorerender.org/ https://luxcorerender.org/ I played my part in this back in the 2010s maintaining the blender integration, fun times :) But both the renderer and the integrations got pretty much entirely re-written in the move to GPU compute shortly after that time.

zokier 3y ago

It is baffling just how well RGB rendering works despite it being such a huge hack; many real-world scenes are practically indistinguishable between rgb and spectral rendering. But still I find the physically based rendering pretty disingenuous considering the huge gap still remaining between real-world and standard pbr although I accept that it is just a domain-specific term that shouldn't be taken too literally. Just to spark discussion, here is one example of research aiming to reduce that gap and bring more physicality: https://ssteinberg.xyz/2023/03/27/rtplt/ https://ssteinberg.xyz/2023/03/27/rtplt/

esperent 3y ago
> I find the physically based rendering pretty disingenuous considering the huge gap still remaining between real-world and standard pbr > ... I accept that it is just a domain-specific term that shouldn't be taken too literally. Actually it is supposed to be taken literally. That's why it's physically based rendering and not physically correct rendering. The goal is to create a system based on real world physics to get as close as possible within the constraints of current computer hardware. I believe this article gets it wrong: > The primary goal of physically-based rendering (PBR) is to create a simulation that accurately reproduces the imaging process of electro-magnetic spectrum radiation incident to an observer. This simulation should be indistinguishable from reality for a similar observer. I don't think the stated goal of PBR is to be indistinguishable from reality for a similar observer. At least, I have never seen this stated anywhere. However, it's worth noting that PBR is a loaded term and includes both physically based lighting, and physically based shading (surface properties). If we are purely talking about lighting, that is, modelling the way light bounces in a physical way, it might be possible to get an exact match to physical reality. Light bounces in a relatively simple way. The problem is that surfaces are not simple. They absorb, reflect, diffract, scatter, and so on. So even if the equations modelling the reaction of light are 100% physically correct, it's an entirely different story to model a complex surface like skin in a physically correct way. This article is about light, so it's talking about physically based lighting. In which case the phrase this simulation should be indistinguishable from reality for a similar observer is more reasonable, but I think it should still be qualified for clarity.

meindnoch 3y ago

Tristimulus values (X, Y, Z) are the dot product of the incoming spectral radiance (L) and the spectral sensitivity curves (x, y, z) of the three cone cell types: X = dot(L, x) Y = dot(L, y) Z = dot(L, z) The spectrum of the light reflected from a surface (Lᵣ) is the pointwise product of the incoming spectral radiance (L) and the spectral reflectance curve of the surface (R): Lᵣ = L * R What RGB rendering does, is it replaces this equation (for x, y, z): X = dot(Lᵣ, x) = dot(L * R, x) (eq I) with this (for x, y, z): X = dot(L, x) * dot(R, x) (eq II) Which are not equal in the general case. On the other hand, they are equal up to a scaling factor if L is a constant function, since in that case: dot(L * R, x) = C * dot(R, x) (from eq I) dot(L, x) * dot(R, x) = D * dot(R, x) (from eq II) where C and D are scaling factors that depend solely on L. The interpretation of this is that as long as the light source has a sufficiently flat spectrum (like sunlight!), the difference between spectral rendering and RGB rendering will be negligible.

david-gpu 3y ago
> as long as the light source has a sufficiently flat spectrum (like sunlight!), the difference between spectral rendering and RGB rendering will be negligible. (Great comment, nitpick ahead) ...to the human eye, that is. For industrial applications spectral rendering can still be useful.
kevingadd 3y ago
one caveat is that (AFAIK) if you are doing optical simulations in your rendering, like prisms or glass, the spectral rendering would produce more accurate caustics/reflections/refractions. But to be fair, most realtime rendering is not doing that. I could imagine it being important for stuff like CG films.
kelsolaar 3y ago
OP author here. This is correct, however, not everything is lit by sunlight, especially onset in the motion picture industry, where solid state lighting and its narrow band irradiance sources, e.g. led lights or led walls, are prevalent. Those emission sources induce a lot of unexpected appearance problems, e.g. metameric failures that RGB rendering cannot possibly solve. Our spectral renderer, i.e. Manuka [1], was specifically designed to address those issues whilst imaging with a virtual motion picture camera. One recent example I have in mind is The Batman (2022): There are sequences/shots lit with high-pressure sodium lights (HPS), those have a flat spectrum with spikes between 500 and 600nm, the appearance of objects lit under them is very specific and can only be accurately reproduced with a spectral renderer such as Manuka, Mitsuba [2] or Art [3]. - [1] https://www.wetafx.co.nz/research-and-tech/technology/manuka/ https://www.wetafx.co.nz/research-and-tech/technology/manuka... - [2] https://www.mitsuba-renderer.org/ https://www.mitsuba-renderer.org/ - [3] https://cgg.mff.cuni.cz/ART/about/ https://cgg.mff.cuni.cz/ART/about/

lsaferite 3y ago

I've been working with a hyperspectral camera (350-1000nm,4nm,12b,410x410,8Hz) lately and this article was quite interesting through that specific lens. I'm pretty much in love with hyperspectral imaging now and it's really opened my eyes to a whole new world of information and technology. Not super relevant, just thought I'd share. Also, this camera has really made me want a real-life tricorder.

alhirzel 3y ago
You might enjoy the ECOSTRESS Spectral Library [1]. It has downloadable reflectance spectra overlapping with your sensor for a variety of materials. These spectra are collected in highly controlled lab conditions for pure material samples. Could be useful for calibration or making your tricorder ;) I do consulting work in this area, in case you've ever got questions or could use a hand developing something. [1]: https://speclib.jpl.nasa.gov/ https://speclib.jpl.nasa.gov/
- lsaferite 3y ago
  Awesome, thanks for the information! I'll certainly keep you in mind if I need/want some help in the future.