4 ms·

It is baffling just how well RGB rendering works despite it being such a huge hack; many real-world scenes are practically indistinguishable between rgb and spe

by zokier 3y ago

It is baffling just how well RGB rendering works despite it being such a huge hack; many real-world scenes are practically indistinguishable between rgb and spectral rendering. But still I find the physically based rendering pretty disingenuous considering the huge gap still remaining between real-world and standard pbr although I accept that it is just a domain-specific term that shouldn't be taken too literally. Just to spark discussion, here is one example of research aiming to reduce that gap and bring more physicality: https://ssteinberg.xyz/2023/03/27/rtplt/ https://ssteinberg.xyz/2023/03/27/rtplt/

esperent 3y ago
> I find the physically based rendering pretty disingenuous considering the huge gap still remaining between real-world and standard pbr > ... I accept that it is just a domain-specific term that shouldn't be taken too literally. Actually it is supposed to be taken literally. That's why it's physically based rendering and not physically correct rendering. The goal is to create a system based on real world physics to get as close as possible within the constraints of current computer hardware. I believe this article gets it wrong: > The primary goal of physically-based rendering (PBR) is to create a simulation that accurately reproduces the imaging process of electro-magnetic spectrum radiation incident to an observer. This simulation should be indistinguishable from reality for a similar observer. I don't think the stated goal of PBR is to be indistinguishable from reality for a similar observer. At least, I have never seen this stated anywhere. However, it's worth noting that PBR is a loaded term and includes both physically based lighting, and physically based shading (surface properties). If we are purely talking about lighting, that is, modelling the way light bounces in a physical way, it might be possible to get an exact match to physical reality. Light bounces in a relatively simple way. The problem is that surfaces are not simple. They absorb, reflect, diffract, scatter, and so on. So even if the equations modelling the reaction of light are 100% physically correct, it's an entirely different story to model a complex surface like skin in a physically correct way. This article is about light, so it's talking about physically based lighting. In which case the phrase this simulation should be indistinguishable from reality for a similar observer is more reasonable, but I think it should still be qualified for clarity.
- magicalhippo 3y ago
  Indeed. The book[1] on the subject states that the point is that, unlike the rendering methods that came before, the math is based on physics and physical principles. Of course one has to simplify and approximate, but the approximations should be grounded in physics and obey the underlying physical principles. For example, in the early days people created reflection models without considering energy reciprocity for example. Well that's a physical principle that's very well established in physics[2] so any physically based renderer should only have reflection models that obey that principle. [1]: https://www.pbrt.org/ https://www.pbrt.org/ [2]: https://en.wikipedia.org/wiki/Helmholtz_reciprocity https://en.wikipedia.org/wiki/Helmholtz_reciprocity
  - zokier 3y ago
    In a sense all (non-stylized) rendering is physics-based in the sense that all that physics is a model, and indeed the field of physics has different models of differing accuracy/precision/applicability. So even pre-PBR could be called physical in the sense that it represents some sort model of light, the rest is more of question on degree of accuracy and applicability. What I'm aiming at is that the line drawn for PBR is drawn in sand and feels somewhat arbitrary.
    - magicalhippo 3y ago
      It's not that arbitrary to me. The earlier work was sometimes physically-motivated, and sometimes even based on physics. For example using Lambertian reflection[1] for diffuse surfaces. But the key, in my view, to PBR is that the whole pipeline is based on physical principles and math, while the old renderers were not. So previous renderers would not bother properly normalizing the Lambertian coeffients for example, leading to violation of energy conservation. [1]: https://en.wikipedia.org/wiki/Lambertian_reflectance https://en.wikipedia.org/wiki/Lambertian_reflectance
- kelsolaar 3y ago
  OP author here! > I don't think the stated goal of PBR is to be indistinguishable from reality for a similar observer. At least, I have never seen this stated anywhere. I’m, admittedly, highly biased as I work for Wētā FX, a high-end visual effects vendor. We have been developing for over a decade a spectral renderer, i.e. Manuka [1], whose design principles and philosophy are aimed at recreating how a motion picture camera, its lenses, filters, and even colour science, image the real world so that we can integrate virtual cg elements into plates as if they were part of the original photography. This hopefully gives some context to your quote. Whether we are successful at this is a different topic but this is a core business offering we are hired for. - [1] https://www.wetafx.co.nz/research-and-tech/technology/manuka/ https://www.wetafx.co.nz/research-and-tech/technology/manuka...
magicalhippo 3y ago
Is it that baffling? After all, the core principle is that instead of representing a continuous spectra, you project the spectra to a few basis functions[1], so you can later reconstruct an approximation of the original spectra. Our eyes have three different cone cells with different responses[2], so the fact that three basis functions[3] that roughly align with the cone cells work well as an approximation isn't that surprising to me. That said, way back when I was doing rendering I was toying with the idea of trying to find a optimal basis functions for rendering, like if scene-specific ones could improve things. [1]: https://www.psych.mcgill.ca/misc/fda/ex-basis-a1.html https://www.psych.mcgill.ca/misc/fda/ex-basis-a1.html [2]: https://en.wikipedia.org/wiki/Cone_cell https://en.wikipedia.org/wiki/Cone_cell [3]: https://en.wikipedia.org/wiki/CIE_1931_color_space https://en.wikipedia.org/wiki/CIE_1931_color_space
- zokier 3y ago
  For basic direct lighting its not all that weird, but for any indirect lighting or translucency etc it is pretty whacky. I guess it basically boils down to most real-world scenes having both fairly broadband light-sources and typical material spectrum being also relatively broadband, and in general having smooth spectrums. I'd imagine if real-world for some reason predominantly consisted of super spiky or notchy spectrums then the RGB rendering would struggle more. Or maybe not.
  - magicalhippo 3y ago
    Yes, indeed. If you have narrow-band light sources, say a mercury vapor lamp[1], and some material that only reflects a different narrow-band spectrum, then you should get very little reflection while with an RGB renderer you'll get some reflection. However as you point out, very few materials and lights are like this, hence why the approximation works well. [1]: https://en.wikipedia.org/wiki/Mercury-vapor_lamp https://en.wikipedia.org/wiki/Mercury-vapor_lamp
    - zokier 3y ago
      yes, I suppose the 'baffling' part is partially from that theoretically there are infinitely (or nearly so) many different spectrums that produce some one specific color perception. So sort of probabilistically it could be seen as lucky that the spectrums of lights and materials align in a way that they do in real world. I wonder what are the pathological cases here. I'm imagining some setup where an object could appear variably green, yellow, or red; this could be maybe accomplished by different mixes of different red/green lasers and maybe white light. The paint might be bigger problem :)

by zokier 3y ago

esperent 3y ago

> I find the physically based rendering pretty disingenuous considering the huge gap still remaining between real-world and standard pbr > ... I accept that it is just a domain-specific term that shouldn't be taken too literally. Actually it is supposed to be taken literally. That's why it's physically based rendering and not physically correct rendering. The goal is to create a system based on real world physics to get as close as possible within the constraints of current computer hardware. I believe this article gets it wrong: > The primary goal of physically-based rendering (PBR) is to create a simulation that accurately reproduces the imaging process of electro-magnetic spectrum radiation incident to an observer. This simulation should be indistinguishable from reality for a similar observer. I don't think the stated goal of PBR is to be indistinguishable from reality for a similar observer. At least, I have never seen this stated anywhere. However, it's worth noting that PBR is a loaded term and includes both physically based lighting, and physically based shading (surface properties). If we are purely talking about lighting, that is, modelling the way light bounces in a physical way, it might be possible to get an exact match to physical reality. Light bounces in a relatively simple way. The problem is that surfaces are not simple. They absorb, reflect, diffract, scatter, and so on. So even if the equations modelling the reaction of light are 100% physically correct, it's an entirely different story to model a complex surface like skin in a physically correct way. This article is about light, so it's talking about physically based lighting. In which case the phrase this simulation should be indistinguishable from reality for a similar observer is more reasonable, but I think it should still be qualified for clarity.

magicalhippo 3y ago
Indeed. The book[1] on the subject states that the point is that, unlike the rendering methods that came before, the math is based on physics and physical principles. Of course one has to simplify and approximate, but the approximations should be grounded in physics and obey the underlying physical principles. For example, in the early days people created reflection models without considering energy reciprocity for example. Well that's a physical principle that's very well established in physics[2] so any physically based renderer should only have reflection models that obey that principle. [1]: https://www.pbrt.org/ https://www.pbrt.org/ [2]: https://en.wikipedia.org/wiki/Helmholtz_reciprocity https://en.wikipedia.org/wiki/Helmholtz_reciprocity
- zokier 3y ago
  In a sense all (non-stylized) rendering is physics-based in the sense that all that physics is a model, and indeed the field of physics has different models of differing accuracy/precision/applicability. So even pre-PBR could be called physical in the sense that it represents some sort model of light, the rest is more of question on degree of accuracy and applicability. What I'm aiming at is that the line drawn for PBR is drawn in sand and feels somewhat arbitrary.
  - magicalhippo 3y ago
    It's not that arbitrary to me. The earlier work was sometimes physically-motivated, and sometimes even based on physics. For example using Lambertian reflection[1] for diffuse surfaces. But the key, in my view, to PBR is that the whole pipeline is based on physical principles and math, while the old renderers were not. So previous renderers would not bother properly normalizing the Lambertian coeffients for example, leading to violation of energy conservation. [1]: https://en.wikipedia.org/wiki/Lambertian_reflectance https://en.wikipedia.org/wiki/Lambertian_reflectance
kelsolaar 3y ago
OP author here! > I don't think the stated goal of PBR is to be indistinguishable from reality for a similar observer. At least, I have never seen this stated anywhere. I’m, admittedly, highly biased as I work for Wētā FX, a high-end visual effects vendor. We have been developing for over a decade a spectral renderer, i.e. Manuka [1], whose design principles and philosophy are aimed at recreating how a motion picture camera, its lenses, filters, and even colour science, image the real world so that we can integrate virtual cg elements into plates as if they were part of the original photography. This hopefully gives some context to your quote. Whether we are successful at this is a different topic but this is a core business offering we are hired for. - [1] https://www.wetafx.co.nz/research-and-tech/technology/manuka/ https://www.wetafx.co.nz/research-and-tech/technology/manuka...

magicalhippo 3y ago

Is it that baffling? After all, the core principle is that instead of representing a continuous spectra, you project the spectra to a few basis functions[1], so you can later reconstruct an approximation of the original spectra. Our eyes have three different cone cells with different responses[2], so the fact that three basis functions[3] that roughly align with the cone cells work well as an approximation isn't that surprising to me. That said, way back when I was doing rendering I was toying with the idea of trying to find a optimal basis functions for rendering, like if scene-specific ones could improve things. [1]: https://www.psych.mcgill.ca/misc/fda/ex-basis-a1.html https://www.psych.mcgill.ca/misc/fda/ex-basis-a1.html [2]: https://en.wikipedia.org/wiki/Cone_cell https://en.wikipedia.org/wiki/Cone_cell [3]: https://en.wikipedia.org/wiki/CIE_1931_color_space https://en.wikipedia.org/wiki/CIE_1931_color_space

zokier 3y ago
For basic direct lighting its not all that weird, but for any indirect lighting or translucency etc it is pretty whacky. I guess it basically boils down to most real-world scenes having both fairly broadband light-sources and typical material spectrum being also relatively broadband, and in general having smooth spectrums. I'd imagine if real-world for some reason predominantly consisted of super spiky or notchy spectrums then the RGB rendering would struggle more. Or maybe not.
- magicalhippo 3y ago
  Yes, indeed. If you have narrow-band light sources, say a mercury vapor lamp[1], and some material that only reflects a different narrow-band spectrum, then you should get very little reflection while with an RGB renderer you'll get some reflection. However as you point out, very few materials and lights are like this, hence why the approximation works well. [1]: https://en.wikipedia.org/wiki/Mercury-vapor_lamp https://en.wikipedia.org/wiki/Mercury-vapor_lamp
  - zokier 3y ago
    yes, I suppose the 'baffling' part is partially from that theoretically there are infinitely (or nearly so) many different spectrums that produce some one specific color perception. So sort of probabilistically it could be seen as lucky that the spectrums of lights and materials align in a way that they do in real world. I wonder what are the pathological cases here. I'm imagining some setup where an object could appear variably green, yellow, or red; this could be maybe accomplished by different mixes of different red/green lasers and maybe white light. The paint might be bigger problem :)