4 ms·

Combining Classic and Modern: A New Approach to Camera Simulation

18 points by SerCe 5y ago

dusted 5y ago
absolutely hate the filters, no, don't throw microchips trying to make bad camera "look" good, make a good camera. Especially the DOF stuff, we're slowly conditioning people to accept this as truth, while it's so flawed, and always will be. Same happens with compression, especially things like grained textures, are hardcore compressed on even highend phones, giving the impression of detail, where none is captured.
PBnFlash 5y ago
Pixel perfect mono depth generation seems an extremely hard problem. Judging by the extreme colors of the example objects this may not be super clean in real cases.
- smat 5y ago
  It is actually impossible as a lens with large aperture gathers rays from behind close objects. So it is a problem with incomplete information.
- notorious_IPD 5y ago
  [article author here] As smat pointed out - absolutely, it's difficult! In the ideal case we'd have a proper 3D reconstruction of the scene; something like a light-field representation that permitted enough lateral movement in the camera extrinsics to adequately sample inside a synthetic aperture. This theoretical development is important, but it doesn't block getting to a point that is good enough for consumer use cases. How it goes in practice - even for high-end VFX shops - is that the result looks and feels right. As an example - the depth maps used in high-end VFX (especially for stereo conversion) is sometimes hilariously and incredibly inaccurate (e.g. having depth on a foreground spaceship but also depth on a planet thousands of kilometers away). Accuracy improvements in absolute modeling in the real world may not translate in a visible, meaningful way when put through a simulation that acts upon it; instead we focused (ha!) on improving the light modeling process from a generic 'textbook' example of kernel-based simulation to one that more closely matches how light moves through an actual physical camera. And yeah, the colours are bit punchy but that's my fault - my photography aesthetic unfortunately is tuned towards ludicrous colour, and I probably need an intervention for someone to take my cursor away from the vibrance slider :P
starky 5y ago
This approach still doesn't fix what I feel is the worst part of "portrait" mode images, which is that the transition to the out of focus areas looks incredibly unrealistic. If your background is so far out of focus, then the entire subject shouldn't be sharp, areas outside of the focus plane need to be slightly out of focus. There is also a strong uncanny valley effect because the Bokeh doesn't have any of the character that comes from a real lens. Personally, I think we are quite a long time away from being able to computationally blur photo backgrounds acceptably well.
- notorious_IPD 5y ago
  [article author here] The issues you're seeing here comes down to quantization errors in the depth maps. The effective range of a depth map for most human-scale scenes is about 20m; what you then have is 8 bits (i.e. 256 levels) to try to encode everything from right up close to the camera out to 20m. Even if there's in interpretation function on top of this to distribute the depth more meaningfully (e.g. storing 1/depth to prioritise detail on closer objects) you still, in the end, have banding. When we developed this approach we validated it against existing techniques and I'd still like to think we have the smoothest, least prone-to-banding technique for simulated defocus out there; the maths was tweaked to specifically address this. Of course, there are limits - aliasing can only be mitigated so much. But that's why we're still working on it and this kind of feedback is enormously useful (thank-you!) in helping us move closer to a virtual camera that feels more like a real one!
poniko 5y ago
yea still looks really bad .. what was the new approach here?
- notorious_IPD 5y ago
  [article author here] It's in a comment elsewhere here but: the new approach is to use a camera simulation that more closely models how light moves through a real camera rather than the textbook circular kernel approach that has stuck since Levoy's early experiments in this space decades ago. We optimised the approach to more closely resemble what happens in and with an actual camera. As to whether it looks good or not - that's entirely subjective. Try it out; it might be that as an individual you prefer to dial down the defocus etc (I'll admit - the examples are a bit exaggerated).
  - poniko 5y ago
    Thanks for the response, the main issue I have is that the blur is more of a tilt-shift then an bokeh. I don't understand why there is blur in front of the subject in many of the examples, also the blur is way dialed up and the edges becomes to sharp. It also seem to be very aggressive on the depth it starts to blur on, the ear of the dog is blurred as well as the foot of the people racing. My example https://imgur.com/a/pitVg0K https://imgur.com/a/pitVg0K .. weird slimmer of tiltshift-focus on the ground, "one pixel" non blurred border on one side of the legs, harsh edge on the other side and blur bleeding into the hat part .. nothing I would use. Disclosure i do work with pixlr.com.
    - notorious_IPD 5y ago
      Generally speaking: I do agree that the degree of blur in the examples is a bit too obnoxious. It was exaggerated for effect, and dialling it back a bit would be a prudent move. As far as the tilt-shift look goes - being precise I think you're thinking of tilt-shift miniaturisation. In cases where you're shooting with really shallow depth of field - like wide open on a portrait lens, or with a large format camera or yes, a tilt-shift lens in the anti-Scheimpflug mode - you absolutely can have foreground elements blurred out. I wrote a bit about that here: https://bhautik.org/tilt-shift-a-diy-guide/ https://bhautik.org/tilt-shift-a-diy-guide/ I'd love to look at the source image for your example; some of this is explainable, some not so much and I'd like to understand why. The sharp aliasing around the edges is weird, and that looks like a legit bug, as well as the blur leakage (we're trying to prevent that!). Also, it looks like the depth was incorrectly estimated for the feet so it's lumped them into the foreground and blurred those too. An easy fix is to suggest trying it with less blur, but there's other stuff going on. We'll look into it :)
deleted 5y ago
[deleted]

18 points by SerCe 5y ago

absolutely hate the filters, no, don't throw microchips trying to make bad camera "look" good, make a good camera. Especially the DOF stuff, we're slowly conditioning people to accept this as truth, while it's so flawed, and always will be. Same happens with compression, especially things like grained textures, are hardcore compressed on even highend phones, giving the impression of detail, where none is captured.

PBnFlash 5y ago

Pixel perfect mono depth generation seems an extremely hard problem. Judging by the extreme colors of the example objects this may not be super clean in real cases.

smat 5y ago
It is actually impossible as a lens with large aperture gathers rays from behind close objects. So it is a problem with incomplete information.
notorious_IPD 5y ago
[article author here] As smat pointed out - absolutely, it's difficult! In the ideal case we'd have a proper 3D reconstruction of the scene; something like a light-field representation that permitted enough lateral movement in the camera extrinsics to adequately sample inside a synthetic aperture. This theoretical development is important, but it doesn't block getting to a point that is good enough for consumer use cases. How it goes in practice - even for high-end VFX shops - is that the result looks and feels right. As an example - the depth maps used in high-end VFX (especially for stereo conversion) is sometimes hilariously and incredibly inaccurate (e.g. having depth on a foreground spaceship but also depth on a planet thousands of kilometers away). Accuracy improvements in absolute modeling in the real world may not translate in a visible, meaningful way when put through a simulation that acts upon it; instead we focused (ha!) on improving the light modeling process from a generic 'textbook' example of kernel-based simulation to one that more closely matches how light moves through an actual physical camera. And yeah, the colours are bit punchy but that's my fault - my photography aesthetic unfortunately is tuned towards ludicrous colour, and I probably need an intervention for someone to take my cursor away from the vibrance slider :P

starky 5y ago

This approach still doesn't fix what I feel is the worst part of "portrait" mode images, which is that the transition to the out of focus areas looks incredibly unrealistic. If your background is so far out of focus, then the entire subject shouldn't be sharp, areas outside of the focus plane need to be slightly out of focus. There is also a strong uncanny valley effect because the Bokeh doesn't have any of the character that comes from a real lens. Personally, I think we are quite a long time away from being able to computationally blur photo backgrounds acceptably well.

notorious_IPD 5y ago
[article author here] The issues you're seeing here comes down to quantization errors in the depth maps. The effective range of a depth map for most human-scale scenes is about 20m; what you then have is 8 bits (i.e. 256 levels) to try to encode everything from right up close to the camera out to 20m. Even if there's in interpretation function on top of this to distribute the depth more meaningfully (e.g. storing 1/depth to prioritise detail on closer objects) you still, in the end, have banding. When we developed this approach we validated it against existing techniques and I'd still like to think we have the smoothest, least prone-to-banding technique for simulated defocus out there; the maths was tweaked to specifically address this. Of course, there are limits - aliasing can only be mitigated so much. But that's why we're still working on it and this kind of feedback is enormously useful (thank-you!) in helping us move closer to a virtual camera that feels more like a real one!

poniko 5y ago

yea still looks really bad .. what was the new approach here?

notorious_IPD 5y ago
[article author here] It's in a comment elsewhere here but: the new approach is to use a camera simulation that more closely models how light moves through a real camera rather than the textbook circular kernel approach that has stuck since Levoy's early experiments in this space decades ago. We optimised the approach to more closely resemble what happens in and with an actual camera. As to whether it looks good or not - that's entirely subjective. Try it out; it might be that as an individual you prefer to dial down the defocus etc (I'll admit - the examples are a bit exaggerated).
- poniko 5y ago
  Thanks for the response, the main issue I have is that the blur is more of a tilt-shift then an bokeh. I don't understand why there is blur in front of the subject in many of the examples, also the blur is way dialed up and the edges becomes to sharp. It also seem to be very aggressive on the depth it starts to blur on, the ear of the dog is blurred as well as the foot of the people racing. My example https://imgur.com/a/pitVg0K https://imgur.com/a/pitVg0K .. weird slimmer of tiltshift-focus on the ground, "one pixel" non blurred border on one side of the legs, harsh edge on the other side and blur bleeding into the hat part .. nothing I would use. Disclosure i do work with pixlr.com.
  - notorious_IPD 5y ago
    Generally speaking: I do agree that the degree of blur in the examples is a bit too obnoxious. It was exaggerated for effect, and dialling it back a bit would be a prudent move. As far as the tilt-shift look goes - being precise I think you're thinking of tilt-shift miniaturisation. In cases where you're shooting with really shallow depth of field - like wide open on a portrait lens, or with a large format camera or yes, a tilt-shift lens in the anti-Scheimpflug mode - you absolutely can have foreground elements blurred out. I wrote a bit about that here: https://bhautik.org/tilt-shift-a-diy-guide/ https://bhautik.org/tilt-shift-a-diy-guide/ I'd love to look at the source image for your example; some of this is explainable, some not so much and I'd like to understand why. The sharp aliasing around the edges is weird, and that looks like a legit bug, as well as the blur leakage (we're trying to prevent that!). Also, it looks like the depth was incorrectly estimated for the feet so it's lumped them into the foreground and blurred those too. An easy fix is to suggest trying it with less blur, but there's other stuff going on. We'll look into it :)

deleted 5y ago

[deleted]