6 ms·

We have spectral renderers that simulate light transport extremely accurately and in fact are indistinguishable from a photograph (see http://www.graphics.corne

We have spectral renderers that simulate light transport extremely accurately and in fact are indistinguishable from a photograph (see http://www.graphics.cornell.edu/online/box/compare.html http://www.graphics.cornell.edu/online/box/compare.html) The problem isn’t solving light transport itself (the rendering equation has been known for years, and is asymptotically exactly solved using unbiased numerical integration techniques); it’s modeling the interaction of light with complex physical materials and doing so quickly that poses the challenge.

sgift 9y ago
> and doing so quickly Most important part. The best solution which arrives too late is completely useless.
sillysaurus3 9y ago
Note the important distinction: no one, anywhere, has ever created a real-time video simulation indistinguishable from the output of a real camcorder. Real-time video is significantly different than still-frames. It's why video compression uses completely different algorithms than jpeg compression, for example. Our visual systems are tuned to notice problems with real-time outputs, whereas a still frame is ambiguous. There are all kinds of transformations you can do to a perfect still frame where it still seems perfect afterwards: You can tweak the contrast, brightness, hue, throw a softpass filter on it, etc, and a human observer still won't notice a big difference. In other words, there are many ways to "cheat"! Even though we're doing operations that don't correspond to how nature behaves, a human observer is still fooled by them. Yet the moment that you string a bunch of still frames together into a video, that human observer will pounce on you immediately and call it out as fake. I'm not sure why motion makes such a massive difference, but it does. To put it another way, if it were possible to make a real-time video whose ground truth was identical to a camcorder, we'd see it in hollywood. But no one has been able to; Avatar was the best we could do. I think it should be possible to train a neural net to paint the scene, similar to a human painter. DaVinci didn't make diffuse/ambient/specular textures; he simply painted the scene according to how it would look in real life. And in real life, color changes are gradual. It's crucial that the colors change in exactly the right way, but the changes themselves are rarely discontinuous. That means a neural net might be able to catch on pretty quickly. Bots are already doing similar work: https://www.reddit.com/r/aww/comments/505zzr/colour_run/d71grax/?context=3 https://www.reddit.com/r/aww/comments/505zzr/colour_run/d71g... Before: https://i.imgur.com/EqMLNFo.jpg https://i.imgur.com/EqMLNFo.jpg After: https://i.imgur.com/omYmn7Q.jpg https://i.imgur.com/omYmn7Q.jpg If it's possible to clean a puppy, it might be possible to extrapolate how light should behave in the general case.
- eru 9y ago
  There are actually also quite a few transformations you can do to a video that would be noticeable in a still frame. In particular, I remember some video stabilization a few years ago that just warped the input images instead of building a proper 3d model. It worked in real time and looked great in the video---but in a still frame you could see that the angles were all slightly off.
- andybak 9y ago
  Don't forget that sometimes too much realism is a problem. The "soap opera" effect with high frame rate source material or displays that interpolate frames is commonly regarded as looking awful. However - it is more realistic but less appealing to our current aesthetic sensibilities. Photorealism is a great tool to have in your box but it's not the end goal in any creative medium.
- modeless 9y ago
  Hollywood is not optimizing for perfect realism. They are operating under many constraints such as budget (yes, even on Avatar they had budgets), schedule, and the need for directorial control (which often conflicts with realism). It would absolutely be possible to produce prerendered videos of mostly static scenes using today's technology that would be indistinguishable from reality, although the expense would be high.
  - sillysaurus3 9y ago
    It would absolutely be possible to produce prerendered videos of mostly static scenes using today's technology that would be indistinguishable from reality, although the expense would be high. And yet, no one has. :) It'd be a fun trophy to claim. My theory is that it's not merely a matter of computing power. We have massive horsepower now, but our algorithms are still wrong at a fundamental level. We still approach it by trying to simulate the underlying physics of light: the transport algorithms, radiance sampling, and so on. But between the RGB textures that have nothing to do with nature and the physics approximations that we're forced to settle with, something ends up lost in translation. No matter what we try, that human observer ends up looking at a final product that looks nothing like a real camcorder video. Back during ~iPhone 4 days, someone forged some pictures of the "next gen iPhone." It was a hoax, but a lot of people were fooled at the time. Remarkably, it was a computer rendering. The reason it looked real is because he took a picture of the rendering using a real camera, which muddied up the colors to the point that you couldn't tell it was a rendering. The idea here would be similar: analyze the actual pixels coming out of a real camcorder and try to mimic the nuances as closely as possible. In the best case, it wouldn't be a physics simulation at all: the program might be able to guess how the scene should look based on partial data and past experience.
PeachPlum 9y ago
They are nothing like indistinguishable and have the exact problems I would look for to see if it was CGI. * The lower edge on the foreground box, particularly the lower left corner. * The lower edges of the mirror box. * To a lesser extent, the bottom corners of the surrounding box. Once you spend time in rendering, you know where to look for flaws. If your example was animated, it would be obviously rendered.
- fenwick67 9y ago
  I'm pretty confident that if you asked laypeople to mark which was real and which was fake they couldn't reliably tell.

sgift 9y ago

> and doing so quickly Most important part. The best solution which arrives too late is completely useless.

sillysaurus3 9y ago

Note the important distinction: no one, anywhere, has ever created a real-time video simulation indistinguishable from the output of a real camcorder. Real-time video is significantly different than still-frames. It's why video compression uses completely different algorithms than jpeg compression, for example. Our visual systems are tuned to notice problems with real-time outputs, whereas a still frame is ambiguous. There are all kinds of transformations you can do to a perfect still frame where it still seems perfect afterwards: You can tweak the contrast, brightness, hue, throw a softpass filter on it, etc, and a human observer still won't notice a big difference. In other words, there are many ways to "cheat"! Even though we're doing operations that don't correspond to how nature behaves, a human observer is still fooled by them. Yet the moment that you string a bunch of still frames together into a video, that human observer will pounce on you immediately and call it out as fake. I'm not sure why motion makes such a massive difference, but it does. To put it another way, if it were possible to make a real-time video whose ground truth was identical to a camcorder, we'd see it in hollywood. But no one has been able to; Avatar was the best we could do. I think it should be possible to train a neural net to paint the scene, similar to a human painter. DaVinci didn't make diffuse/ambient/specular textures; he simply painted the scene according to how it would look in real life. And in real life, color changes are gradual. It's crucial that the colors change in exactly the right way, but the changes themselves are rarely discontinuous. That means a neural net might be able to catch on pretty quickly. Bots are already doing similar work: https://www.reddit.com/r/aww/comments/505zzr/colour_run/d71grax/?context=3 https://www.reddit.com/r/aww/comments/505zzr/colour_run/d71g... Before: https://i.imgur.com/EqMLNFo.jpg https://i.imgur.com/EqMLNFo.jpg After: https://i.imgur.com/omYmn7Q.jpg https://i.imgur.com/omYmn7Q.jpg If it's possible to clean a puppy, it might be possible to extrapolate how light should behave in the general case.

eru 9y ago
There are actually also quite a few transformations you can do to a video that would be noticeable in a still frame. In particular, I remember some video stabilization a few years ago that just warped the input images instead of building a proper 3d model. It worked in real time and looked great in the video---but in a still frame you could see that the angles were all slightly off.
andybak 9y ago
Don't forget that sometimes too much realism is a problem. The "soap opera" effect with high frame rate source material or displays that interpolate frames is commonly regarded as looking awful. However - it is more realistic but less appealing to our current aesthetic sensibilities. Photorealism is a great tool to have in your box but it's not the end goal in any creative medium.
modeless 9y ago
Hollywood is not optimizing for perfect realism. They are operating under many constraints such as budget (yes, even on Avatar they had budgets), schedule, and the need for directorial control (which often conflicts with realism). It would absolutely be possible to produce prerendered videos of mostly static scenes using today's technology that would be indistinguishable from reality, although the expense would be high.
- sillysaurus3 9y ago
  It would absolutely be possible to produce prerendered videos of mostly static scenes using today's technology that would be indistinguishable from reality, although the expense would be high. And yet, no one has. :) It'd be a fun trophy to claim. My theory is that it's not merely a matter of computing power. We have massive horsepower now, but our algorithms are still wrong at a fundamental level. We still approach it by trying to simulate the underlying physics of light: the transport algorithms, radiance sampling, and so on. But between the RGB textures that have nothing to do with nature and the physics approximations that we're forced to settle with, something ends up lost in translation. No matter what we try, that human observer ends up looking at a final product that looks nothing like a real camcorder video. Back during ~iPhone 4 days, someone forged some pictures of the "next gen iPhone." It was a hoax, but a lot of people were fooled at the time. Remarkably, it was a computer rendering. The reason it looked real is because he took a picture of the rendering using a real camera, which muddied up the colors to the point that you couldn't tell it was a rendering. The idea here would be similar: analyze the actual pixels coming out of a real camcorder and try to mimic the nuances as closely as possible. In the best case, it wouldn't be a physics simulation at all: the program might be able to guess how the scene should look based on partial data and past experience.

PeachPlum 9y ago

They are nothing like indistinguishable and have the exact problems I would look for to see if it was CGI. * The lower edge on the foreground box, particularly the lower left corner. * The lower edges of the mirror box. * To a lesser extent, the bottom corners of the surrounding box. Once you spend time in rendering, you know where to look for flaws. If your example was animated, it would be obviously rendered.

fenwick67 9y ago
I'm pretty confident that if you asked laypeople to mark which was real and which was fake they couldn't reliably tell.