8 ms·

Isn't it frustrating? All of this effort, and it looks nothing like real camcorder footage. That's not dismissive -- no one has ever made any program that outp

Isn't it frustrating? All of this effort, and it looks nothing like real camcorder footage. That's not dismissive -- no one has ever made any program that outputs a string of images indistinguishable from a real camcorder. It's just that hard. I think whatever the next leap forward looks like, it will come from a nontraditional approach. Something strange, like powering your real-time lighting model by an actual camcorder -- set it up, point it at a real-world scene, then write a program that analyzes the way the light and color behaves in the camcorder's ground truth input. Then you'd somehow extrapolate that behavior across the rest of your scene. That last step sounds a lot like "Just add magic," but we have deep learning pipelines now. You could train it against your camcorder's input feed. Neural nets tend to work well when you have a reliable model, and we have the perfect one. So more precisely, you'd train your neural net against the camera's input video stream: at each generation, the program would try to paint your scene using whatever it thinks is the best guess for how the colors should look. Then you move your camcorder around, capturing how the colors actually looked, giving the pipeline enough data to correct itself. Rinse and repeat a few thousand times. The key to realism, and the central problem, is that colors affect colors around them. The way colors blend and fade across a wall has to be exactly right. There's no room for deviation from real life. Our visual systems have been tuned for a billion years to notice that. There are all kinds of issues with this idea: the real-world scene would need to be identical to the virtual scene, at least to start. The program would need to know the camera's orientation in order to figure out how to backproject the real-life illumination data onto the virtual scene. But at the end of it, you should wind up with a scene whose colors behave identically to real life. It seems like a promising approach because it gets rid of the whole idea of diffuse/ambient/specular maps, which don't correspond to reality anyway My favorite example: What does it mean to multiply a light's RGB color by a diffuse texture's RGB value? Nothing! It's a completely meaningless operation which happens to approximate reality quite well. There are huge advantages with that approach, like the flexibility of letting an artist create textures. But if the goal is precise, exact realism as defined by your camcorder, then we might be able to mimic nature directly. (Those dynamic occluders looked incredibly cool, by the way!)

mixedbit 9y ago
One of the problems with your idea is that the camera position is one of the parameters of the rendering equation, so the same surface looks different from different points in 3D space. To fully capture how light interacts with the space, you can't make one photo, you would need to place the camera in infinite number of positions and store the results in infinite amount of memory.
- sillysaurus3 9y ago
  It's similar to what Tesla is doing with autopilot. Theoretically they'd need to train their model on an infinite number of positions, but after a certain amount of training and tweaking you start getting good results.
Xcelerate 9y ago
We have spectral renderers that simulate light transport extremely accurately and in fact are indistinguishable from a photograph (see http://www.graphics.cornell.edu/online/box/compare.html http://www.graphics.cornell.edu/online/box/compare.html) The problem isn’t solving light transport itself (the rendering equation has been known for years, and is asymptotically exactly solved using unbiased numerical integration techniques); it’s modeling the interaction of light with complex physical materials and doing so quickly that poses the challenge.
- sgift 9y ago
  > and doing so quickly Most important part. The best solution which arrives too late is completely useless.
- sillysaurus3 9y ago
  Note the important distinction: no one, anywhere, has ever created a real-time video simulation indistinguishable from the output of a real camcorder. Real-time video is significantly different than still-frames. It's why video compression uses completely different algorithms than jpeg compression, for example. Our visual systems are tuned to notice problems with real-time outputs, whereas a still frame is ambiguous. There are all kinds of transformations you can do to a perfect still frame where it still seems perfect afterwards: You can tweak the contrast, brightness, hue, throw a softpass filter on it, etc, and a human observer still won't notice a big difference. In other words, there are many ways to "cheat"! Even though we're doing operations that don't correspond to how nature behaves, a human observer is still fooled by them. Yet the moment that you string a bunch of still frames together into a video, that human observer will pounce on you immediately and call it out as fake. I'm not sure why motion makes such a massive difference, but it does. To put it another way, if it were possible to make a real-time video whose ground truth was identical to a camcorder, we'd see it in hollywood. But no one has been able to; Avatar was the best we could do. I think it should be possible to train a neural net to paint the scene, similar to a human painter. DaVinci didn't make diffuse/ambient/specular textures; he simply painted the scene according to how it would look in real life. And in real life, color changes are gradual. It's crucial that the colors change in exactly the right way, but the changes themselves are rarely discontinuous. That means a neural net might be able to catch on pretty quickly. Bots are already doing similar work: https://www.reddit.com/r/aww/comments/505zzr/colour_run/d71grax/?context=3 https://www.reddit.com/r/aww/comments/505zzr/colour_run/d71g... Before: https://i.imgur.com/EqMLNFo.jpg https://i.imgur.com/EqMLNFo.jpg After: https://i.imgur.com/omYmn7Q.jpg https://i.imgur.com/omYmn7Q.jpg If it's possible to clean a puppy, it might be possible to extrapolate how light should behave in the general case.

mixedbit 9y ago

One of the problems with your idea is that the camera position is one of the parameters of the rendering equation, so the same surface looks different from different points in 3D space. To fully capture how light interacts with the space, you can't make one photo, you would need to place the camera in infinite number of positions and store the results in infinite amount of memory.

sillysaurus3 9y ago
It's similar to what Tesla is doing with autopilot. Theoretically they'd need to train their model on an infinite number of positions, but after a certain amount of training and tweaking you start getting good results.

Xcelerate 9y ago

We have spectral renderers that simulate light transport extremely accurately and in fact are indistinguishable from a photograph (see http://www.graphics.cornell.edu/online/box/compare.html http://www.graphics.cornell.edu/online/box/compare.html) The problem isn’t solving light transport itself (the rendering equation has been known for years, and is asymptotically exactly solved using unbiased numerical integration techniques); it’s modeling the interaction of light with complex physical materials and doing so quickly that poses the challenge.

sgift 9y ago
> and doing so quickly Most important part. The best solution which arrives too late is completely useless.
sillysaurus3 9y ago
Note the important distinction: no one, anywhere, has ever created a real-time video simulation indistinguishable from the output of a real camcorder. Real-time video is significantly different than still-frames. It's why video compression uses completely different algorithms than jpeg compression, for example. Our visual systems are tuned to notice problems with real-time outputs, whereas a still frame is ambiguous. There are all kinds of transformations you can do to a perfect still frame where it still seems perfect afterwards: You can tweak the contrast, brightness, hue, throw a softpass filter on it, etc, and a human observer still won't notice a big difference. In other words, there are many ways to "cheat"! Even though we're doing operations that don't correspond to how nature behaves, a human observer is still fooled by them. Yet the moment that you string a bunch of still frames together into a video, that human observer will pounce on you immediately and call it out as fake. I'm not sure why motion makes such a massive difference, but it does. To put it another way, if it were possible to make a real-time video whose ground truth was identical to a camcorder, we'd see it in hollywood. But no one has been able to; Avatar was the best we could do. I think it should be possible to train a neural net to paint the scene, similar to a human painter. DaVinci didn't make diffuse/ambient/specular textures; he simply painted the scene according to how it would look in real life. And in real life, color changes are gradual. It's crucial that the colors change in exactly the right way, but the changes themselves are rarely discontinuous. That means a neural net might be able to catch on pretty quickly. Bots are already doing similar work: https://www.reddit.com/r/aww/comments/505zzr/colour_run/d71grax/?context=3 https://www.reddit.com/r/aww/comments/505zzr/colour_run/d71g... Before: https://i.imgur.com/EqMLNFo.jpg https://i.imgur.com/EqMLNFo.jpg After: https://i.imgur.com/omYmn7Q.jpg https://i.imgur.com/omYmn7Q.jpg If it's possible to clean a puppy, it might be possible to extrapolate how light should behave in the general case.