4 ms·

Note the important distinction: no one, anywhere, has ever created a real-time video simulation indistinguishable from the output of a real camcorder. Real-time

Note the important distinction: no one, anywhere, has ever created a real-time video simulation indistinguishable from the output of a real camcorder. Real-time video is significantly different than still-frames. It's why video compression uses completely different algorithms than jpeg compression, for example. Our visual systems are tuned to notice problems with real-time outputs, whereas a still frame is ambiguous. There are all kinds of transformations you can do to a perfect still frame where it still seems perfect afterwards: You can tweak the contrast, brightness, hue, throw a softpass filter on it, etc, and a human observer still won't notice a big difference. In other words, there are many ways to "cheat"! Even though we're doing operations that don't correspond to how nature behaves, a human observer is still fooled by them. Yet the moment that you string a bunch of still frames together into a video, that human observer will pounce on you immediately and call it out as fake. I'm not sure why motion makes such a massive difference, but it does. To put it another way, if it were possible to make a real-time video whose ground truth was identical to a camcorder, we'd see it in hollywood. But no one has been able to; Avatar was the best we could do. I think it should be possible to train a neural net to paint the scene, similar to a human painter. DaVinci didn't make diffuse/ambient/specular textures; he simply painted the scene according to how it would look in real life. And in real life, color changes are gradual. It's crucial that the colors change in exactly the right way, but the changes themselves are rarely discontinuous. That means a neural net might be able to catch on pretty quickly. Bots are already doing similar work: https://www.reddit.com/r/aww/comments/505zzr/colour_run/d71grax/?context=3 https://www.reddit.com/r/aww/comments/505zzr/colour_run/d71g... Before: https://i.imgur.com/EqMLNFo.jpg https://i.imgur.com/EqMLNFo.jpg After: https://i.imgur.com/omYmn7Q.jpg https://i.imgur.com/omYmn7Q.jpg If it's possible to clean a puppy, it might be possible to extrapolate how light should behave in the general case.

eru 9y ago
There are actually also quite a few transformations you can do to a video that would be noticeable in a still frame. In particular, I remember some video stabilization a few years ago that just warped the input images instead of building a proper 3d model. It worked in real time and looked great in the video---but in a still frame you could see that the angles were all slightly off.
andybak 9y ago
Don't forget that sometimes too much realism is a problem. The "soap opera" effect with high frame rate source material or displays that interpolate frames is commonly regarded as looking awful. However - it is more realistic but less appealing to our current aesthetic sensibilities. Photorealism is a great tool to have in your box but it's not the end goal in any creative medium.
modeless 9y ago
Hollywood is not optimizing for perfect realism. They are operating under many constraints such as budget (yes, even on Avatar they had budgets), schedule, and the need for directorial control (which often conflicts with realism). It would absolutely be possible to produce prerendered videos of mostly static scenes using today's technology that would be indistinguishable from reality, although the expense would be high.
- sillysaurus3 9y ago
  It would absolutely be possible to produce prerendered videos of mostly static scenes using today's technology that would be indistinguishable from reality, although the expense would be high. And yet, no one has. :) It'd be a fun trophy to claim. My theory is that it's not merely a matter of computing power. We have massive horsepower now, but our algorithms are still wrong at a fundamental level. We still approach it by trying to simulate the underlying physics of light: the transport algorithms, radiance sampling, and so on. But between the RGB textures that have nothing to do with nature and the physics approximations that we're forced to settle with, something ends up lost in translation. No matter what we try, that human observer ends up looking at a final product that looks nothing like a real camcorder video. Back during ~iPhone 4 days, someone forged some pictures of the "next gen iPhone." It was a hoax, but a lot of people were fooled at the time. Remarkably, it was a computer rendering. The reason it looked real is because he took a picture of the rendering using a real camera, which muddied up the colors to the point that you couldn't tell it was a rendering. The idea here would be similar: analyze the actual pixels coming out of a real camcorder and try to mimic the nuances as closely as possible. In the best case, it wouldn't be a physics simulation at all: the program might be able to guess how the scene should look based on partial data and past experience.
  - deleted 9y ago
    [deleted]
  - falsedan 9y ago
    > And yet, no one has. :) Plenty of people have. How would you spot it in a movie or music video?
falsedan 9y ago
> That's not dismissive -- no one has ever made any program that outputs a string of images indistinguishable from a real camcorder. It's just that hard. > Note the important distinction: no one, anywhere, has ever created a real-time video simulation indistinguishable from the output of a real camcorder You redefined what you're talking about: clarifying that it has to be real-time. What if someone generated 2 images within 1/24th of a second? Would that would count as a 33.367ms long video? To learn more about how the eye interprets the structure of light, I recommend skimming Minnaert's Light and Color in the Outdoors (won't cover motion).
tasty_freeze 9y ago
> Real-time video is significantly different than still-frames. It's why video compression uses completely different algorithms than jpeg compression, for example. You are right that video is different than still images, namely there is a lot of temporal redundancy that can be used to compress things to an acceptable level. But h.264 and whatnot are, at their root, a series of jpg frames (the "I" intra frames), and various temporal compression techniques are used to interpolate between the I frames (P = predicted, B = bidirectional reconstruction). Motion jpg is a thing, and it is often preferred in video editing over h.264 or whatever because each frame is self contained. It is essentially nothing but a stream of I frames.
joefourier 9y ago
> Real-time video is significantly different than still-frames. It's why video compression uses completely different algorithms than jpeg compression, for example. That's not quite true, mainstream video codecs are all based around discrete cosine transforms (or integer approximations thereof) + scalar quantization, followed by entropy coding. If you look at early codecs like MPEG1 or MPEG2, individual frames are compressed in a manner almost identical to JPEG. The main difference is that video allows you to do motion compression, but that still leaves you with a residual (or difference) frame, which is compressed much like a jpeg. Modern codecs like h264 and h265 are far more complex, but the basic algorithms at the heart of them are not fundamentally different from JPEG. In fact you can use h264 as a still image codec, at about 2x the compression efficiency! > If it's possible to clean a puppy, it might be possible to extrapolate how light should behave in the general case. That bot did not "clean" a puppy, it just converted it to black and white. Since the dye only changed the fur color (chroma) without darkening/lightening it, it was not visible in the newly added synthetic colors. If you look at the image in black and white, the dye is obviously not present: https://i.imgur.com/xIkCP2m.jpg https://i.imgur.com/xIkCP2m.jpg
djmips 9y ago
There's should be no disagreement that given a graphics Turing test that there are going to be scenes that when attempted to be replicated with REALTIME 3D will not even come close to looking like real life let alone something recorded and viewed on screen. People need to stop defending real-time graphics. They are cool but they don't look real. But I also don't think that a neural net based magic wand is a viable solution.
bcoates 9y ago
The problems with full motion video are mostly not light transport problems, they're animation problems. Capturing realistic human movement is tricky enough, let alone generating it, and people are exquisitely sensitive to the creepy effect of unnatural movement. Also, even James Cameron's budget might have been insufficient to model out the material properties of physically credible alternate world, making everything weirdly glossy and fluorescent as an artistic oversight rather than a limitation of the rendering technology. Creating realistic video panning over still-life scenes, like the view out of a helicopter, is a solved problem at sufficient modelling budgets, you don't need AI just fully described scene and lots of flops to run your light transport algorithm. Colorizing NNs are cute but not relevant to this problem. There's a good analogue to sound: a long time ago we mastered algorithms to precisely simulate any musical instrument, acoustic environment, pre-digital audio tranformation, etc. but we still use voice actors because modelling human speech behavior is a much harder problem that modelling audio physics.

eru 9y ago

There are actually also quite a few transformations you can do to a video that would be noticeable in a still frame. In particular, I remember some video stabilization a few years ago that just warped the input images instead of building a proper 3d model. It worked in real time and looked great in the video---but in a still frame you could see that the angles were all slightly off.

andybak 9y ago

Don't forget that sometimes too much realism is a problem. The "soap opera" effect with high frame rate source material or displays that interpolate frames is commonly regarded as looking awful. However - it is more realistic but less appealing to our current aesthetic sensibilities. Photorealism is a great tool to have in your box but it's not the end goal in any creative medium.

modeless 9y ago

Hollywood is not optimizing for perfect realism. They are operating under many constraints such as budget (yes, even on Avatar they had budgets), schedule, and the need for directorial control (which often conflicts with realism). It would absolutely be possible to produce prerendered videos of mostly static scenes using today's technology that would be indistinguishable from reality, although the expense would be high.

sillysaurus3 9y ago
It would absolutely be possible to produce prerendered videos of mostly static scenes using today's technology that would be indistinguishable from reality, although the expense would be high. And yet, no one has. :) It'd be a fun trophy to claim. My theory is that it's not merely a matter of computing power. We have massive horsepower now, but our algorithms are still wrong at a fundamental level. We still approach it by trying to simulate the underlying physics of light: the transport algorithms, radiance sampling, and so on. But between the RGB textures that have nothing to do with nature and the physics approximations that we're forced to settle with, something ends up lost in translation. No matter what we try, that human observer ends up looking at a final product that looks nothing like a real camcorder video. Back during ~iPhone 4 days, someone forged some pictures of the "next gen iPhone." It was a hoax, but a lot of people were fooled at the time. Remarkably, it was a computer rendering. The reason it looked real is because he took a picture of the rendering using a real camera, which muddied up the colors to the point that you couldn't tell it was a rendering. The idea here would be similar: analyze the actual pixels coming out of a real camcorder and try to mimic the nuances as closely as possible. In the best case, it wouldn't be a physics simulation at all: the program might be able to guess how the scene should look based on partial data and past experience.
- deleted 9y ago
  [deleted]
- falsedan 9y ago
  > And yet, no one has. :) Plenty of people have. How would you spot it in a movie or music video?

falsedan 9y ago

> That's not dismissive -- no one has ever made any program that outputs a string of images indistinguishable from a real camcorder. It's just that hard. > Note the important distinction: no one, anywhere, has ever created a real-time video simulation indistinguishable from the output of a real camcorder You redefined what you're talking about: clarifying that it has to be real-time. What if someone generated 2 images within 1/24th of a second? Would that would count as a 33.367ms long video? To learn more about how the eye interprets the structure of light, I recommend skimming Minnaert's Light and Color in the Outdoors (won't cover motion).

tasty_freeze 9y ago

> Real-time video is significantly different than still-frames. It's why video compression uses completely different algorithms than jpeg compression, for example. You are right that video is different than still images, namely there is a lot of temporal redundancy that can be used to compress things to an acceptable level. But h.264 and whatnot are, at their root, a series of jpg frames (the "I" intra frames), and various temporal compression techniques are used to interpolate between the I frames (P = predicted, B = bidirectional reconstruction). Motion jpg is a thing, and it is often preferred in video editing over h.264 or whatever because each frame is self contained. It is essentially nothing but a stream of I frames.

joefourier 9y ago

> Real-time video is significantly different than still-frames. It's why video compression uses completely different algorithms than jpeg compression, for example. That's not quite true, mainstream video codecs are all based around discrete cosine transforms (or integer approximations thereof) + scalar quantization, followed by entropy coding. If you look at early codecs like MPEG1 or MPEG2, individual frames are compressed in a manner almost identical to JPEG. The main difference is that video allows you to do motion compression, but that still leaves you with a residual (or difference) frame, which is compressed much like a jpeg. Modern codecs like h264 and h265 are far more complex, but the basic algorithms at the heart of them are not fundamentally different from JPEG. In fact you can use h264 as a still image codec, at about 2x the compression efficiency! > If it's possible to clean a puppy, it might be possible to extrapolate how light should behave in the general case. That bot did not "clean" a puppy, it just converted it to black and white. Since the dye only changed the fur color (chroma) without darkening/lightening it, it was not visible in the newly added synthetic colors. If you look at the image in black and white, the dye is obviously not present: https://i.imgur.com/xIkCP2m.jpg https://i.imgur.com/xIkCP2m.jpg

djmips 9y ago

There's should be no disagreement that given a graphics Turing test that there are going to be scenes that when attempted to be replicated with REALTIME 3D will not even come close to looking like real life let alone something recorded and viewed on screen. People need to stop defending real-time graphics. They are cool but they don't look real. But I also don't think that a neural net based magic wand is a viable solution.

bcoates 9y ago

The problems with full motion video are mostly not light transport problems, they're animation problems. Capturing realistic human movement is tricky enough, let alone generating it, and people are exquisitely sensitive to the creepy effect of unnatural movement. Also, even James Cameron's budget might have been insufficient to model out the material properties of physically credible alternate world, making everything weirdly glossy and fluorescent as an artistic oversight rather than a limitation of the rendering technology. Creating realistic video panning over still-life scenes, like the view out of a helicopter, is a solved problem at sufficient modelling budgets, you don't need AI just fully described scene and lots of flops to run your light transport algorithm. Colorizing NNs are cute but not relevant to this problem. There's a good analogue to sound: a long time ago we mastered algorithms to precisely simulate any musical instrument, acoustic environment, pre-digital audio tranformation, etc. but we still use voice actors because modelling human speech behavior is a much harder problem that modelling audio physics.