5 ms·

Can anyone tell me if it's reasonable to think this technology might substantially replace or augment movie CGI in the future? I have no domain experience with

Can anyone tell me if it's reasonable to think this technology might substantially replace or augment movie CGI in the future? I have no domain experience with CGI or graphical simulation, but I have a passing interest as an occasional admirer of what shows up on /r/simulated. Does modern CGI make use of machine learning like this? If not, what does it have in common aside from significant use of GPUs? What are the computational intensive parts of CGI, and what are the heavily manual and labor intensive parts of the CGI development workflow?

delinka 8y ago
Keep in mind that these are stills. Once the motion starts, humans detect the not-quite-realness easily. Even the 2D video creations by AI aren’t stitched correctly, let alone getting all the details of an animated 3D person right.
- dylan604 8y ago
  They are getting closer, but I do agree that it is still not quite there. It's pretty common practice now at the beginning of each film production to do full image capture of all of the main talent. It's a back up in case something untoward happens that the film can still be finished by using their 3D likeness.
  - cm2187 8y ago
    You mean for an Oliver Reed/Gladiator scenario?
    - ptrott2017 8y ago
      More commonly for postproduction fixes (think stunts and inserting talent likeness, relighting/reframing a shot, inserting reflections when CG-ing the background of a window etc.…) so you can fix sequences without having to recall talent which can produce availability delays and additional expense beyond what it takes to fix digitally.
- agumonkey 8y ago
  It used to be the case but even with not too recent CGI tech they can make ~alright main characters (see SW ep. VII). With the most recent siggraph tricks I'm sure most of the population won't notice (if not abused).
  - delinka 8y ago
    Good CGI tools in the hands of a talented human can make for good results.
    - agumonkey 8y ago
      Oh yeah don't misread me, I'm not asking for fancier tech. Movies != CGI, far from that. I cry when I compare old movies that had flesh to the hyper hectic pixel fest of today..
      - FiveSquared 8y ago
        No cgi no cry
- computerex 8y ago
  https://goo.gl/Z5KnrZ https://goo.gl/Z5KnrZ https://youtu.be/9Yq67CjDqvw https://youtu.be/9Yq67CjDqvw
  - hellofunk 8y ago
    Lip-synching is quite different from full body motion. And what you have posted is actually using original real source video for most of the pixels if I understand correctly.
    - computerex 8y ago
      I was mainly responding to the parent's comment: > Keep in mind that these are stills. Once the motion starts, humans detect the not-quite-realness easily. Even the 2D video creations by AI aren’t stitched correctly, let alone getting all the details of an animated 3D person right. I agree this isn't end to end video synthesis but the end results are still very impressive and realistic. I can easily imagine this type of technology being a problem, because video evidence will be devalued and mean nothing. They basically just animate Obama's lips using a RNN and then composite the output onto a target video of Obama to change the appearance of what he is saying. Yes they aren't generating all of Obama's head yet but this result in itself is impressive. Also I can't wait to see where GAN's take us 10-20 years down the road. Don't forget they are still in their infancy relatively speaking.
- lambdadmitry 8y ago
  Even in those stills (cherry-picked for sure) it's easy to spot NN artifacts, it's just that most people on HN trained themselves to spot "classic" CGI problems that are local. Check the general geometry and long-distance order, that's what problematic for NN because of their nature. Even using photos as references NNs can't avoid shifting ears or gaze or warping faces because there's no concept of the underlying 3D object there, just a bunch of flat transformations.
mtgx 8y ago
Have you seen Aquaman yet? Studios are already starting to replace popular actors in the movies with younger/better looking CGI versions of themselves. As the tech gets cheaper, you should expect a lot more of this in future blockbuster movies.
- dmoy 8y ago
  > younger/better Or, a bit earlier (star wars?), not-dead versions of themselves.
menzoic 8y ago
NVIDIA created an entire CGI city with Machine Learning "Research at NVIDIA: The First Interactive AI Rendered Virtual World " - https://www.youtube.com/watch?v=ayPqjPekn7g https://www.youtube.com/watch?v=ayPqjPekn7g
jtolmar 8y ago
It depends on how broad of a brush you're painting with when you say "this technology." Neural network based or neural network enhanced rendering techniques are certainly going to get applied to movie CGI. This specific adversarial network only does stills, so it won't. A lot of the recent "wow" research is directly generating images, which I don't think will scale up to realistic scenes as soon as anything moves and rotates, but there's other research in hybridizing 3D model rendering with AI[1] that seems like it could get there. And maybe the adversarial network model in article works with those techniques too. Also, it's important to consider control when talking about CGI or other creative endeavors. Generating a perfectly realistic scene is useless without being able to control the details of the scene. So the completely end-to-end magic demos aren't nearly as useful as things that generate off of armatures or performance transfer. [1] https://www.theverge.com/2018/12/3/18121198/ai-generated-video-game-graphics-nvidia-driving-demo-neurips https://www.theverge.com/2018/12/3/18121198/ai-generated-vid...
deleted 8y ago
[deleted]
csomar 8y ago
These are only images. Theoretically it is possible. But I guess we are still years away even with exponential improvement in the field.
goberoi 8y ago
Yes, this sort of tech is likely to make its way into CGI. Some relevant resources if you're interested: 1. This post shows how GANs were used to replicate the Princess Leia scene in Rogue One. The scene in the movie required a team of special effects artists, whereas this required access to lots of images of Carrie Fischer, a GPU, and this technology: https://io9.gizmodo.com/this-video-uses-the-power-of-deepfakes-to-re-capture-th-1828907452 https://io9.gizmodo.com/this-video-uses-the-power-of-deepfak... 2. I wrote this popular post on this topic (first page on Google for "deepfakes") that explains how it works, and where it can go for commercial applications: https://hackernoon.com/exploring-deepfakes-20c9947c22d9 https://hackernoon.com/exploring-deepfakes-20c9947c22d9 3. If you prefer video, I gave a humorous talk on the same recently. It is designed for lay people, and has lots of examples: https://www.youtube.com/watch?v=wajS0XHzfpU&feature=youtu.be&t=153 https://www.youtube.com/watch?v=wajS0XHzfpU&feature=youtu.be...
lordnacho 8y ago
I'm not the expert you're looking for, but surely we are not far from being able to generate movies with bespoke content for every person. Performance art itself is about to be upturned. AI can already string bad poetry together, surely in not long it can do a 5 act play, vary the characters a bit, jumble up the situations, create 3D realistic models, and put the whole thing together into your very own Breaking Bad that caters to your particular biases. It can then take your feedback and do spinoffs, parodies, and so forth forever. You'll never be able to get up.
- liamcardenas 8y ago
  Surely? Doing this in any meaningful way requires so much insight into the human mind and culture.
  - lordnacho 8y ago
    Well I'm just saying it looks like we're close. Just based on the pieces being prototyped already. There's people on this thread posting photoreal fake videos, text generators exist at least as satire sites (POMO generator), and it's not like you couldn't make a tree of story elements. Someone is gonna glue those pieces together and make it a business.
    - hellofunk 8y ago
      I think we’re nowhere near close. Yeah we can generate an image, but even AI music tends to get pretty boring pretty quickly, and is far from having the human element. A movie or 5-act play? You’re being quite optimistic.
    - badpun 8y ago
      I’ve read somewhere that there already are publishers machine generating genre fiction (crime stories), that are only manually polished by humans after the generation. However, a distance between a shitty genre novel and Dostojewski is not to be underestimated - msybr the only thing that connects them is that they both are texts with plot and characters.
backpropaganda 8y ago
Note that isn't tabula rasa generation. All these faces are generated from reference images. See this timestamped video for example: https://youtu.be/kSLJriaOumA?t=80 https://youtu.be/kSLJriaOumA?t=80. Current deep generative tech requires a carefully designed dataset of images with controlled variation. For the faces above, a dataset of celebrity faces was carefully constructed from a larger wild dataset, so as to not have any rare features, and the eyes and noses were aligned, making the dataset digestible by a GAN. This particular method then also uses multiple style images which it takes certain features from to mix up and create the mixture face. The main problem with this tech is that the quality can not be reliably controlled. Whatever it generates has to be passed through a human checker/curator pretty much. You can't just serve up generations to the audience without checking what they are. So, imo, this tech would be very useful for the concert art stage of the creative process, being able to just create new things, or mixup existing things, but only to create an intermediate artifact, which would then be used as inspiration for the artists to create the final art. This is also devoid of any form of higher-level reasoning. It doesn't know concepts a human artist would know, such as object permanence, intuitive physics, etc. Something as simple as a ball falling down should squish a certain way, and when hit the ground should squash another way. It also doesn't have artistic sensibilities like anticipation, etc., that have been laid out in the 12 rules of animation, and even more, there's almost no way to communicate it to these models, short of just creating a huge dataset which has those 12 rules, and just hoping, rather praying, it learns it.
grizzles 8y ago
I've been tweeting about this a bit lately. [1] No doubt in my mind that soon it will be possible to take in a movie script and a couple hundred dollars of electricity later output a full movie. Think of the implications, not just for entertainment but for society (eg. politics & criminal justice) in general. [1] https://twitter.com/ericfaccer/status/1077463342761951232 https://twitter.com/ericfaccer/status/1077463342761951232
- usrusr 8y ago
  > Think of the implications, not just for entertainment but for society (eg. politics & criminal justice) in general. Take the Reuters feed, render as Fox News, then apply a Wes Anderson style transfer. Maybe exploring all those permutations will be more interesting than creating deceptive fakes?

delinka 8y ago

Keep in mind that these are stills. Once the motion starts, humans detect the not-quite-realness easily. Even the 2D video creations by AI aren’t stitched correctly, let alone getting all the details of an animated 3D person right.

dylan604 8y ago
They are getting closer, but I do agree that it is still not quite there. It's pretty common practice now at the beginning of each film production to do full image capture of all of the main talent. It's a back up in case something untoward happens that the film can still be finished by using their 3D likeness.
- cm2187 8y ago
  You mean for an Oliver Reed/Gladiator scenario?
  - ptrott2017 8y ago
    More commonly for postproduction fixes (think stunts and inserting talent likeness, relighting/reframing a shot, inserting reflections when CG-ing the background of a window etc.…) so you can fix sequences without having to recall talent which can produce availability delays and additional expense beyond what it takes to fix digitally.
agumonkey 8y ago
It used to be the case but even with not too recent CGI tech they can make ~alright main characters (see SW ep. VII). With the most recent siggraph tricks I'm sure most of the population won't notice (if not abused).
- delinka 8y ago
  Good CGI tools in the hands of a talented human can make for good results.
  - agumonkey 8y ago
    Oh yeah don't misread me, I'm not asking for fancier tech. Movies != CGI, far from that. I cry when I compare old movies that had flesh to the hyper hectic pixel fest of today..
    - FiveSquared 8y ago
      No cgi no cry
computerex 8y ago
https://goo.gl/Z5KnrZ https://goo.gl/Z5KnrZ https://youtu.be/9Yq67CjDqvw https://youtu.be/9Yq67CjDqvw
- hellofunk 8y ago
  Lip-synching is quite different from full body motion. And what you have posted is actually using original real source video for most of the pixels if I understand correctly.
  - computerex 8y ago
    I was mainly responding to the parent's comment: > Keep in mind that these are stills. Once the motion starts, humans detect the not-quite-realness easily. Even the 2D video creations by AI aren’t stitched correctly, let alone getting all the details of an animated 3D person right. I agree this isn't end to end video synthesis but the end results are still very impressive and realistic. I can easily imagine this type of technology being a problem, because video evidence will be devalued and mean nothing. They basically just animate Obama's lips using a RNN and then composite the output onto a target video of Obama to change the appearance of what he is saying. Yes they aren't generating all of Obama's head yet but this result in itself is impressive. Also I can't wait to see where GAN's take us 10-20 years down the road. Don't forget they are still in their infancy relatively speaking.
lambdadmitry 8y ago
Even in those stills (cherry-picked for sure) it's easy to spot NN artifacts, it's just that most people on HN trained themselves to spot "classic" CGI problems that are local. Check the general geometry and long-distance order, that's what problematic for NN because of their nature. Even using photos as references NNs can't avoid shifting ears or gaze or warping faces because there's no concept of the underlying 3D object there, just a bunch of flat transformations.

mtgx 8y ago

Have you seen Aquaman yet? Studios are already starting to replace popular actors in the movies with younger/better looking CGI versions of themselves. As the tech gets cheaper, you should expect a lot more of this in future blockbuster movies.

dmoy 8y ago
> younger/better Or, a bit earlier (star wars?), not-dead versions of themselves.

menzoic 8y ago

NVIDIA created an entire CGI city with Machine Learning "Research at NVIDIA: The First Interactive AI Rendered Virtual World " - https://www.youtube.com/watch?v=ayPqjPekn7g https://www.youtube.com/watch?v=ayPqjPekn7g

jtolmar 8y ago

It depends on how broad of a brush you're painting with when you say "this technology." Neural network based or neural network enhanced rendering techniques are certainly going to get applied to movie CGI. This specific adversarial network only does stills, so it won't. A lot of the recent "wow" research is directly generating images, which I don't think will scale up to realistic scenes as soon as anything moves and rotates, but there's other research in hybridizing 3D model rendering with AI[1] that seems like it could get there. And maybe the adversarial network model in article works with those techniques too. Also, it's important to consider control when talking about CGI or other creative endeavors. Generating a perfectly realistic scene is useless without being able to control the details of the scene. So the completely end-to-end magic demos aren't nearly as useful as things that generate off of armatures or performance transfer. [1] https://www.theverge.com/2018/12/3/18121198/ai-generated-video-game-graphics-nvidia-driving-demo-neurips https://www.theverge.com/2018/12/3/18121198/ai-generated-vid...

deleted 8y ago

[deleted]

csomar 8y ago

These are only images. Theoretically it is possible. But I guess we are still years away even with exponential improvement in the field.

goberoi 8y ago

Yes, this sort of tech is likely to make its way into CGI. Some relevant resources if you're interested: 1. This post shows how GANs were used to replicate the Princess Leia scene in Rogue One. The scene in the movie required a team of special effects artists, whereas this required access to lots of images of Carrie Fischer, a GPU, and this technology: https://io9.gizmodo.com/this-video-uses-the-power-of-deepfakes-to-re-capture-th-1828907452 https://io9.gizmodo.com/this-video-uses-the-power-of-deepfak... 2. I wrote this popular post on this topic (first page on Google for "deepfakes") that explains how it works, and where it can go for commercial applications: https://hackernoon.com/exploring-deepfakes-20c9947c22d9 https://hackernoon.com/exploring-deepfakes-20c9947c22d9 3. If you prefer video, I gave a humorous talk on the same recently. It is designed for lay people, and has lots of examples: https://www.youtube.com/watch?v=wajS0XHzfpU&feature=youtu.be&t=153 https://www.youtube.com/watch?v=wajS0XHzfpU&feature=youtu.be...

lordnacho 8y ago

I'm not the expert you're looking for, but surely we are not far from being able to generate movies with bespoke content for every person. Performance art itself is about to be upturned. AI can already string bad poetry together, surely in not long it can do a 5 act play, vary the characters a bit, jumble up the situations, create 3D realistic models, and put the whole thing together into your very own Breaking Bad that caters to your particular biases. It can then take your feedback and do spinoffs, parodies, and so forth forever. You'll never be able to get up.

liamcardenas 8y ago
Surely? Doing this in any meaningful way requires so much insight into the human mind and culture.
- lordnacho 8y ago
  Well I'm just saying it looks like we're close. Just based on the pieces being prototyped already. There's people on this thread posting photoreal fake videos, text generators exist at least as satire sites (POMO generator), and it's not like you couldn't make a tree of story elements. Someone is gonna glue those pieces together and make it a business.
  - hellofunk 8y ago
    I think we’re nowhere near close. Yeah we can generate an image, but even AI music tends to get pretty boring pretty quickly, and is far from having the human element. A movie or 5-act play? You’re being quite optimistic.
  - badpun 8y ago
    I’ve read somewhere that there already are publishers machine generating genre fiction (crime stories), that are only manually polished by humans after the generation. However, a distance between a shitty genre novel and Dostojewski is not to be underestimated - msybr the only thing that connects them is that they both are texts with plot and characters.

backpropaganda 8y ago

Note that isn't tabula rasa generation. All these faces are generated from reference images. See this timestamped video for example: https://youtu.be/kSLJriaOumA?t=80 https://youtu.be/kSLJriaOumA?t=80. Current deep generative tech requires a carefully designed dataset of images with controlled variation. For the faces above, a dataset of celebrity faces was carefully constructed from a larger wild dataset, so as to not have any rare features, and the eyes and noses were aligned, making the dataset digestible by a GAN. This particular method then also uses multiple style images which it takes certain features from to mix up and create the mixture face. The main problem with this tech is that the quality can not be reliably controlled. Whatever it generates has to be passed through a human checker/curator pretty much. You can't just serve up generations to the audience without checking what they are. So, imo, this tech would be very useful for the concert art stage of the creative process, being able to just create new things, or mixup existing things, but only to create an intermediate artifact, which would then be used as inspiration for the artists to create the final art. This is also devoid of any form of higher-level reasoning. It doesn't know concepts a human artist would know, such as object permanence, intuitive physics, etc. Something as simple as a ball falling down should squish a certain way, and when hit the ground should squash another way. It also doesn't have artistic sensibilities like anticipation, etc., that have been laid out in the 12 rules of animation, and even more, there's almost no way to communicate it to these models, short of just creating a huge dataset which has those 12 rules, and just hoping, rather praying, it learns it.

grizzles 8y ago

I've been tweeting about this a bit lately. [1] No doubt in my mind that soon it will be possible to take in a movie script and a couple hundred dollars of electricity later output a full movie. Think of the implications, not just for entertainment but for society (eg. politics & criminal justice) in general. [1] https://twitter.com/ericfaccer/status/1077463342761951232 https://twitter.com/ericfaccer/status/1077463342761951232

usrusr 8y ago
> Think of the implications, not just for entertainment but for society (eg. politics & criminal justice) in general. Take the Reuters feed, render as Fox News, then apply a Wes Anderson style transfer. Maybe exploring all those permutations will be more interesting than creating deceptive fakes?