9 ms·

Although this is very impressive, don't get me wrong. I still cannot shake off that video-gamey feel the entire thing has. I watched the demo on YouTube and it

• jjice 4y ago
  I wonder if we had the same feeling. I thought that the characters upper lip moved up too much during some of the more dramatic words.
• yehudalouis 4y ago
  I felt it too - but this seems like the kind of solution that will get you 90% of the way there, but still requires an animator to make it feel 'right'. This is insane though.
• londons_explore 4y ago
  I agree - the resulting look and feel just isn't quite right for a human, and it falls in the uncanny valley.
• vanilla_nut 4y ago
  There's definitely something uncanny about the lips. I feel like the edges of the face are also more static than I expect (especially when she makes dramatic facial expressions like the "angry" expression, which ought to pull the skin a bit). Very video-gamey, not super convincing. But then again this is running a model in just a few seconds, with input from consumer hardware! Makes me wonder what you could do with more capable hardware (or if the iPhone hardware is really THAT good). It does make me think of one excellent application for this software: scanning your own face to make an in-game character who looks exactly like you. I assume I'm not the only one who's labored over character generators for hours to produce a decent-looking character. I think there's a neat opportunity here to get video game players to scan their own face as input instead.
  • ambicapter 4y ago
    This demo doesn't demonstrate scanning your own face to make an in-game character though, just using your face to create the animation mesh. You then have to use an existing high-quality model to generate the final result. Generating a high-quality model from a camera phone is a whole other can of worms.
    • pr0zac 4y ago
      Epic released the tooling for doing an iPhone face scan to create a mesh that can then be used to create a rigged Metahuman model about 6 months ago! https://www.unrealengine.com/en-US/blog/how-to-use-realitycapture-to-scan-yourself-for-mesh-to-metahuman https://www.unrealengine.com/en-US/blog/how-to-use-realityca... My significant other works for Epic on the cinematics team so we've played around with Metahuman a bit. Scans we've done weren't immediately perfect (like the video on that linked page mentions, hair and ears don't scan well) but cleaning it up was basically just deleting those portions of the mesh, letting RealityCapture fill in the rest of the head for the model, and then adding similar looking hair from Metahuman's database. So there's probably a little bit of work needed before we get "scan your head with your phone to create a video game character automatically" but if you're cool requiring hair selection as opposed to auto generating it from the scan we're pretty much there.
• the_af 4y ago
  Yes, and also: the actual actress felt a tiny bit uncomfortable, maybe the performance in front of a live audience, maybe because she could have a slip of the tongue or whatever. This made her performance feel more "human". The rendered animation lacked this aspect, the character didn't seem nervous in the slightest. Still, this is impressive tech, don't get me wrong. But we are still in the Uncanny Valley. Getting better though!
  • tehchromic 4y ago
    My understanding of the uncanny valley is that it doesn't get better, it only gets worse!
    • fknorangesite 4y ago
      It's a valley, not a cliff.
    • the_af 4y ago
      It's not a given that it will happen, but you can get out of the valley. It's just that the effect is more disturbing as you get close (but not outside) the upper limit of the valley.
  • jalk 4y ago
    I immediately wondered if her facial expressions were deliberately chosen to fit what the algorithm is good at - her angry face looked rather silly but the generated version matched pretty good. The generated “gaze to the side” however lacked most of the emotion I read of the “original”
    • Zetice 4y ago
      I also got that sense, specifically around how much she showed her teeth during her video recording. I suspect the algorithm does that with teeth, so she was told "show teeth the entire time" to make it seem like the algorithm was more correct.
      • 4y ago
• albert_e 4y ago
  Yes but maybe if we wanted to smooth over those uncanny valley imperfections we spend more than 5 minutes on our production and get a better result. Seems promising if we allow for iterative improvements.
• cassianoleal 4y ago
  > video-gamey feel I agree. It's very impressive though, a clear leap forward. Hellblade 2's demo is even worse though. It looks like the lips are completely disconnected from the gums and flap around loosely as they move. https://www.youtube.com/watch?v=NCYMNmkjRS4 https://www.youtube.com/watch?v=NCYMNmkjRS4
  • ajmurmann 4y ago
    This exact problem was even called out in kotaku's reporting on this. The exaggerated mouth movement really lean into the weakness of the engine. I'm surprised that they didn't notice this during production of the clip and adjusted how the scene was acted accordingly.
  • Geee 4y ago
    I think the problem is that in computer animation skin is allowed to stretch and compress in ways which is unnatural. Real skin is quite inelastic. It wrinkles and slides which creates the appereance of elasticity. Also, when real materials are stretched or compressed, they tend to form wrinkles in particular ways. I think that human eye can tell these very small discrepancies to guess that this material is not human skin (or any physical material).
• JustSomeNobody 4y ago
  > ...something about the way her mouth moves felt off. Am I the only one or am I just crazy? No, you're not the only one. You may still be crazy.
• nstart 4y ago
  There are definitely a few issues with it especially in the skin stretch and squeeze around the lips, nose, and cheeks areas. There are a lot of muscles and skin movement that’s supposed to happen and rendering that needs time and good attention to detail with the materials. But outside of that, there are the limits of rotoscoping coming into play. Human motion has a lot going on and our 3d captures aren’t anywhere near that kind of fidelity yet. Therefore we still need to rely on what the old masters of animation discovered when they were getting past the rotoscoped era. To make a performance more believable there has to be a little more acting and exaggeration brought in. The eyes can’t just get wider. They need to have anticipation, movement, secondary movement through other pieces of the face, etc. Same with the mouth. It’s not that the mouth can’t stretch in an abnormal way. It’s that it requires a little bit extra like a neck movement creating anticipation of an angry snarl. As far as we have gotten along with mocap tech, when it comes to film, there’s a loooot of hard work that goes into fine tuning done by professional animators to make the performance of each actor look good when it’s translated to the 3d character.
• ghostpepper 4y ago
  I agree. This is impressive, but the article calls it "almost indistinguishable from the original video" which is definitely not true.
• whazor 4y ago
  You can really see it when she closes the eyes, that particular part makes it look super video-gamey.
• vlunkr 4y ago
  Maybe I'm in the minority here, but I have zero interest in realistic character models in video games. There are limitless possibilities when it comes to art styles in gaming, and the one AAA picks over and over is realism. It's so boring.
  • glasss 4y ago
    It also is one of the art styles that turns dated / bad quickly. Any stylized art with a decent to strong design will still look good for years and years.
    • kerpotgh 4y ago
      [dead]
  • Arrath 4y ago
    I think I have to agree. I much prefer a good stylistic approach over hyper-realism in most of my games. I always point to World of Warcraft, imo the style and 'cartoony' color palette not only looked visually appealing but helped patch over the low poly models and relatively low res textures as the game aged. I understand that recent expansions have given it an HD facelift, yet the style remains strong.
  • tshaddox 4y ago
    Expanding the capabilities of the tools seems important even if literally photorealistic results isn’t the goal of the creative team. It’s hard to go wrong by having the ability to render light in physically realistic ways, have realistic character animations, etc. It’s a bit like complaining about cameras with higher resolution and dynamic range because “I have zero interest in movies that simply portray how reality looks as accurately as possible.”
    • jfengel 4y ago
      It’s a bit like complaining about cameras with higher resolution and dynamic range because “I have zero interest in movies that simply portray how reality looks as accurately as possible.” That's exactly what happened. People complain about the way high-frame-rate cameras look cheap. The language of cinema is always unreal. Even the most "natural" performances are always highly stylized. Camera motion, lighting, makeup, etc are all very different from real life. You can see it even in behind-the-scenes footage; everything looks so fake. It's designed to feel right in the captured image, and that's the only goal. They'll use whatever tools they have for it. This might be a good tool, but it's hard to predict. They never really worked out a good use for 3D in film, even though it's also more "natural" than a moving 2D image. It's a different kind of uncanny valley: it didn't resemble real life enough, but you were content with the really unrealistic language of 2D cinema.
• kzrdude 4y ago
  Her first smiling mouth gestures in the playback definitely looked like the mouth of Wallace (from the clay animation in Wallace and Gromit), not entirely human.
• lproven 4y ago
  No, you're completely right. I was expecting something cutesy like Apple's animated animal avatar emoji things, but instead, it's near-live generation of more creepy frozen-faced fizzogs I've seen in various videogame demos -- and that rather strange Final Fantasy movie, 20Y or so back.
• megaman821 4y ago
  I noticed the weird mouth movements right away. You would think this is an area were a GAN would help a lot. Just reject movement patterns of the mouth that don't seem to match videos of real humans, even if the AI thinks it is capturing that motion of the mouth.
  • xapata 4y ago
    From the article, it sounds like they are using a GAN. The system was trained on a large database of facial expressions/movements.
• lucideer 4y ago
  > something about the way her mouth moves felt off To be honest, the way her mouth moves in real life felt very off. The "(over-)acting" during the capture was very unnatural, likely down to pressure of being on-stage as well as that just being a common problem for untrained actors doing anything scripted. Would be interested to see how it handles a capture of someone just engaging in natural conversation or something else less dramatic & exaggerated.
  • mecsred 4y ago
    I don't have a source for this information, I think it's something I remember from looking into this technology years ago. I believe actors are instructed to over-act on purpose, because it's better in the end to ham it up a bit to help the tracking system capture fine detail. If you risk the system losing the fine detail in your expression the end result can look a lot worse and reduce the number of useable takes.
    • Nevermark 4y ago
      It would be wonderful to allow overacting, to give the actor high dynamic control over nuance “production”, … and then be able to dial back the overacting automatically & manually for generation.
    • Groxx 4y ago
      You get similar advice pretty much any time you get near a stage, because the audience is far away and can't see subtle movements. Also they're generally talking louder than normal. I'm not sure that explains it all, but there are expected differences compared to how someone would move while talking normally.
• owaislone 4y ago
  I think she was told to do it on purpose to exaggerate the effect on screen.
• RajT88 4y ago
  I mean... It's a game engine tech demo by a video game company. So yeah. Maybe what you're getting at is "uncanny valley" - which I did get a little bit. But it's impressive how little I felt the valley for such a low-effort demo. I am sure with good actors and professional grade capture hardware this would be significantly less. What's really exciting to me is that this tech lowers the bar. Indie game developers are going to soon be have this sort of photorealistic quality to their characters.
• alexfromapex 4y ago
  I think the uncanny valley sensation would go away if they tweaked the material settings in Unreal Engine. Like her lips don't look shiny enough to me and maybe there's some light bounce issues overall. But it's super close to being convincingly real.
• dgunay 4y ago
  Mouth movements in particular tend to set off my uncanny valley senses too. I was recently playing God of War (2018), a very pretty game but when I paid close attention to the mouth movements as the characters speak it was noticeably low-fidelity (the movements are very coarse, and also had slight but perceptible rushing/dragging relative to the spoken audio). Compared to how good everything else looked, I found it slightly jarring. In my case I just try not to focus too hard on any one thing, and that helps me get immersed.
• kerpotgh 4y ago
  [dead]
• poundtown 4y ago
  i had the same thing with the mouth..it used to be the eyes ..now its the mouth. the shifting uncanny valley.