12 ms·
Epic’s new motion-capture animation tech
- pappler 4y agoThe demo on that link. State of Unreal. There’s something unnatural about the facial movement… it seems overly exaggerated and forced.
- junon 4y agoPretty cool. Maybe someone more knowledgeable can explain why this is different than existing facial capture techniques? Is it just the same stuff repackaged into a quick and user friendly package?
- meh8881 4y agoYou can do it on an iPhone with no additional equipment and can apply it to a MetaHuman model, which is a relatively high quality human model that epic has tools to generate.
- dorkwood 4y agoIt’s worth noting that both the FaceCap and Moves By Maxon apps can also capture facial animation using only an iPhone with no additional equipment. It’s hard to tell from the video, but I wouldn’t be surprised if they used their face capture animation to drive wrinkle maps[1] on their MetaHuman models. This is something that other apps don’t offer out of the box, but which can significantly increase realism. [1] https://m.youtube.com/watch?v=nydjtjIncSk https://m.youtube.com/watch?v=nydjtjIncSk
- meh8881 4y agoWhat do you do with those app outputs though? I’m guessing it still feels hard because it won’t automatically map to a mesh that is good to go in your game engine? The MetaHumans drop in models are the thing that makes this magic imo. It’s not the motion capture tech so much as the complete pipeline to produce game assets.
- dorkwood 4y agoFaceCap gives you an FBX animated with about 50 different morph targets. If you wanted to use that in a game, you’d probably need to load all the morph targets into a shader yourself. I agree that the automatic integration with MetaHuman is the main benefit.
- nmfisher 4y agoActually you’ve already been able to do that for a few years now with Epic’s LiveLinkFace app (which was basically just a wrapper around Apple’s ARKit). This is just their own (much higher quality version) of it.
- meh8881 4y agoI’ve played around with it (the initial version). I didn’t think LiveLink was good enough personally.
- WheatMillington 4y agoI keep seeing people say this without explaining what the huge thing strapped to the actor's face is, that clearly isn't a phone. Not explained in the article either.
- meh8881 4y agoI’m not sure what you mean. There’s one shot in the video where they show a full motion rig and clearly explain that this system works with both a phone and/or a professional stereo system. I’m guessing that is for when you want to move your body around and capture the facial expressions associated with doing so. You need some sort of rig to keep the camera in your face if you’re moving your face.
- pzone 4y agoA few differences compared to other commercial iPhone-based capture techniques: - Whole-clip solve instead of noisy frame-by-frame streaming - Automatically generating a face model to calibrate the rig - Solving from raw sensor data instead of Apple ARKit pose data VFX studios typically develop proprietary solutions a few years in advance, so it can be hard to say what is truly "new and different."
- barrysteve 4y agoThis Bebylon presentation at SIGGRAPH 2018 is an example of the (now much older) way of doing the same iPhone face capture. https://www.youtube.com/watch?v=lXZhgkNFGfM https://www.youtube.com/watch?v=lXZhgkNFGfM The detail on the new Metahuman face capture is visibly better and a smoother capture pipeline has been a valuable goal for a some time. Noisy mocap and re-rigging animations to share them between models, takes time. [Quote from article] The algorithm uses a "semantic space solution" that Mastilovic said guarantees the resulting animation "will always work the same in any face logic... it just doesn't break when you move it onto something else."
- GeekyBear 4y ago>why this is different than existing facial capture techniques In the past, it was common to use video capture with reference markings on the actor so you could figure out how they were moving in three diminsions. From this demo, it looks like they are directly leveraging the 3D data from the iPhone's Lidar sensor in addition to the video camera data.
- junon 4y agoRight but that's been available for a while through various apps.
- mdale 4y agoThe moment the actor asks and "you only need to do this once for each actor?" seems to have a subtext of questions of actors future job security and odd combination of marketing latest tech and wtf am I going to do realization packed into a single presentation.
- bshipp 4y agoI have no dog in this fight, but perhaps I can offer a contrarian position. I hope people are able to look past how this technology will disrupt the status quo into the future of how such technology will enable much better cinematic story-telling by those without large studio budgets. Artists of the future will have opportunities to use lower-cost talent to enable their visions that will open up the floodgates of creativity, no different than what was seen when YouTube took off. Both actors and producers can benefit when access to talent is not held behind the gates of a few powerful entities. We've seen the damage such gatekeepers can wield with the Weinstein case and numerous subsequent claims, where vulnerable actors have their careers held hostage to avoid upsetting predators who wield all the power. If an actor can pay the bills by doing many short-term mocap jobs for independent producers armed with sophisticated software, everyone can benefit. In addition, motion capture opens up opportunities for very talented but otherwise less conventionally attractive actors who traditionally are relegated to type-cast bit roles.
- bkq 4y agoI consider the advent of this technology as whole, with the current state of Hollywood, to be concerning. It will ultimately lead to a world where actors will have their autonomy stripped away from them, even more so than they do now. Consider the status quo, whereby an actor's autonomy is somewhat determined by their start power. If you're trying to break into the industry then you have to put up and shut up with what you're given. With the advent of motion-capture like this, actors will be reduced to a marionette of flesh to be puppeted around as the producers see fit. Perhaps Hollywood will be going towards a future where actors no longer act in movies, instead their likeness is simply licensed to a studio for a certain number of movies. Anyway, I'm a cynic when it comes to this, the industry itself is already plenty abusive and exploitive, and this could further that.
- lofatdairy 4y agoVery impressive demo and use of the iPhone's depth camera. I suppose as long as FaceID exists there's no reason to try to use only RGB over RGB-D, though I am curious if they'll bring it to Android phones that have the required hardware like the Pixel.
- bundie 4y agoAlthough this is very impressive, don't get me wrong. I still cannot shake off that video-gamey feel the entire thing has. I watched the demo on YouTube and it looks great and all, but the moment the character began to speak, something about the way her mouth moves felt off. Am I the only one or am I just crazy?
- jjice 4y agoI wonder if we had the same feeling. I thought that the characters upper lip moved up too much during some of the more dramatic words.
- yehudalouis 4y agoI felt it too - but this seems like the kind of solution that will get you 90% of the way there, but still requires an animator to make it feel 'right'. This is insane though.
- londons_explore 4y agoI agree - the resulting look and feel just isn't quite right for a human, and it falls in the uncanny valley.
- vanilla_nut 4y agoThere's definitely something uncanny about the lips. I feel like the edges of the face are also more static than I expect (especially when she makes dramatic facial expressions like the "angry" expression, which ought to pull the skin a bit). Very video-gamey, not super convincing. But then again this is running a model in just a few seconds, with input from consumer hardware! Makes me wonder what you could do with more capable hardware (or if the iPhone hardware is really THAT good). It does make me think of one excellent application for this software: scanning your own face to make an in-game character who looks exactly like you. I assume I'm not the only one who's labored over character generators for hours to produce a decent-looking character. I think there's a neat opportunity here to get video game players to scan their own face as input instead.
- ambicapter 4y ago
- DDayMace 4y agoThis could really help low budget games as well. One person could do all the animations for low-poly models with text based dialog or generated voices.
- deleted 4y ago[deleted]
- ffhhj 4y agoWhy not capturing the hair motion as well? Let her have hair in front and move it aside by quickly turning the head and blowing it.
- UncleEntity 4y agoBecause hair is a whole other complicated subject.
- hoseja 4y agoYet another great technology that does nothing for gameplay and only furthers moviefication.
- ModernMech 4y agoYou're being downvoted but I feel the same exact way. I've been playing video games since pong, and through the years the 3D effects have really been used to bring the story to life. These days, the story is used to drive the 3D effects, and I'm just not here for it. I could already start to feel it in the Final Fantasy 7 days - like, yeah, it's really neat that the characters are 3D now for the first time, but why am I sitting through this 10 minute "Knights of the Round" casting animation every single battle? Let me get back to the things I found fun about the Final Fantasy series, which was fine in 2D pixel art. And yet, for some reason, the industry ran with those cutscenes. They got worse in FF8, and then today we see the end result. I mean, they really gave it away when they made Final Fantasy: The Sprits Within and they made such a big deal about how they did the hair. The whole movie was just to showcase their hair model. So now here we are in 2023, and we are playing the same games over and over again, but with better graphics every year. I'm sure this face capture tech or similar will make its way into all the next AAA games including the next Final Fantasy, which will have amazing hair and clothes and eyes and facial movements, but it will be the same game that's been made for 30 years. Thankfully, not all is lost. The best game released in decades is hands down in my opinion Dwarf Fortress, which took the graphics budget and put it toward innovative gameplay. Then there's Factorio which similarly isn't using AAA graphics, and could be called "retro". Of course Minecraft was a huge hit, and it had FFF graphics, only proving that really the graphics are next to meaningless if the gameplay is fun enough. Today there's a whole retro-gaming movement that's dedicated to breathing new life into consoles like the NES, but embracing the graphical limitations of the platform to focus on innovative gameplay and rich storytelling. The indy game scene is pretty strong to the point of being oversaturated really. I doubt it will cool down though with all these new AI technologies; now more than ever it's possible to take a game idea all the way to fruition with an AI helper generating assets for you.
- washadjeffmad 4y agoI'm also not a fan of the interactive movie genre. It's a weird niche, and I don't really know who it serves. Two decades ago, I would have generalized it to "console gamers", but I'm not so sure after the God of War / Elden Ring GOTY vote.
- dmix 4y agoThis would have big implications for AR/Google Glasses if the processing can be packaged like that.
- throwaway743 4y agoSlightly off topic, but does anyone know of any tools that convert realtime audio to lip animation for a 3d rig?
- spupy 4y agoNvidia Omniverse Audio2Face perhaps?
- MilStdJunkie 4y agoOn Blender, the cheap answer used to be Rhubarb addon, but there's probably something better now. Also also will probably depend on the rig and other accumulata.
- JustSomeNobody 4y agoThat's amazing. But there's still something discernibly ... off about it. But so so close.
- davidg109 4y agoWe’re getting there. Pair ChatGPT with this kind of technology, and you have a very impressive AI assistant.
- VoodooJuJu 4y agoI'd be interested to know, for the people who find this convincingly human-like, for those who are impressed, if you're on the autism/Aspberger's spectrum. I understand people on the spectrum can have difficulty reading facial expressions, so I'm curious if this is close enough for them that it appears real, or sufficiently human. As someone not on the spectrum, I can tell you that it's immediately obviously digital, fake, and gives me an uneasy feeling. It's that "uncanny valley" thing. So, I just wonder if the presenters, creators are on the spectrum, as they felt confident enough to showcase it and pass it off as sufficiently novel or convincing.
- renewiltord 4y agoWell, I'm not diagnosed as any of that and most people would describe me as popular and friendly in real life so it seems unlikely I'm undiagnosed but ill. And I thought it was bloody amazing. In such a short time, highly precise motion capture. This is tremendous technology. Looks fantastic!
- mrandish 4y ago> pass it off as sufficiently novel or convincing. IMHO, that's not the central point. It's notable not because it's super convincing but because it's convincing enough to be creatively useful while being incredibly fast, reasonably inexpensive and relatively easy. AAA game studios have already been able to reach similar (and even better) results for years. It's just super expensive and labor intensive. This demo is exciting because of how it can empower small game studios and even ambitious indies.
- iancmceachern 4y agoI can't wait to see what Corridor Crew does with this
- woefblaf123 4y agoThis is really impressive. I think part of the issue building the 'metaverse', is that content generation is just so incredibly labour-intense. I think visual AI generation and this kind of tech will really help a ton.
- zokier 4y agoTo me its weird that UE5 was originally revealed almost 3 years ago and there afaik aren't still any games using it (besides Fortnite), this despite one of the promises of UE5 was ease of development. I realize that the cycle times for games are long, but this still feels pretty extreme
- jayd16 4y agoIt officially launched last April...
- dougmwne 4y agoNo, that’s exactly it. Game development cycles are long. UE was officially released just 1 year ago. Projects take on risk if they are one of the very first adopters of a new engine since they will end up dealing with all the teething issues. Dozens of UE5 games are in active development.
- barrysteve 4y agoThere has been a lot of underutilized tech in the entertainment space. The summer blockbusters and AAA games later this decade might truly be a hair-raising experience.
- Capricorn2481 4y agoFrom what I understand, the real value add of this is getting motion capture close enough with a solid facial model, and being able to tweak facial animations from a few seconds of acting, with parameters automatically mapped. It's not meant to be an instant coffee animation.
- balozi 4y agoIsn't instant coffee the next logical progression in this tech roadmap?
- Capricorn2481 4y agoYes, but I'm noticing a lot of people complaining that it doesn't immediately look like a real person.
- nickpeterson 4y agoI think people are noticing the uncanny valley of these animation demos, but I was really struck by the fact that one good character actor could generate a lot of great animations that could be used across tons of models. I feel like this sort of tech mixed with recent breakthrough in AI is going to open up really crazy RPG open world games in the next few years.
- t0bia_s 4y agoUncanny valley effect is still there. Effort of doubling real life is unbelievable, but... I prefer stylized, artistically enhanced virtual reality.
- aschearer 4y agoClassic slacker news with all the negativity. This is incredible, wow! Stuff like this is why I got into tech in the first place, indistinguishable from magic.
- dougmwne 4y agoNo kidding! What a bunch of joyless grumps. This tech looks amazing and like it will bring mocap to the game development masses.
- heywhatupboys 4y ago> This is incredible it is video tracking of 30 points of the face, and then those points are extrapolated in a model. Seems like 20 year old tech tbh
- yazaddaruvala 4y ago20 years ago Peter Jackson had a whole team to do it, across months, with Andy Serkis in a specialized suit, on a specialized camera rig, with amazing lighting and green screens. All to map to a single animation model Golem. This was done on an iPhone, in subpar conditions, by 2 people, with regular clothes and background, for any/all humanoid models in under 90seconds!!! Amazing!!! It’s the only word for it and it doesn’t do the accomplishment justice.
- Groxx 4y ago>20 years ago Peter Jackson had a whole team to do it, across months, with Andy Serkis in a specialized suit, on a specialized camera rig, with amazing lighting and green screens. All to map to a single animation model Golem. ... and to do it outside while doing full-body motion capture while splashing around in streams next to other live actors on film. It's like comparing music festival hardware with home streamer equipment. Home streamer tech can sound significantly better, for a thousandth of the cost... because you can put up acoustic tiles. It's a big achievement, but wild comparisons don't do really do it any benefit.
- 4y ago
- caycep 4y agoI wonder if this sort of thing will become trivial w/ generative models like gans/llms and such. To paraphrase Arnold...If you can parameterize (facial expressions - already not too hard w/ motion capture software), you can generate it?
- sys32768 4y agoNow there's no excuse to not make an L.A. Noire sequel.
- angryasian 4y agois this already released for people to try out ?
- dirtyid 4y agoNeat. A few years away from sending fiverr to outsource acting to a developing country while face animation and voice gets synthesized.
- gedy 4y agoThis type of tech should be what the billions spent on the 'Metaverse' should be focusing on, not the creepy Wiimoji torso things.
- alexfromapex 4y agoWhat's amazing is that given another 10 years, with vastly superior hardware, this can happen in basically realtime.
- tippytippytango 4y agoActing will become remote work soon the way voice acting is. This’ll be a big deal for small creators and create a lot of acting opportunities.
- kaveh808 4y agoPhotorealism in CG is a milestone which is there, and can be strived for. It may be a necessary step to get through before interest in more interpretive styles truly takes off. Analogous to how modern art in painting didn't really take off until photography had made photorealism attainable and no longer interesting for painters.
- wkat4242 4y agoI hope this kind of thing will come to unity too. It's just a lot more accessible to beginners. And easier to work with for VR.