5 ms·
Highly realistic talking head video generation
- deleted 2y ago[deleted]
- xixixao 2y agoIt’s getting there, but you can still see that certain sounds are not realistically rendered. For example V in “vengence” the bottom lip has to touch the upper teeth, but in the renderings it’s only approximated.
- Anotheroneagain 2y agoI just have to think about the implications of going the other way round.
- Loughla 2y agoI don't know what that means. What do you mean?
- Anotheroneagain 2y agoLip reading. Video to speech.
- CoastalCoder 2y agoAutomated lip-reading? Ugh... paired with laws that restrict audio recording but not video recording.
- amelius 2y agoIt means that soon we'll all be wearing burqas to mask our lips.
- 123yawaworht456 2y agohave you overslept 2020-2022?
- amelius 2y agoI could never stand the ear-straps.
- BoredPositron 2y agoIt's still on the wav2lip[1] level from 5 years ago (look at the teeth) just slightly higher resolution. The only real player with a moat is probably flawlessai. [2] [1] https://github.com/Rudrabha/Wav2Lip https://github.com/Rudrabha/Wav2Lip [2] https://www.flawlessai.com/ https://www.flawlessai.com/
- HuiLi1998 2y agoActually, our work not only targets lip movements, it can also produce more realistic head movements and facial muscle movements. ps: It also achieved a good syncnnet score on 200 randomly collected wild image-audio pair. And it is open sourced.:)
- BoredPositron 2y agoI get exactly what you are doing but I don't understand what would be novel about it? It really looks just like a simple SD/wav2vec/insightface/animatediff pipeline, everyone can plug together in comfy or with diffusers. The muscle claim is also a bit dubious... PS: With insightface models in your reqs the oss aspect is also pretty much void for any use other than research and your readme should reflect that.
- lossolo 2y agoWhat's SOTA open source?
- Loughla 2y agoThe flawlessai is interesting, but even in their formal example for marketing there are weird tells. The actresses eyes are too jerky and weird during the changed sections. It's strange, but in a way that I'm not sure I could tag without the context of the rest of the video. Is this what we're in for? Human movement that is just normal enough to pass, but not really natural enough to be comfortable with?
- kevindamm 2y agoWe're sure to be in an uncanny valley for a little while first, the real question is whether it is followed by a winter or some of that sweet verisimilitude.
- smusamashah 2y agoI have found EMO (not open though) [1] to be the best yet. Look at the rapping example near the end. The lip sync is nearly flawless. The first black and white lady singing is also almost perfect. It even gives them the subtle jerk to pause for breathing. Unless you know and really are looking for flaws, you won't find anything that stands out making them look real. [1] https://humanaigc.github.io/emote-portrait-alive/ https://humanaigc.github.io/emote-portrait-alive/
- bamboozled 2y agoThe black and white lady is nightmare fuel for me personally.
- daveguy 2y agoAudrey Hepburn? Was she in a scary movie or play someone scary?
- ziggy_star 2y agoShe was a very graceful lady in wholesome movies. The juxtaposition of modern facial expressions of an influencer type singer covering Ed Sheeran at some X factor type television show are what makes it creepy. It is somehow doubly fake and extremely out of character if you are familiar with her.
- cess11 2y agoIn the synthetic video she looks like some kind of Frankenstein's monster, brought to life with electrodes or hidden motors, similar to the other video. Both 'move' in ways that are very unnatural.
- hammyhavoc 2y agoGlad it isn't just me.
- smusamashah 2y ago
- amelius 2y agoLooks like it allows only 1 reference image, so I'm not sure how realistic this is going to be in practice.
- crazygringo 2y agoPerhaps realistic physically, but not emotionally. It's truly bizarre to watch these talking heads because their lips are moving, but their eyes and cheeks aren't moving along with them, except for blinking. Real people speak with their whole face, not just their lips. Of course, to do that "right", you need to actually understand the emotional content of what is being spoken. And I'm not talking about highly "emotional" content like in TV drama -- even in a technical presentation, the speaker's face contains lots of emotional signals. Whether warmth, or a sense of humor, or being proud, or excitement of what they're about to reveal, or curiosity about whether the audience understands, etc.
- kleiba 2y ago> It's truly bizarre to watch these talking heads because their lips are moving, but their eyes and cheeks aren't moving along with them, except for blinking. Not true?! Please rewatch the video at the top: especially the eyebrows are quite animated, and the contours of the face also change as well as the shadows that indicate muscle movement in the phase.
- crazygringo 2y agoIt's just not looking real to me at all. Yeah there's a little bit of random movement, but none of the patterns of movement reflect the ways in which people's faces are actually expressive when speaking. They look kind of lobotomized, sure with maybe some random eyebrow raises thrown in. It's nothing like whole-face expression.
- ryandrake 2y agoI think the point of comparison should be news anchors and other "talking head" media, not real life humans emoting. Maybe it's just me, but people looking directly into and speaking in front of a TV camera about boring things also look lobotomized and uncanny. Check out your local newscasters some time and tell me they're really more realistic than these results.
- czl 2y ago
- add-sub-mul-div 2y agoThe AI slop shovelware spam posts will continue until morale improves.
- closedsans 2y agoThe most comforting thought I've had in a while is that AI just somehow fades away. I'm all for AI and use it regularly, but I can't help but feel uneasy about where this is all headed.
- m3kw9 2y agoLmao, it is not realistic at all, every movement is grossly exaggerated.
- deleted 2y ago[deleted]
- rhelz 2y agoI had logo on my Atari 800 back in '82. After having learned BASIC, it took a while for me to wrap my head around list-oriented programing. After a few weeks of beating my head against the manual, however, I got halfway good at it. Then, my little brother, 12 years younger than me, in 2nd grade, sits down and 2 days later he's programming circles around me. He could make that atari dance like Fred Astaire. And I was the one who wanted to be a computer programmer. He had no inclination in that direction at all, and became a businessman. I learned a very painful and embarrassing lesson about how learning the wrong programming language can give you brain damage.
- tiptup300 2y agoJust for level setting. A huge amount of people who may be users of a platform such as Facebook are very likely able to watch a video generated using a technology such as this and not consider it possibly to be fraudulent. Even if the content is somewhat strange or farfetched, these people, the majority of people are likely to not only "not notice" there's something off about the video, but believe what is being said. They are unlikely to act on the information or regurgitate it unprovoked, but likely to just remember it as a small fact they have received in the past.
- btbuildem 2y agoI've always thought this is where blockchain technologies can finally be of some practical use - as the age of Deep Fake fully unfolds. If an organization/entity releases media that is (by them) marked as "real" (or genuine, or trustworthy, original, not tampered with, etc), the stream can be checksummed and committed to a ledger. This way, any content coming from sources participating in this "verity scheme" can be validated against a distributed record (which is presumably incredibly difficult to modify surreptiously).
- idiotsecant 2y agoThat doesn't need block chain though, just a public key.
- btbuildem 2y agoHow would a public key suffice? Once the media has been published, the publisher could alter it, since they have the private key. I don't mean to encrypt the media, I mean to make a hash stream of the media stream, and store that in a way that cannot be adulterated.