7 ms·
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
- kevingadd 5y agoSurprisingly flexible algorithm. I threw a headshot illustration of a comic villain into it and it was able to successfully do all the rotations and even manipulate the eyes despite it not being photorealistic.
- thebruce87m 5y agoThere was an app going around a few months ago called “wombo.ai” that will make a headshot sing. It had the same outcome when given non human pictures as a source. Easily killed an afternoon trying out different things.
- villgax 5y agoThat was using something similar to First Order Motion Model & not the same model as what Nvidia has been using for Maxine or subsequent improvements to it as in the linked demo
- krylon 5y agoWow, this works pretty well. Makes me think of that chapter in Infinite Jest where videoconferencing gets popular, until people start using "optimized" computer-rendered images instead of showing their actual faces, at which point everyone goes back to audio-only.
- wongarsu 5y agoI wouldn't mind video conferencing with computer generated avatars. I don't video conference to know what the other person looks like, and in fact knowing what they look like just creates lots of unnecessary bias. I do it for the cues from their gestures, facial expressions, the direction they are looking, etc. With a good tracking setup that works perfectly well today with digital avatars.
- lwhi 5y agoI like faces. Why deny a key part of being human?
- dheera 5y agoIf you don't know why one might want this, you might be a white male, or at least white, or at least male, or at least a member of the majority race in your locality. I've definitely wanted this on a few occasions, to avoid being discriminated.
- batch12 5y agoSo following this logic train, the only actual faces one may see would be white males. Everyone else gets a mask?
- dheera 5y agoMaybe? But I'm going to get what I wanted.
- batch12 5y agoWould that then just be a proxy for not being a white male and present the same problems? I am not sure what this solves to prevent discrimination.
- dheera 5y agoIt doesn't prevent discrimination, it just allows me to do what I want to do in the short term, e.g. raise funding for startups or get dream jobs or whatever. I mean if a VC hands me a term sheet or a hiring manager hands me a job and the ONLY thing I misrepresented is my face, I don't think they have any ethical grounds to retract their offer. Solving the discrimination problem is another matter, and will take years if not decades, my dreams cannot wait for that.
- godelski 5y agoPermutation City has another take, where people virtually meet one another but use masks to hide their emotions.
- crazygringo 5y ago> at which point everyone goes back to audio-only That defeats the entire purpose of using facial and body expressions that only video provides. We already have video filters that remove wrinkles and blemishes in videoconferencing to make you look better. Even if we replace ourselves entirely with computer-rendered images, they're still going to be reproducing our expressions, movements and gestures, which is what matters.
- XorNot 5y agoMy phone already has a video call beautification setting built into the OS at the camera level. We've been skirting the line for a while. If I could, right now I absolutely would prefer to be sending a synthesized avatar then the real me - my desktop setup doesn't allow very optimal camera placement with large monitors, but for maximum impact I ideally want to send my face making direct eye contact with the camera.
- kbelder 5y agoIs there a filter available that just fixes the apparent direction of the pupil, so the image is looking at the camera, without doing any other edits? That would be really useful.
- arriu 5y agoI'm getting a "Failed to load resource: the server responded with a status of 500 (INTERNAL SERVER ERROR)". Looks like we broke it?
- knicholes 5y agoI'm getting connection refused errors on the request to http://54.186.34.220:443/face_vid2vid_rotate http://54.186.34.220:443/face_vid2vid_rotate port 443 on http? Hm.
- fulafel 5y agoInteresting that they're running plaintext HTTP on the HTTPS port. Are some networks filtering 80 on egress now and this is how people get around it? Or are the devs just using some cloud setup that tries to force serving HTTPS but the builders of the service don't want to?
- msoad 5y agoThis makes a request to server to get the result back. Hacker News hug of death has already happened. I wish this was deployable to browsers so it was fully stand alone.
- sundvor 5y agohttps://www.youtube.com/watch?v=xzLHZbBvKNQ https://www.youtube.com/watch?v=xzLHZbBvKNQ talks of the https://developer.nvidia.com/maxine https://developer.nvidia.com/maxine which requires GPU, so I'm assuming an RTX level GPU will be required for full motion? The demo calls an API on what appears to be a static IP which returns the end result using the algorithm. The full paper is on https://nvlabs.github.io/face-vid2vid/main.pdf https://nvlabs.github.io/face-vid2vid/main.pdf . (It only mentions GPU once, for the training set.) I'm quite impressed by how NVIDIA Broadcast cleans up a simple webcam image already, on a 3070 GPU; the background blur will get the gap between headphone bridge and head with sharp cuts - it's impressive enough in my books to warrant such a gaming grade GPU for work purposes, if a remote worker. I have my cam off to the side; I'm really looking forward to being able to try the angle correction!
- dheera 5y agoI wish they had done this client-side with e.g. Tensorflow.js. Would have been much more fun to play with.
- thedudeabides5 5y agoThe side by side against other algos is pretty cool: https://www.youtube.com/watch?v=nLYg9Waw72U&t=125s https://www.youtube.com/watch?v=nLYg9Waw72U&t=125s
- EthanHarv 5y agoWhile the client-side has input validation, it appears that the server-side does not, as I can edit the request body freely and it'll return accordingly. It's interesting to see how the model fails at extreme values. I can see why they chose the cutoffs they did!
- snorrah 5y agoNothing about this is nice.
- deleted 5y ago[deleted]
- derekhsu 5y agoUnable to upload custom photos and tried to use Chrome and Firefox but both failed either.
- koilke 5y agoInviting terms as usual: When you upload, submit, store, send or receive User Content to or through the NVIDIA Research AI Playground, you give NVIDIA (and parties NVIDIA works with, including its affiliates, suppliers and customers) a worldwide license to use (including without limitation for neural network training), host, store, reproduce, modify, create derivative works (such as those resulting from translations, adaptations or other changes), communicate, publish, publicly perform, publicly display and distribute such User Content. The rights you grant in this license are for the limited purpose of operating, promoting, and improving the NVIDIA Research AI Playground and content available to all users, and to develop new NVIDIA offerings. This license continues even if you stop using the NVIDIA Research AI Playground. The NVIDIA Research AI Playground may offer you ways to access, download, and remove content that has been provided, but make sure to keep your own back-up copies of your User Content. Also, the scope of services is limited and not all content in all formats can be loaded in the NVIDIA Research AI Playground.
- dillondoyle 5y agoIt doesn't sound like they are trying to sell or somehow leverage to make profit? I guess 'develop new NVIDIA offerings' is kind of vaguely suspicious but the rest doesn't seem onerous to me unless I'm missing something?
- kadoban 5y agoThose license terms seem to me like they'd allow using someone's picture in unlimited advertising, for free. They also sound like they'll definitely be using them to train models. I wouldn't be surprised if some of that means they can sell them to advertisers for whatever nonsense.
- midnitewarrior 5y agoMy take is that they are limiting their liability in case anyone is overly letigius, or if they mishandle the data. Yes, they could do the things you describe, but my guess is they don't want to get sued by people they are providing a free service to.
- GranPC 5y agoInteresting tool, but it's pretty ridiculous that the clientside JavaScript is obfuscated (not only minified) for what little it does.
- davbryn1 5y agoSeems to use the EXIF data to display the image, so if uploading an image from a portait image you might want to strip that or it breaks. Its pretty cool, but one thing that always bugs me - why is the demo page so bad? For the majority of people, this is the only way they are going to see this thing working - would it really have hurt to have had an actual frontend dev nock something together for this? It takes away from the great work behind the scenes imho
- ycosynot 5y agoIf you allow me, about such animated pictures, you should be interested in my pet project -- 1000x animated(!) fantasy avatar faces. 100% AI-Generated. Check it out, it's free and beautiful, it feels like next-gen. https://www.cgtrader.com/free-3d-models/character/fantasy/3d-faces-with-2d-portraits-x1000 https://www.cgtrader.com/free-3d-models/character/fantasy/3d... I made it shamelessly with https://www.fantasy-faces.com/ https://www.fantasy-faces.com/ for the GAN and myheritage.fr for animation -- it was still a lot of work to select 1000 of the most beautiful. Ultimately, I did it for the lulz...