9 ms·
Eye Contact Correction: Redirecting the eyes to look at the camera
- vintagedave 2y agoThe results here in their sample video look _really good_: other tech I’ve seen in the past looked “wrong”. But the sample input is not one I’d characterize as looking away from the screen. Eyes move around like the person is thinking. The result video only looks more focused. It’s effective in carrying focus (it really does matter when someone looks directly at you), but it’s making tiny changes. > Limitations > Works best with frontal face views and moderate head rotations. > Extreme head poses or gaze directions may produce less accurate results. There it is. To use this I’d like to see an example showing it stop adjusting when “extreme” aka normal head poses are used. If it can handle real behavior and improve eye tracking in the optimal case so it’s seamless adjusting / not as someone moves around, that would be a good product.
- EdwardDiego 2y agoMy issue is the usual with laptop cameras, if I'm looking at you, my eyes are looking downwards, and it's very awkward speaking into the camera without seeing your face as I speak.
- AStonesThrow 2y agoThis is unfortunate, and perhaps more pernicious than obvious deep fakes, is a video filter that lies to the recipients. Several years ago during the pandemic, I enlisted a job coach to get me hired. One of her paramount concerns was my eye-contact with the camera. She said it's so important. Am I paying attention? Am I an honorable man who maintains eye contact when I'm in a conversation? If I look away, am I collecting my thoughts, or prevaricating? Many supervisors, managers, and teachers will judge their employees by whether they can pay attention during meetings, or if they're distracted, in their phone's screen, looking at keyboard, glancing off at children or spouse. Even more important, if you're meeting your wife and she can't even maintain your attention, what kind of husband are you? If you employ a gadget to lie about this, then I hope they fire you and find someone who'll be honest. I hope your wife sends you to sleep on the sofa.
- maximilianroos 2y agoSounds like the coach helped you maintain eye-contact with the camera. But if we get a tool to do this, then we're lying. Would you say the coach helped you lie?
- CGamesPlay 2y agoThat doesn't even make sense. The lie is that you're not doing the thing you are projecting as doing. You just said the coach helped the poster do the thing they projected as doing.
- karlgkk 2y ago> If you employ a gadget to lie about this This has been enabled on iPhones, by default, for like 5 years now. You never even noticed. Their implementation only does a small adjustment, which works so well that most people don't even know it's being done.
- olyjohn 2y agoIf we never noticed it, do we even need it? I don't use FaceTime, but have never been bothered by where people are looking in any other video conferencing software.
- bravetraveler 2y ago> You never even noticed I have seen three cameras in use in nearly a decade. They were all in interviews. I'm not avoiding opportunities, either. Legitimately 4+ hours a day Might be fair to say not many cared to see/be seen
- function_seven 2y agoI would go so far as to say the uncorrected gaze is a lie. When I’m on a videoconference, I am looking directly at whoever is speaking, but the camera’s physical placement tells the “lie” that I’m looking down at something else. This is because we haven’t figured out a good way of placing the camera literally wherever the eyes of the other party show up on the screen. So the camera is, by necessity, in the wrong position for video conferencing. But if we can fix it in software, then we can mitigate the “lie” somewhat. This is especially true for my set up, where I have two screens side-by-side with the camera replaced right between them. I just stare at the camera because otherwise it looks like I’m looking way off to the left or right. If I do look at the people who are talking, what they see is me looking off at “something else.” That’s a lie! :)
- Retr0id 2y agoDoes what it says on the tin, but honestly I find the "uncorrected" video more comfortable to watch.
- hanniabu 2y agoI think it's the lack of subtle movement, it's too strict and really locks the pupils front and center
- lloeki 2y agoYou mean frequent saccades? https://en.wikipedia.org/wiki/File:This_shows_a_recording_of_the_eye_movements_of_a_participant_looking_freely_at_a_picture.webm https://en.wikipedia.org/wiki/File:This_shows_a_recording_of... or the occasional look away? (for which there appears to be a feature for that) > Look Away: enable_look_away helps create a more natural look by allowing the eyes to look away randomly from the camera when speaking I expect both to be different: while saccades do happen when occasionally looking _away_ from a person, they also happen when looking _directly at_ one person because we don't constantly stare at a very specific unique and precise point on their face. For the demo video, try enable_look_away = true, look_away_offset_max = 10, look_away_interval_min = 1 and look_away_interval_range = 1 (then submit), which from the result I got should really be the default for a more natural result.
- karlgkk 2y agoThere are other implementations that do a better job, such as Apple and Google's. They also are less willing to correct eye contact when it's "out of range" so to speak.
- ddfs123 2y agoI think it's just that naturally nobody is keeping 100% eye contact ( except maybe like TV news reporter ), it feels like an interrogation.
- XorNot 2y agoThis is what I realized is uncomfortable about camera on group meetings in teams - I can't mute other people's video, and so it feels intensely weird to have a wall of people staring blankly at you.
- isuckatcoding 2y agoCool but why…?
- deepfriedchokes 2y agoVideoconferencing.
- karlgkk 2y agoApple does this on the iPhone, by default. When you're looking at someone's face on FaceTime, it modifies the position of your eyes to be looking directly at the camera - so the person on the other end sees you looking at them.
- s4i 2y agoReally? We FaceTime a lot with my wife and I keep telling her that I can see her looking at her own face in the corner instead or me. Is that tech accurate enough to tell that the person is looking at themselves and not the other participant, and then not correcting the eyes if that’s the case? Anyway, I’d much prefer if Apple didn’t silently alter the eye direction of people calling me.
- karlgkk 2y agoI think their adjustment is very minor and only happens when you're looking directly at the camera. It's minor enough that you've almost certainly seen it and not noticed. She may have the setting disabled, or she may be looking far enough away from the camera that it isn't triggering.
- deleted 2y ago[deleted]
- richdougherty 2y agoKudos to the dev for coming up with the eye position fixing solution. Building further on this idea, I wonder if instead of changing the image to look at the camera, we could change the "camera" to be where we're looking. In other words we could simulate a virtual camera somewhere in the screen, perhaps over the eyes of the person talking. We could simulate a virtual camera by using the image of the real camera (or cameras), constructing a 3D image of ourselves and re-rendering it from the virtual camera location. I think this would be really cool. It would be like there was a camera in the centre of our screen. We could stop worrying about looking at the camera and look at the person talking. Of course this is all very tricky, but does feel possible right now. I think the Apple Vision Pro might do something similar already?
- mvoodarla 2y agoThis is an interesting idea. We are a little farther off from being able to do this but agree it would look really cool.
- scotty79 2y agoI think you'd get a lot by just transforming eyes so the gaze is relative to the virtual camera located on the screen at the place of the face of a person you are talking to. This way you get eye contact only when you are looking on their face on the screen, but not when you look somewhere else.
- newaccount74 2y agoThere is already a lot of research on the 3D reconstruction and camera movement part, for example this SIGGRAPH 2023 paper: https://research.nvidia.com/labs/nxp/lp3d/ https://research.nvidia.com/labs/nxp/lp3d/ In order for this to work for gaze correction, you'd probably need to take into consideration the location of the camera relative to the location of the eyes of the person on the screen, and then correct for how the other person is holding the phone, and it would probably only work for one-on-one calls. Probably need to know the geometry of the phone (camera parameters, screen size, position of camera relative to phone) Would be amazing, not sure how realistic it is.
- xnx 2y agoNvidia has free Broadcast software with an eye contact feature: https://www.nvidia.com/en-us/geforce/news/jan-2023-nvidia-broadcast-update/ https://www.nvidia.com/en-us/geforce/news/jan-2023-nvidia-br... It's from January 2023, so I don't know if they've improved it further since then. The video conferencing software providers have been way to slow to put whoever is speaking top-center (near where the camera typically is).
- deleted 2y ago[deleted]
- zamadatix 2y agoThey also release it as an SDK these days so you can use it from tools like OBS https://www.nvidia.com/en-us/geforce/broadcasting/broadcast-sdk/resources/ https://www.nvidia.com/en-us/geforce/broadcasting/broadcast-...
- mvoodarla 2y agoOriginal dev here. That's right, NVIDIA has a version available which we reference in our blog. https://www.sievedata.com/blog/eye-contact-correction-gaze-correction-api https://www.sievedata.com/blog/eye-contact-correction-gaze-c... Newer models have come out that allow the same thing to be done and control even more than the eyes. See here: https://github.com/KwaiVGI/LivePortrait/blob/main/assets/docs/changelog/2024-08-19.md https://github.com/KwaiVGI/LivePortrait/blob/main/assets/doc... For web-conferencing, local use is great so NVIDIA's tools are what we recommend in that case.
- eisenman 2y agoI appreciated using the Nvidia Tools for remapping webcam eye-contact until I was reviewing a recording and noticed that it changed my eye color. But it’s been a bit. Perhaps an undocumented feature that newer versions/models fixed.
- Der_Einzige 2y agoThank you very much! - Signed, everyone whose currently trying to cheat on interviews that think that forcing peoples videos on does anything at all to keep them honest.
- boiler_up800 2y agoLooks really good and seems fast. My guess would be that this effect needs to be 99% or else people will notice something / although they may not be sure exactly what.
- thekevan 2y agoFrom a development standpoint, this is cool. But the resultant video has a tad bit of uncanny valley going on. I'd rather learn from the guy on the right.
- mvoodarla 2y agoOriginal dev here. Agree this video looks like uncanny valley but it's likely because the lighting of the original video is off + I baggy eyes (I was sleep deprived). Would recommend trying it on other videos, it is surprisingly good. Although there definitely are areas to improve.
- jpeggtulsa 2y ago10 cents per minute of video... Pass.
- albert_e 2y agoPractically -- I feel hardware technology can improve further to allow under-the-LED-display cameras .... so that we can actually look at both the camera and the screen at the same time. (There are fingerprint sensors under mobile screens now ...and I think even some front facing cameras are being built in without sacrificing a punch hole / pixels. There is scope to make this better and seamless so we can have multiple cameras if we want behind a typical laptop screen or desktop monitor.) This would make for a genuine look-at-the-camera video whether we are looking at other attendees in a meeting or reading off our slide notes (teleprompter style). There would be no need to fake it. More philosophically -- I don't quite like the normalization of AI tampering with actual videos and photos casually -- on mobile phone cameras or elsewhere. Cameras are supposed to capture reality by default. I know there is already heavy noise reduction, color correction, auto exposure etc ... but no need to use that to justify more tampering with individual facial features and expressions. Videos are and will be used for recording humans as they are. The capturing of their genuine features and expressions should be valued more. Video should help people bond as people with as genuine body lanuage as possible. Videos will be used as memories of people bygone. Videos will be used as forensic or crime scene evidence. Let us protect the current state of video capture. All AI enhancements should be marketed separately under a different name, not silently added into existing cameras.
- aitchnyu 2y agoWill we have video with sensor signature for evidence purposes? One high court in India rejected any video evidence as a potential deepfake.
- prmoustache 2y ago> One high court in India rejected any video evidence as a potential deepfake. Well I would have expected any court would have stopped accepting audio and video as evidence by now.
- agos 2y agothat would be a lot of baby throwing along the bathwater
- not_a_bot_4sho 2y agoI've never seen an implementation of this that wasn't super creepy past the initial tech demo
- lelandfe 2y agoiOS Facetime quietly does this by default, I don't notice it
- Applejinx 2y agoI put a monitor screen behind my camera for video-making, adjusted so the eyes are just barely showing over the camera. Then, when my eyes are drawn to me on the screen, I'm looking at the camera (just over it) which works pretty well. The eye contact is pretty good, but I can look away or around all I want: I'll just tend to be drawn to 'display of eyes' so I'm hacking that instinct to make better videos. That said, plenty of people don't make eye contact with the camera much at all :)
- throwaway290 2y agoI would pay for the opposite.
- advisedwang 2y agoThis is the real killer feature of Google's project starline, although they also achieve a 3D display.
- AyyEye 2y agoOnly a techbro would think that "eye contact" means just synthesizing eyes. It's a high bandwidth communication medium and synthesizing it removes what little we had. Yes I know this isn't the first, no I don't think any of this reality-meddling is any less creepy.
- patrickhogan1 2y agoReally cool application. Just a heads up – your main website is showing an error. You might want to fix it since your post is gaining traction. Here's the link: https://www.sievedata.com/ https://www.sievedata.com/ The error message reads: 'Application error: a client-side exception has occurred (check the browser console for more details).'
- mvoodarla 2y agoOriginal dev here. Unable to replicate this on my end, try refreshing?
- patrickhogan1 2y agoInteresting. The issue occurs because I have WebGL disabled, causing the createShader function you're using to throw an error. You can reproduce this by going to chrome://settings, disabling "Use hardware acceleration when available," refreshing the page, and then triggering the same error. Sorry for duplicate post. Also this feature is enabled by default, but causes issues with several sites.
- ta8645 2y agoGreat. Now, do the rest of me sitting in the seat. If you don't need my real eyes, you don't need any of the real me. We can discuss, whatever it is, in email.
- Bengalilol 2y agoI, for some reason, prefer the original video. I may have an eye contact problem. Otherwise, the feature is nice and almost perfect: there could be some spaces where eye contact shouldn't be always on, I bet this would make it more human.
- DidYaWipe 2y agoCreepy and misguided. Do people stare at you fixedly and unwaveringly during in-person conversations? And if they do, do you like it?
- lloeki 2y ago> Look Away: enable_look_away helps create a more natural look by allowing the eyes to look away randomly from the camera when speaking For the demo video, try enable_look_away = true, look_away_offset_max = 10, look_away_interval_min = 1 and look_away_interval_range = 1 (then submit), which from the result I got should really be the default for a more natural result.
- qwertox 2y agoUsually looking away is part of a gesture which involves the context, like facial muscles and the information being shared ("Hmm, when was this?": makes the eyes looks up)
- dTal 2y agoOkay, these options are far enough down the slippery slope to present a compelling argument that the whole thing is a Bad Idea. Short hop from here to suppress_yawns=true, and then breezing through enthusiasm_multiplier=1.4 on to enable_AI_avatar=true with bon_mot_interval=240s...
- lloeki 2y agoDid you watch the transformed video with the settings above? I think you misunderstand the role of "Look Away": it's not like it looks completely sideways, inventing behaviour that does not exist; instead it looks "away" _from the fixed point that would be dead-on camera center_ (that results in this "I'm gonna pierce through your skull with laser eyes" look), substituting it with "when looking - not aiming/scrutinizing - at something, even continuously, human eyes have saccades" The whole premise of such software (which has already been implemented by Apple in FaceTime with great success) is to _restore_ the reality which is "I'm looking at you but the mechanical offset between camera and window-on-screen destroys the information that I'm in fact looking at you", not invent something that is not real. Ideally it would even: - notice actual saccades and reproduce them, only cancelling the offset (super tough, so the next best thing is to fake it, but since these are small, uncontrolled, random-ish movements the approximation is quite sufficient) - take into account video window position relative to the camera so that if I'm looking away from the window then it stops compensating. But hey, first implementations are often naive. I give them credit for implementing Look Away because that's one step beyond the naive implementation. I guess it's not the default + tuneables are there because it's still early.
- blkhawk 2y agoDuring corona I build a fold down thing that put my webcam at eye-level on my monitor. turns out with a large enough monitor it really isn't that bad to have the camera in front of it.
- drewbitt 2y agoThere was a Kickstarter-esque product at the time that did this too. Tiny camera that dangles down in front of the screen.
- jedisct1 2y agoDoesn't FaceTime already do that?
- kleiba 2y agoIt would be great to have a feature where my closed eyes are replaced with open eyes looking at the camera - then I could sleep through boring meetings.
- chucksmash 2y ago> It would be great to have a feature where my closed eyes are replaced with open eyes looking at the camera There's already a feature that does this called HR
- ClassyJacket 2y agoIt's just a matter of time until half of us are having an LLM deepfake voice clone of us attend "meetings" so we don't have to.
- boomskats 2y agoThe thing with eye contact, though, is that it is worthless if you are never able to look away. When it's artificial like this, it's worse than not being there at all. It's just creepy. It was the same with nvidia's implementation a couple of years ago. It was just weird. I do appreciate that this is a problem worth solving though, and I spent a lot of my time during COVID worrying about the negative impact that normalising loss of eye contact would have on the social interactions of our younger generations. Back in 2021, I took one of those £50 teleprompter mirrors that YouTubers use, put a 7in raspberry pi display in the slot where you're meant to put your phone, and made it my 'work calls display' for a couple of days. The interesting thing is that the only people that noticed without me pointing it out were completely non-technical, and when they did they complemented me on the quality of my webcam rather than the fact I was looking straight at them; they could tell something was better, but couldn't quite put their finger on it. Which is funny because I'm sure being stuck behind a cheap perspex one way mirror made my actual camera quality a bit worse. I remember I got to the point where I started playing with cv2 trying to do realtime facial landmark detection on the incoming feed and having a helper process shift the incoming video window around the little screen so that it would keep the bridge of the other person's nose (the point I naturally made eye contact with) pinned to the bit of the screen that was directly in front of the webcam lens. Then one morning I walked into my office, saw this monstrosity on my desk, realised I was nerd sniping myself and gave up. One thing I do remember though is how odd it felt looking at yourself in a mirror without your image being mirrored. Not sure my brain was ready for that one after thousands of years of looking at itself in mirrored surfaces. Bit of a weird pic but the only one I can find: https://pasteboard.co/BXE6zhbpOD7E.jpg https://pasteboard.co/BXE6zhbpOD7E.jpg
- blitzar 2y agoI wanted to do this but got stuck in the rabbit hole of picking out telepromters, screens and sizes. In the end my solution was to mount my webcam in the middle of the monitor (with the other party partially obscured). Previously my technique was to look at the camera not the screen (or have the other party in a very small window at the top of my screen) so partially obscured is an improvement!
- lloeki 2y ago
- lemonad 2y agoI can definitely see the use case as it is annoying having to choose between actively looking at participants of a meeting on screen or _appear_ to look at the participants by gazing into the camera and not actually looking at them. I sometimes use an Elgato Prompter to better enable eye contact during meetings. The camera and lens is mounted behind the screen so looking at the screen is also looking at the participants. The downside is that the screen is tiny and you leaning forward to read, say, a document does not look that great on camera. So either you have to zoom it substantially or read it on another screen, thus looking away from the participants. In this case though, you are not looking at the participants and faking that eye contact in this case would be kind of weird.
- asdff 2y agoSomehow the idea of everyone looking at the camera to wave goodbye, while in the process only seeing the camera and not the people you are trying to make virtual eye contact with, is hilarious to me. Like some dystopian comedy.
- jbverschoor 2y agoIsn’t this built in FaceTime?
- qwertox 2y agoLooks somewhat creepy. The normal thing is not to uninterruptedly look at a person (which the camera is supposed to be). For example when you make a gesture of trying to remember something by looking somewhere else.
- 123pie123 2y agoit's definately in the uncanny valley for me he turns his head a little and his eyes look wrong
- EasyMark 2y agoThe creepy portrait that watches you no matter what part of the room you observe it from. For example the two in this jpg https://publish.purewow.net/wp-content/uploads/sites/2/2024/05/prince-charles-portrait-ghostbusters.jpg https://publish.purewow.net/wp-content/uploads/sites/2/2024/...
- nicholasbraker 2y agoI assumed this was a roadmapped feature on (at least) Facetime on OSX/IOS. I never saw any implementation of it, but I see value in such feature. Also for Teams etc.
- theodorton 2y agoIt's available in Settings > FaceTime > Eye Contact.
- nicholasbraker 2y agoThanx!
- froh 2y agowow that's creepy :-) technically cool, however I'd rather prefer some semi transparent mirror set up. such a set up keeps the eyes alive.
- rzzzt 2y agoDIY Perks has a build log on such an optics-based system: https://youtu.be/2AecAXinars https://youtu.be/2AecAXinars
- kmfrk 2y agoWe've come a long way from red eye correction. I think it's great that this is labelled as "correction" as in a means of optional postprocessing when it's convenient. Nvidia implying that it's something we should enable by default rubs me the wrong way, but then again, I don't spend my day stuck in virtual meetings.
- krosaen 2y agoThis is really cool, glad to see someone making a real product out of this - have seen demos in the past. One thing I've always wondered is if this could be made to work for group video chats - depending on the tile you are looking at, that person would know, so you could tell who is paying attention to you, or even exchange a furtive glance with a colleague in reaction to someone else said like IRL. Even harder, but also cool would be updating the gaze dynamically so you could tell what they were looking at in your scene - say you have a whiteboard behind you and you can tell when the person is making eye contact with you vs looking at something you drew on the board. Original dev make it so! :)
- Arch-TK 2y agoWhat is the modern obsession with video calls? I have been working for a company which allowed full remote work without any qualms since before COVID and nobody did video calls back then. Since we end up on site in secure environments we also just get told to disable the camera in the BIOS as part of our laptop hardening. For things like bi-annual meetings with your manager you would go into your local office.
- nonameiguess 2y agoI don't remotely get it, either. Most group calls I'm in involve one person presenting something and we're watching that, not each other. For things like a one-on-one call with my manager, we just talk. Shit, sometimes we even talk on the phone, but usually still a computer. He might be wearing no pants for all I care. I usually have a cat on me and am probably laying flat. If it's before 10 AM, because my wife goes to work so late, she's probably walking around naked behind me and nobody needs to be seeing that. My company was also 100% remote from its start, even before Covid. As others have stated, it's also just unnerving to have people making nonstop eye contact or staring at any part of your body at all, even if it isn't your eyes. Maybe 90s Los Angeles was an abnormally shitty place to grow up because of the gang activity, but this is the kind of thing kids started fights over all the time. Robert DeNiro's most famous movie scene is about how threatening it feels to have someone looking at you. This isn't even unique to humans. When you regularly interact with animals, you're taught to look away and not hold direct eye contact because they'll see it as a challenge or threat. I've learned to do this with my own cats to make them more at ease. You learn to blink, narrow your eyes, look to the side.
- asdff 2y agoNot only that but the obsession with higher and higher resolution video calls. Lets upgrade the laptop cams. No wait lets use a mirrorless camera and a ring light. Meanwhile the upload speed from your home isp is still early 2000s tier speeds and zoom downsamples you to 144p.
- schmichael 2y agoAlso a fully remote company worker since pre-COVID, and I vastly prefer calls with the video on. It's the norm at our company. Facial expressions convey enormous amounts of information. Also our team silently claps for people which is admittedly quite silly but hey, we like it. I'm rarely on calls without video, but when I am I find it jarring when voices just appear out of the ether with only a little flashing icon to indicate who it is I'm listening to. To each their own!
- neveroddoreven 2y agoI wonder how soon we'll just be rendering real-time AI avatars of ourselves that are traced by our facial movements. Don't have to worry about fixing your hair or lighting or wearing nice clothes; just render whatever looks most appealing with a model.
- alexsmirnov 2y agoSome decades ago, I worked with professional photo artists. Some did make photos for fashion magazines, one won "Word press photo" award couple times. When they make portraits, one rule is to always ask model to look slightly off the camera. As some explained, it adds some "life" to portrait. So, the tool does just the opposite to the art rules.
- EasyMark 2y agoPro-tip, put something interesting near wherever your camera is and look at that. I usually put the window itself (or whatever window is most interesting) up at the top center and that will get your eyes close enough to emulate “eye-contact”
- whaaaaat 2y agoLet me throw out an idea I haven't seen elsewhere here yet -- this corrects for a part of who I am in a way I find uncomfortable and negative. I'm autistic. I do not make eye contact easily. That's part of who I am, and a useful signal for the people I interact with that I might be neurodiverse. Something like this could mask or hide that autistic trait, and make me appear allistic. And, for some autistic folks, maybe that's desirable. However, I find the traits that I have around communication to be a deep part of me. Not making eye contact is nothing to be ashamed of, because being autistic is nothing to be ashamed. The introduction of tools like this could lead to pressure to use them to conform to some standard of "normalcy" that people expect. While the technical achievement here is neat (and other commenters pointed out places where it's struggling to look good), but there's a sort of meta impact that it can have on people who fall outside of normal ranges.
- deleted 2y ago[deleted]
- onemoresoop 2y agoThis is cool but I don't find it natural and a bit creepy to keep the gaze fixed at the camera at all times. Maybe they should add a threshold of variability and allow the gaze to look away from the camera from time to time.
- FabianBeiner 2y agoThere is a startup called “Casablanca.ai,” which appeared on the German version of Shark Tank (“Die Höhle der Löwen”), which provides exactly this as their offering: https://www.casablanca.ai/en https://www.casablanca.ai/en.