7 ms·

I once tried a few examples of eye-tracking as an input device at a visit to Tobii some years ago. One of these was a eye-tracking shooting game. It felt comple

I once tried a few examples of eye-tracking as an input device at a visit to Tobii some years ago. One of these was a eye-tracking shooting game. It felt completely wrong and unnatural. My eyes are my input devices. If I have to constrain my gaze and look only at the object to control then I would have to strain myself to perceive the blurry objects outside the centre of my vision just to be able to see those objects at all. IMHO, eye tracking should only be applied to make the experience feel more natural, not the opposite. For example: in combination with gesture controls to find the object that the gesture is intended for. Or in VR headsets to render at highest resolution only where the user is watching, to keep the FPS rate up.

mlajtos 4y ago
Yes, what you describe is the Golden Gaze problem (related to golden touch of the King Midas). There are several techniques mentioned in the article to mitigate the problem. Combination of gaze and hand gestures (and voice) seems like the most efficient and natural. Foveated rendering is also interesting, but that is pretty straightforward application of gaze contingency.
- macrolime 4y ago
  I think a combination of gaze and controllers (like the quest controllers) seem more natural, unless it's for AR and you're out and about.
- brobdingnagians 4y ago
  Talon allows using the Tobii with other triggers so that gaze does nothing without a trigger (a keyword or noise), which then zooms the area looked at, then allows using another trigger to perform the click.
acomjean 4y ago
Canon tried to use eye tracking cameras. Eg “look in the viewfinder where you want to focus”. It didn’t take initially (90s). But it’s back . https://m.dpreview.com/articles/6931257631/is-the-canon-eos-r3-s-eye-control-af-good-enough-for-sports-shooting https://m.dpreview.com/articles/6931257631/is-the-canon-eos-...
- ghaff 4y ago
  Interesting. I'll have to ask my friend who covers Canon to some degree about this. Yeah, they dropped it after trying it out in one of two film camera models in the 90s. They were always a bit cagey about why although, in my experience, it didn't work all that well.
  - CarVac 4y ago
    In the R3 it's used only briefly to select an initial autofocus point, after which it hands over control to the subject-tracking algorithms. From what I've read, for some it can be startlingly effective but outright refuses to work on other people's eyes.
psychstudio 4y ago
I used to work in eeg and eye tracking software (for neuro-motor rehabilitation and control of robotic arms). Using gaze for fine control is very difficult and terribly unnatural due to saccades and general scene scanning. One easy way to demonstrate how difficult it is is to have a cursor appear at the point of gaze. You're vision "chases" the cursor away from your target like it does a floater. Controlling anything directly with gaze is very very hard.
- jcims 4y ago
  I always find it odd how smoothly the eye can track something but voluntary movement is herky jerky.
  - formerly_proven 4y ago
    Ah you see these are different “hardware features” https://en.m.wikipedia.org/wiki/Smooth_pursuit https://en.m.wikipedia.org/wiki/Smooth_pursuit
- MichaelCollins 4y ago
  > One easy way to demonstrate how difficult it is is to have a cursor appear at the point of gaze. You're vision "chases" the cursor away from your target like it does a floater. That sounds like a slight miscalibration of the eye tracker. Assuming perfect eye control, if the tracker estimates your gaze slightly incorrectly then the cursor will always move away when you try to look at it.
  - brtkdotse 4y ago
    > Assuming perfect eye control No such thing. The eye does constant micro movements and the brain filters this out. You can compensate for this in software but it lowers the target accuracy.
- dopu 4y ago
  It’s no wonder that it takes us many months of training to get monkeys to perform simple saccade-based tasks.
atoav 4y ago
I mean ideally VR would use eye tracking for: - setting the focus plane of the virtual cameras, this way when you look at a close virtual object the far plane blurs as it would naturally. This is of course dependent on the brightness of the screen and picture. But any blurring will already feel better than no blurring at all - interactive stories can make use of knowing where you looked at and for how long. This must not necessarily be very obvious - virtual NPCs can react to you gazing at or past them
jayd16 4y ago
Yeah, agreed. Using eye tracking as input actually removes the ability to look and interact with different things. I really do not like it. Its pretty good for things like AI characters making eye contact or adding eye movement to avatars though.
modeless 4y ago
Tobii's UI stuff is lame. But it is possible to make good UI for eye tracking. It requires rethinking how UI should work. What everyone has been doing so far is like slapping a touchscreen on a desktop operating system. Doesn't work! You need to redesign all the interactions for touch, like the iPhone did. Change how scrolling works, how selection works, how menus work. Redo the foundations. Eye tracking requires the same treatment. Touchscreens sucked before the iPhone. Eye tracking can have an iPhone moment too, but someone needs to spend years rethinking everything first.
- roberttod 4y ago
  Is that conjecture, or are there UIs you have in mind that work? I made an eye tracking UI for my masters thesis, with a few different components. Although they all worked, having my eyes tracked to control the UI felt very intense, it caused a lot of mental and physical strain. I'm not saying that my thesis is evidence it absolutely can't be done, but it was surprisingly uncomfortable even for what seemed like basic functionality like for example autofocusing the input boxes you are looking at. But maybe there are tweaks that can make it work, I think the main issue is the screen giving feedback on where you happen to be looking, which gives this unnatural feeling that where you look is causing side affects.
  - modeless 4y ago
    I'd be interested to see your masters thesis! I made my own rudimentary eye tracker many years ago [1] which got me a job at a startup called Eyefluence, where I made a much better eye tracker. The deep learning revolution was very new at the time, and this was possibly the first deep learning based eye tracker. We did some really cool experimentation with eye controlled UI, in some ways more advanced than anything I've seen since. We were acquired by Google and I believe the technology is now shelved. But I still think that there is a path to a good eye controlled UI. Comfort is a matter of knowing the constraints of the eye and designing to them. https://james.darpinian.com/blog/eye-tracker https://james.darpinian.com/blog/eye-tracker
    - mlajtos 4y ago
cma 4y ago
> My eyes are my input devices. Eyes are very important in non-verbal communication so I don't think this is the full story and they are also used for output with deliberate control, but as far as something like being bad as a fine-grained cursor type thing it may hold.

mlajtos 4y ago

Yes, what you describe is the Golden Gaze problem (related to golden touch of the King Midas). There are several techniques mentioned in the article to mitigate the problem. Combination of gaze and hand gestures (and voice) seems like the most efficient and natural. Foveated rendering is also interesting, but that is pretty straightforward application of gaze contingency.

macrolime 4y ago
I think a combination of gaze and controllers (like the quest controllers) seem more natural, unless it's for AR and you're out and about.
brobdingnagians 4y ago
Talon allows using the Tobii with other triggers so that gaze does nothing without a trigger (a keyword or noise), which then zooms the area looked at, then allows using another trigger to perform the click.

acomjean 4y ago

Canon tried to use eye tracking cameras. Eg “look in the viewfinder where you want to focus”. It didn’t take initially (90s). But it’s back . https://m.dpreview.com/articles/6931257631/is-the-canon-eos-r3-s-eye-control-af-good-enough-for-sports-shooting https://m.dpreview.com/articles/6931257631/is-the-canon-eos-...

ghaff 4y ago
Interesting. I'll have to ask my friend who covers Canon to some degree about this. Yeah, they dropped it after trying it out in one of two film camera models in the 90s. They were always a bit cagey about why although, in my experience, it didn't work all that well.
- CarVac 4y ago
  In the R3 it's used only briefly to select an initial autofocus point, after which it hands over control to the subject-tracking algorithms. From what I've read, for some it can be startlingly effective but outright refuses to work on other people's eyes.

psychstudio 4y ago

I used to work in eeg and eye tracking software (for neuro-motor rehabilitation and control of robotic arms). Using gaze for fine control is very difficult and terribly unnatural due to saccades and general scene scanning. One easy way to demonstrate how difficult it is is to have a cursor appear at the point of gaze. You're vision "chases" the cursor away from your target like it does a floater. Controlling anything directly with gaze is very very hard.

jcims 4y ago
I always find it odd how smoothly the eye can track something but voluntary movement is herky jerky.
- formerly_proven 4y ago
  Ah you see these are different “hardware features” https://en.m.wikipedia.org/wiki/Smooth_pursuit https://en.m.wikipedia.org/wiki/Smooth_pursuit
MichaelCollins 4y ago
> One easy way to demonstrate how difficult it is is to have a cursor appear at the point of gaze. You're vision "chases" the cursor away from your target like it does a floater. That sounds like a slight miscalibration of the eye tracker. Assuming perfect eye control, if the tracker estimates your gaze slightly incorrectly then the cursor will always move away when you try to look at it.
- brtkdotse 4y ago
  > Assuming perfect eye control No such thing. The eye does constant micro movements and the brain filters this out. You can compensate for this in software but it lowers the target accuracy.
dopu 4y ago
It’s no wonder that it takes us many months of training to get monkeys to perform simple saccade-based tasks.

atoav 4y ago

I mean ideally VR would use eye tracking for: - setting the focus plane of the virtual cameras, this way when you look at a close virtual object the far plane blurs as it would naturally. This is of course dependent on the brightness of the screen and picture. But any blurring will already feel better than no blurring at all - interactive stories can make use of knowing where you looked at and for how long. This must not necessarily be very obvious - virtual NPCs can react to you gazing at or past them

jayd16 4y ago

Yeah, agreed. Using eye tracking as input actually removes the ability to look and interact with different things. I really do not like it. Its pretty good for things like AI characters making eye contact or adding eye movement to avatars though.

modeless 4y ago

Tobii's UI stuff is lame. But it is possible to make good UI for eye tracking. It requires rethinking how UI should work. What everyone has been doing so far is like slapping a touchscreen on a desktop operating system. Doesn't work! You need to redesign all the interactions for touch, like the iPhone did. Change how scrolling works, how selection works, how menus work. Redo the foundations. Eye tracking requires the same treatment. Touchscreens sucked before the iPhone. Eye tracking can have an iPhone moment too, but someone needs to spend years rethinking everything first.

roberttod 4y ago
Is that conjecture, or are there UIs you have in mind that work? I made an eye tracking UI for my masters thesis, with a few different components. Although they all worked, having my eyes tracked to control the UI felt very intense, it caused a lot of mental and physical strain. I'm not saying that my thesis is evidence it absolutely can't be done, but it was surprisingly uncomfortable even for what seemed like basic functionality like for example autofocusing the input boxes you are looking at. But maybe there are tweaks that can make it work, I think the main issue is the screen giving feedback on where you happen to be looking, which gives this unnatural feeling that where you look is causing side affects.
- modeless 4y ago
  I'd be interested to see your masters thesis! I made my own rudimentary eye tracker many years ago [1] which got me a job at a startup called Eyefluence, where I made a much better eye tracker. The deep learning revolution was very new at the time, and this was possibly the first deep learning based eye tracker. We did some really cool experimentation with eye controlled UI, in some ways more advanced than anything I've seen since. We were acquired by Google and I believe the technology is now shelved. But I still think that there is a path to a good eye controlled UI. Comfort is a matter of knowing the constraints of the eye and designing to them. https://james.darpinian.com/blog/eye-tracker https://james.darpinian.com/blog/eye-tracker
  - mlajtos 4y ago

cma 4y ago

> My eyes are my input devices. Eyes are very important in non-verbal communication so I don't think this is the full story and they are also used for output with deliberate control, but as far as something like being bad as a fine-grained cursor type thing it may hold.