5 ms·

I agree the keyboard will remain as the best text input device but I'm pretty confident that eye tracking will be categorically better than a mouse/trackpad. I'

I agree the keyboard will remain as the best text input device but I'm pretty confident that eye tracking will be categorically better than a mouse/trackpad. I'll expect that eye tracking solutions for normal monitors will also become more mainstream once people get used to it in the Vision Pro and want to use it elsewhere.

samwillis 3y ago
As we use an iPhone so close to our face, I would love to see an eye tracking system work for them. But maybe it really need that incredible close proximity, two cameras per eye, and IR illumination in the Vision Pro to work.
henryfjordan 3y ago
How do I click though? Blink? What about a right-click? Asking me to "blink twice" or anything like that is going to make my eyes lose focus on the screen/content
- samwillis 3y ago
  On the Vision Pro, tap your index finger and thumb together. Maybe they could add other features from just your face though for other platforms, wiggling nose, raise eyebrows, stick out tong, blow a raspberry? That or they add voice commands, "open", "select" and such. I suppose ultimately in spatial computing with voice interface and eye tracking the concept of a "click" may die.
  - crooked-v 3y ago
    There's also apparently other gestures and voice commands that can be selected in the settings, though I haven't seen any real detail on it other than "they're there".
- ladberg 3y ago
  The pinch gesture on Vision Pro is pretty smooth and easy, about as much physical exertion as a mouse click. Right click is harder, I'd expect most app developers to do something similar to iOS and treat long clicks as right clicks but that's pure speculation. I'd love to have an eye tracking setup though on a normal desktop computer where I could devote a keyboard key to clicking and I'd never have my hands leave the keyboard.
chrisrogers 3y ago
Eye tracking will probably lose the productivity metric, though. When you remove the ability to look at one thing and interact with another, you get farther from the way that humans tend to interact with their immediate physical environment. I can cut vegetables without looking at them. I can use that dynamic to offset the planning and acting phases of my thought process. Falling short of that efficiency will feel limiting.
- htag 3y ago
  I don't click without looking at my mouse pointer. I think it's possible eye tracking is just better than mouse/trackpad.
  - yurishimo 3y ago
    Interesting, I just tried to use my pointer with my peripheral vision, and while possible, it was indeed more difficult and harder to focus on. Something I hadn't tried before.
  - BakeInBeens 3y ago
    I often am looking at another element before interacting with it with my mouse. With eye tracking enabled menus hovering over an element immediately because I'm looking at it becomes an annoyance.
    - htag 3y ago
      Yea, I don't think a 1-to-1 translation of mouse movements with eye tracking is the correct answer here. I do think it's probable that eye tracking + {X} can be a 1-to-1 translation for clicking for 80%+ clicks. Maybe X is a button on the keyboard. Maybe X is a gesture.
  - CrazyStat 3y ago
    I do, when gaming. I can think of some Portal puzzles in particular where timing is important, and you need to hold your aim but wait to click until something happens somewhere else on the screen (so the place you're clicking is not the same as the place you're looking). I think the same thing applies to e.g. recording network activity in Chrome dev tools. My eyes are on the page to see when the thing I'm interested in finishes loading; my mouse cursor is on the button to stop recording. It's not a super common pattern, but probably common enough that it would be annoying not to be able to do it.
jasonpbecker 3y ago
I definitely want to see a monitor with Vision Pro-style eye tracking and hand tracking. I'd love for eyesight and subtle gestures to fully replace pointer devices.
UI_at_80x24 3y ago
I primarily use i3wm with 4-40 terminal windows + firefox open. I'd LOVE it if the terminal that I'm looking at was the `active` one. I can't count the number of times I'd lookup some error code, type in more commands, hit enter; only to discover that I had been typing away in the wrong window.
- sleepybrett 3y ago
  This will be no problem at all in the sdk. Hell I saw someone do this as some point on the mac using the macbook camera popping windows to the front based on eye-dwell... I think they started with cursor control.. essentially eye-mouse. Oh wait some version of that is built into accessibility on mac already (eye tracking mouse): https://support.apple.com/lv-lv/guide/mac-help/mchl437b47b0/mac https://support.apple.com/lv-lv/guide/mac-help/mchl437b47b0/...
mrguyorama 3y ago
Eye tracking has been $100 to own for many years now through a certain company. Why hasn't it taken off? The mouse is the superior input device. When people who actually need a better input device get one, they get more advanced mice: https://3dconnexion.com/us/spacemouse/ https://3dconnexion.com/us/spacemouse/
- ladberg 3y ago
  I'm not sure what the SOTA is currently but the Vision Pro has much better eye tracking that other systems I've used in the past. Idk if the sensors are actually better or if it's just the native integration into the OS and apps, but it feels more seamless than a mouse and I've never "misclicked" on the wrong thing with my eyes. It kinda feels like if the mouse cursor instantly jumped to the right place every time with no travel time or physical movements required. I've used a 3D mouse for CAD but am not sure where else it would be helpful?
  - mrguyorama 3y ago
    Tobii https://gaming.tobii.com/product/eye-tracker-5/ https://gaming.tobii.com/product/eye-tracker-5/ is standard in consumer eye tracking, they even have integration with some popular video games! There's just no value in it. I tried an older version in a store, and it was basically perfect, tracking my eye's movements exactly, with zero delay. Nobody wants it for day to day computer interaction. Most people using eye tracking for computer interaction are disabled, because it's a terrible experience.
throwuwu 3y ago
Voice input is going to replace the keyboard shortly.
- frant-hartm 3y ago
  Sure, just right after Linux takes over desktop.

samwillis 3y ago

As we use an iPhone so close to our face, I would love to see an eye tracking system work for them. But maybe it really need that incredible close proximity, two cameras per eye, and IR illumination in the Vision Pro to work.

henryfjordan 3y ago

How do I click though? Blink? What about a right-click? Asking me to "blink twice" or anything like that is going to make my eyes lose focus on the screen/content

samwillis 3y ago
On the Vision Pro, tap your index finger and thumb together. Maybe they could add other features from just your face though for other platforms, wiggling nose, raise eyebrows, stick out tong, blow a raspberry? That or they add voice commands, "open", "select" and such. I suppose ultimately in spatial computing with voice interface and eye tracking the concept of a "click" may die.
- crooked-v 3y ago
  There's also apparently other gestures and voice commands that can be selected in the settings, though I haven't seen any real detail on it other than "they're there".
ladberg 3y ago
The pinch gesture on Vision Pro is pretty smooth and easy, about as much physical exertion as a mouse click. Right click is harder, I'd expect most app developers to do something similar to iOS and treat long clicks as right clicks but that's pure speculation. I'd love to have an eye tracking setup though on a normal desktop computer where I could devote a keyboard key to clicking and I'd never have my hands leave the keyboard.

chrisrogers 3y ago

Eye tracking will probably lose the productivity metric, though. When you remove the ability to look at one thing and interact with another, you get farther from the way that humans tend to interact with their immediate physical environment. I can cut vegetables without looking at them. I can use that dynamic to offset the planning and acting phases of my thought process. Falling short of that efficiency will feel limiting.

htag 3y ago
I don't click without looking at my mouse pointer. I think it's possible eye tracking is just better than mouse/trackpad.
- yurishimo 3y ago
  Interesting, I just tried to use my pointer with my peripheral vision, and while possible, it was indeed more difficult and harder to focus on. Something I hadn't tried before.
- BakeInBeens 3y ago
  I often am looking at another element before interacting with it with my mouse. With eye tracking enabled menus hovering over an element immediately because I'm looking at it becomes an annoyance.
  - htag 3y ago
    Yea, I don't think a 1-to-1 translation of mouse movements with eye tracking is the correct answer here. I do think it's probable that eye tracking + {X} can be a 1-to-1 translation for clicking for 80%+ clicks. Maybe X is a button on the keyboard. Maybe X is a gesture.
- CrazyStat 3y ago
  I do, when gaming. I can think of some Portal puzzles in particular where timing is important, and you need to hold your aim but wait to click until something happens somewhere else on the screen (so the place you're clicking is not the same as the place you're looking). I think the same thing applies to e.g. recording network activity in Chrome dev tools. My eyes are on the page to see when the thing I'm interested in finishes loading; my mouse cursor is on the button to stop recording. It's not a super common pattern, but probably common enough that it would be annoying not to be able to do it.

jasonpbecker 3y ago

I definitely want to see a monitor with Vision Pro-style eye tracking and hand tracking. I'd love for eyesight and subtle gestures to fully replace pointer devices.

UI_at_80x24 3y ago

I primarily use i3wm with 4-40 terminal windows + firefox open. I'd LOVE it if the terminal that I'm looking at was the `active` one. I can't count the number of times I'd lookup some error code, type in more commands, hit enter; only to discover that I had been typing away in the wrong window.

sleepybrett 3y ago
This will be no problem at all in the sdk. Hell I saw someone do this as some point on the mac using the macbook camera popping windows to the front based on eye-dwell... I think they started with cursor control.. essentially eye-mouse. Oh wait some version of that is built into accessibility on mac already (eye tracking mouse): https://support.apple.com/lv-lv/guide/mac-help/mchl437b47b0/mac https://support.apple.com/lv-lv/guide/mac-help/mchl437b47b0/...

mrguyorama 3y ago

Eye tracking has been $100 to own for many years now through a certain company. Why hasn't it taken off? The mouse is the superior input device. When people who actually need a better input device get one, they get more advanced mice: https://3dconnexion.com/us/spacemouse/ https://3dconnexion.com/us/spacemouse/

ladberg 3y ago
I'm not sure what the SOTA is currently but the Vision Pro has much better eye tracking that other systems I've used in the past. Idk if the sensors are actually better or if it's just the native integration into the OS and apps, but it feels more seamless than a mouse and I've never "misclicked" on the wrong thing with my eyes. It kinda feels like if the mouse cursor instantly jumped to the right place every time with no travel time or physical movements required. I've used a 3D mouse for CAD but am not sure where else it would be helpful?
- mrguyorama 3y ago
  Tobii https://gaming.tobii.com/product/eye-tracker-5/ https://gaming.tobii.com/product/eye-tracker-5/ is standard in consumer eye tracking, they even have integration with some popular video games! There's just no value in it. I tried an older version in a store, and it was basically perfect, tracking my eye's movements exactly, with zero delay. Nobody wants it for day to day computer interaction. Most people using eye tracking for computer interaction are disabled, because it's a terrible experience.

throwuwu 3y ago

Voice input is going to replace the keyboard shortly.

frant-hartm 3y ago
Sure, just right after Linux takes over desktop.