13 ms·

Eye-tracking is a missing input device for VR experiences

134 points by mlajtos 4y ago

hoseja 4y ago
This would be worth it for foveated rendering alone.
- moron4hire 4y ago
  Turns out foveated rendering is not the panacea it was imagined to be. Eye saccades are so fast that there's just too much motion-to-photon latency to do it well. https://twitter.com/ID_AA_Carmack/status/1391869530327052291 https://twitter.com/ID_AA_Carmack/status/1391869530327052291
presentation 4y ago
Sounds tiring to me, I already get fatigued looking down to make the iPhone’s Face ID happy.
- xattt 4y ago
  I beg to differ. Canon marketed several film and video cameras with eye-tracking for focus. I remember it being a very intuitive interface. Considering this through a modern perspective of inclusivity, there are probably issues around accessibility for people with eye-mobility issues (amblyopia, stroke-related impairments, etc). (1) https://m.dpreview.com/articles/6531126959/looking-back-canons-eye-controlled-focus https://m.dpreview.com/articles/6531126959/looking-back-cano...
  - ghaff 4y ago
    They appeared during the late days of film EOS models but went away and never returned. I've never heard a good explanation. I even asked a friend of mine who sometimes covers camera tech for one of the big online pubs and Canon would never give an explicit reason.
    - Auracle 4y ago
      They’re back now, actually.
      - ghaff 4y ago
        Yeah. Someone else mentioned that you can apparently use eye tracking to set an initial focus point in the R3. And it apparently works well for some and not at all for others.
m00dy 4y ago
I think a comfortable vr headset is the missing device for VR experiences.
- PretzelPirate 4y ago
  I find the HP Reverb G2 to be incredibly comfortable and can wear it for hours. I do have an air conditioning unit in my VR room which keeps me cool while moving during that time. The Quest 2 is the lest comfortable thing I've ever put on my face and due to its popularity, seems to skew perspectives around how comfortable VR can be.
- mlajtos 4y ago
  Agree 100%. I am curious what fruit company has in their sleeve.
  - ohgodplsno 4y ago
    An overpriced device that doesn't run any games nor has developers for it ? The Valve Index showed that as long as your VR headset is $1000 and above, barely anyone will buy it. And that's without taking into account the hardware you need. If anyone has a chance to succeed this generation, it's Sony with their PSVR. And even then, it's $500, and you need another $150 for their headset.
    - mlajtos 4y ago
      Standalone headsets (like Meta Quest line) are much more interesting then PCVR which requires beefy PC, base stations and lot of cables and setting up. Paying $1000+ for a standalone headset might sound too ridiculous but when you understand that it is the whole computer, it is much more acceptable.
      - ohgodplsno 4y ago
        And you expect Apple to put out a $300 device like the Quest ? Because this is the main subject. Paying $1000+ for a standalone headset, even if you can play Half Life Alyx in 4K is still not something that will reach any proper market adoption. Unlike a phone, which you can use at all times, a VR device requires a ton of space, requires buy-in from developers (Apple's current list of games and apps that support VR is... right, zero. Planning to run your game on Rosetta + MoltenVK ?), etc. It's not the whole computer. It's a computer that you can't really bring outside, that is heavily limited because haptics are still terrible (and will stay terrible). Do you expect people to bring their VR headset to starbucks ?
oneoff786 4y ago
Rye tracking would be great. But you’ll never have ready player one quality experiences with a headset at all
- solardev 4y ago
  The gluten-free movement has gone too far.
lagrange77 4y ago
While thinking about it, i realized, that our vision apparatus has an interesting feature: When you move your head, your field of sight moves with that motion. But when you hold your head still and only move your eyeballs, the field of sight stays constant, while the focus point (gaze point?) changes, scanning the still standing field of sight. Imagine replacing the eyeballs along with their mounting and motion with cameras. Every motion would translate the field of sight as a whole. Anyone knows, how this mechanism is called?
- baxtr 4y ago
  Damn. Now that I did that experiment I was reminded of my eye floaters. Thanks for that... My day is ruined. /s https://en.wikipedia.org/wiki/Floater https://en.wikipedia.org/wiki/Floater
  - lagrange77 4y ago
    Good ol' faithful friends
- GistNoesis 4y ago
  In the radio domain, this is called beam-forming.
- xeonmc 4y ago
  gimbal? And the reflex for keeping the gaze fixed is oculovestibular reflex
  - lagrange77 4y ago
    As a layman, i would say the gimbal mechanism merely lets you lock focus on some global coordinates, while moving your head around, but maybe i'm wrong.
- numpad0 4y ago
  Mind blown. Indeed: - when I move my head to (+30, +45), my brain reports viewport.LeftTop =(0,0), and, - when I move my eyes to (+30, +15), my brain reports viewport.LeftTop = (30, -15). But our eyes are built in such way that the whole lens and sensor assembly rotates, rather than having sensor fixed to the head and lens rotating, so it makes no sense. Framebuffer should always span from (0,0) to (size.x, size.y). Perhaps the framebuffer always covers the full viewing range, and DMA start addresses are dynamically determined by current eye positions, or perhaps even by feature matching?
atemerev 4y ago
I have thought so before, but in my experience, wide field of vision works fine so you can just look at different spots. However, there is no focus/accommodation tracking in VR, which makes your experience hyperreal (everything is always in focus). Incidentally, it is very close to lucid dreaming, where you also see everything at once.
- mlajtos 4y ago
  I have never realized the similarity between VR and lucid dreams is because of missing depth. Thank you for this. :)
Findecanor 4y ago
I once tried a few examples of eye-tracking as an input device at a visit to Tobii some years ago. One of these was a eye-tracking shooting game. It felt completely wrong and unnatural. My eyes are my input devices. If I have to constrain my gaze and look only at the object to control then I would have to strain myself to perceive the blurry objects outside the centre of my vision just to be able to see those objects at all. IMHO, eye tracking should only be applied to make the experience feel more natural, not the opposite. For example: in combination with gesture controls to find the object that the gesture is intended for. Or in VR headsets to render at highest resolution only where the user is watching, to keep the FPS rate up.
- mlajtos 4y ago
  Yes, what you describe is the Golden Gaze problem (related to golden touch of the King Midas). There are several techniques mentioned in the article to mitigate the problem. Combination of gaze and hand gestures (and voice) seems like the most efficient and natural. Foveated rendering is also interesting, but that is pretty straightforward application of gaze contingency.
  - macrolime 4y ago
    I think a combination of gaze and controllers (like the quest controllers) seem more natural, unless it's for AR and you're out and about.
  - brobdingnagians 4y ago
    Talon allows using the Tobii with other triggers so that gaze does nothing without a trigger (a keyword or noise), which then zooms the area looked at, then allows using another trigger to perform the click.
- acomjean 4y ago
  Canon tried to use eye tracking cameras. Eg “look in the viewfinder where you want to focus”. It didn’t take initially (90s). But it’s back . https://m.dpreview.com/articles/6931257631/is-the-canon-eos-r3-s-eye-control-af-good-enough-for-sports-shooting https://m.dpreview.com/articles/6931257631/is-the-canon-eos-...
ElCheapo 4y ago
*look down to pay respects*
Daub 4y ago
I worked on an eye tracking project that compared the difference between the way artists and non artists look at images. Very interesting. Non artists tends to dwell on faces, hands and other evidently interesting things. Artists find interest in less obvious thing… odd angular protrusions, unexpected colors etc. It also showed me how active and (frankly) jittery all human vision is: never resting, scanning. Also... unexpectedly, we seem to evaluate objects according to their features (points, blobs, angles).
epakai 4y ago
Fove [0] built a headset with eye tracking features. I had a senior project trying to use it for visual field testing in 2017. [0] https://www.kickstarter.com/projects/fove/fove-the-worlds-first-eye-tracking-virtual-reality https://www.kickstarter.com/projects/fove/fove-the-worlds-fi...
ohgodplsno 4y ago
As said in other comments, eye tracking to aim at things would be absolutely horrendous. Try using a mouse on maximum sensitivity, and you'll see the kind of accuracy you'll need. However, eye tracking does give you one thing (that is already getting into VR devices today, and that's foveated rendering. Increasing resolution where needed and lowering it elsewhere. Couple this with various techniques to turn on fewer pixels [https://www.immersivecomputinglab.org/publication/color-perception-guided-display-power-reduction-for-virtual-reality/ https://www.immersivecomputinglab.org/publication/color-perc...] and save up even more on performance/energy usage, and you kind have pleasant, high refresh rate experiences.
- thedorkknight 4y ago
  The article talks about foveated rendering
  - ohgodplsno 4y ago
    The article barely mentions foveated rendering, in the middle of a sea of "use your gaze to click it'll be great I promise".
    - mlajtos 4y ago
      Foveated rendering should be an invisible technology — the image you see should be perceptually indistinguishable from the fully rendered image. How to get there is an interesting engineering problem, but for me that isn’t an interesting domain. You summed up the article quite good, but I don’t claim it will be great. I just think that gaze as an input device is an underresearched topic. Ken Pfeuffer’s research [0] is beyond awesome, but we need masses of thinkerers to explore the domain more. New wave of VR headsets will enable it. [0] https://kenpfeuffer.com/publications-2/ https://kenpfeuffer.com/publications-2/
blululu 4y ago
I’ve heard a lot of these claims before and seen them fall flat when you actually start building it. Things like foveated rendering that many (the author included) claim will lead to better graphics and lower power just don’t add up. You need at least two additional cameras running a pretty compute heavy ml algorithm to track eyes and this costs more watts than just rendering the full screen as your gpu intended. As an interface it can be quick but don’t underestimate that it is one really fucking annoying to get targeting feedback on everything you look at (which sucks for dwell oriented tasks like reading). Finally there is the elephant in the room that this tech works somewhat well for people from Northern Europe, but gets super shaken for a wider swath of the global population. Maybe the author has seen some demos that were better than what I’ve seen, but this all seems like the project pitches that I’ve seen for why we need to explore this area and no the final report.
- LordHeini 4y ago
  That is completely wrong. For foveated rendering one could easily build dedicate hardware into the device which would require almost no additional power. And those algorithms are not that expensive to begin with. Every camera has face tracking build in (which basically is the same problem) and that does not drain the battery either. And why should northern Europeans eyes work differently? Even if that would be true one could add a bunch of settings for personalisation. Eye distance for example is a common setting on dodays devices...
  - blululu 4y ago
    Eye trackers typically require 2-4 additional cameras trained on the eyes. The algorithms for eye tracking are typically pretty involved, but maybe you know some very lower power algorithm to track the eye at 120 hz? I'm not sure how dedicated hardware will reduce the power consumption for 2 cameras and an ML pipeline to almost nothing. Northern Europeans have very light irises. It makes it easier to track the pupil from a single signal-to-noise perspective. But there are a ton of weird ethnic challenges for eye tracking. Things like reflectivity and moisture of eyes varies a lot across different across different ethnic groups. Personalization add complexity and will only take you so far when you are dealing with physical sensing challenges.
sys_64738 4y ago
What if I have a lazy eye? Isn't that discrimination?
- micromacrofoot 4y ago
  no
- thedorkknight 4y ago
  At worst you're just back to a regular VR display in that case, so unless current non-eye-tracking headsets are discriminatory, no. Plus I would think it would still be able to use your non-lazy eye
- lm28469 4y ago
  Everything is a discrimination to someone if you start with such crazy takes
- pessimizer 4y ago
  What if I'm blind? Is having any visual component to VR bigotry?
  - sys_64738 4y ago
    This is why you have Americans with disabilities acts forcing accessibility for those deprived of all sensory and motor function.
twirlock 4y ago
picsao 4y ago
Philip-J-Fry 4y ago
PSVR 2 has eye tracking. So, we'll be seeing more usage of it.
k__ 4y ago
Could be used for games like Silent Hill or Amnesia for effects that only happen if you don't look directly at them.
- mlajtos 4y ago
  Weeping angels from Doctor Who
oblak 4y ago
> Eye as a Pointing Device As a long time gamer and VR (Valve Index) owner, this is where I stopped reading. No, no, no. As others have said, we could use gaze tracking for plenty, just not pointing devices. People have been trying this approach for decades and it's always been the wrong one. It could work for disabled people/animals, but it should not be the main pointing device. Not my kind of article, that's for sure
- glaslong 4y ago
  This exactly. It’s terrible as a precision pointer, but great as an attention heuristic. A strong secondary intent signal.
  - pwython 4y ago
    What came to mind first was sniping in games like Counter Strike. Your eye is generally within the crosshair aiming at a certain spot, but suddenly someone comes in your peripheral and you have to make a quick flick shot, which requires more muscle memory and not even focusing on the target.
- seejayseesjays 4y ago
  Exactly. I've got Nystagmus so I reckon that if the eye tracking is particularly accurate, it'd be like trying to control a mouse that uncontrollably for me.
- sli 4y ago
  I only want eye tracking on my Index for VRChat integration so my avatar's eyes move with mine. I just cannot imagine using my eyes as some kind of input device, that sounds immensely awkward for someone that has more traditional options (i.e. no physical disabilities). I'm not using VR for productivity in any case, anyway.
jrm4 4y ago
Speaking of VR, it really feels like the motion sickness part is still woefully underacknowledged. I've enjoyed quite a few VR games (I have access to most of the systems at my job) but there's no way I can spend more than e.g. 30 min a week "in there." The body REALLY REALLY REALLY hates when movement and what you see don't match up, and very few experiences deal with this well and also are really compelling.
- fknorangesite 4y ago
  > underacknowledged Because it doesn't affect everyone to the same degree. What you describe sounds really severe (and I don't doubt it) - but for example I'm personally way at the other end of the spectrum: I can easily spend hours in VR, and I can think of only one game that ever made me even slightly motion sick.
- status200 4y ago
  I had the opposite experience, my family has a history of motion sickness, but myself and other family members that have tried it have felt really natural in VR. Even my sister could use smooth movement and "advanced" settings, which I never would have predicted. Our best guess is that the 6DoF system at least matches head movement, which might be the difference that makes our brain(s) not freak out compared to reading in the car, etc.
  - ShamelessC 4y ago
    The truth is that it's highly subjective to each individual and simply because it has gotten better for some, doesnt make it a solved issue (and still makes it a non-starter for people effected).
- apike 4y ago
  I’m susceptible to this too, but I believe the two reasons VR motion sickness is much less discussed nowadays are: 1. Often it’s triggered by frame rate hitches, which seems solvable with the usual advance of software and hardware over time. 2. Most folks can train their mind and body out of the motion sickness with practice.
status200 4y ago
Fairly certain that eye tracking will be standard on the next generation of VR devices (at least from the major players like the Quest Pro [0] and Pico 4 [1]), which is exciting and terrifying at the same time. [0] https://www.roadtovr.com/oculus-quest-pro-eye-face-tracking/ https://www.roadtovr.com/oculus-quest-pro-eye-face-tracking/ [1] https://www.roadtovr.com/pico-4-pro-enterprise-eye-tracking-face/ https://www.roadtovr.com/pico-4-pro-enterprise-eye-tracking-...
- shafyy 4y ago
  Yes, pretty certain the Meta Cambria (or Quest Pro), which will be announced on Oct 11, will have eye tracking. Main use is to enable foveated rendering for better performance and quality, but yeah, it's kind of terrfying what other uses there could be. Edit: Just saw you edited and updated your post mentioning the Quest Pro after posting this.
- squeaky-clean 4y ago
  Playstation VR 2 as well
- jimmySixDOF 4y ago
  According to some just leaked Quest 3 CAD files [1] it will not have eye tracking to reduce the camera costs. Project Cambria has both eyes and upper and lower facial cameras which all need real time processing. [1] https://youtu.be/tq57TPTsBQQ https://youtu.be/tq57TPTsBQQ SadleyItsBradley leak of Quest 3
barbariangrunge 4y ago
Speaking for myself, I don’t really want companies studying my eye movements, and I generally enjoy being to be able to look at things without them reacting to my glances. I don’t want to see targeted ads because of where my eyes wander, or to be profiled based on it The only thing that I want eye tracking for is maybe foveated rendering, and even then, hardware is catching up and we eventually won’t need that. Head tilt or body orientation is more useful for determining if a character in a game is being ”addressed” or not Speaking for myself at least
- GuB-42 4y ago
  > The only thing that I want eye tracking for is maybe foveated rendering, and even then, hardware is catching up and we eventually won’t need that. I don't like the idea of wasting tremendous amount of power on things you don't see. It may be a thing now because eye tracking with foveated rendering is more expensive then rendering the full frame at full resolution, which is currently less than ideal. Nailing down foveated rendering would allow us to greatly improve quality where it matters, while at the same time provide a wide field of view on a limited power budget. Adaptive focus is another area where eye tracking could be important. Even more important than foveated rendering. For now it is almost impossible to read a book in VR as you would do in real life, that's the problem. Unless you have severe presbyopia (yes, it is an advantage in VR!), everything close will appear blurry. Varifocal lens with eye tracking is a way to solve that problem (see Oculus "half dome")
tzs 4y ago
Heck, I'd like eye-tracking for my desktop computer experiences. In particular, I'd like eye-tracking combined with a system setting that makes closing windows with ⌘-W on my Mac put up a confirmation dialog if I'm not looking at the window that would be closed.
augasur 4y ago
One of the features I am keen to try for eye-tracking is radial menu selection in action games.
janoc 4y ago
Sorry, as someone who has worked in VR and including with eye tracking for over 20 years, no. Eye or gaze tracking is definitely *not* an input device and should never be used as one. At least not in the sense of using it as some sort of conscious, user controlled input. Using it to *passively monitor* where and what the user is looking at is fine and useful, though, but maybe less so than many think. The reason for this is simple - humans don't consciously control eye movement to the same degree as e.g. their limbs. A lot of eye movement is completely automatic. And even the little that we can consciously control would get extremely fatiguing fast - imagine not being able to move your gaze even a millimeter while driving a car without risking a crash! We have made such experiment around 2004 where the user was asked to guide a character walking on a projection screen (not HMD) by gaze. Most people got a horrible headache within few minutes and it was very uncomfortable. Another aspect is that eye tracking isn't magic. There is literally an infrared camera looking at your eye from a close distance. This camera has a finite resolution and the eyeball/iris moves relatively little. With the target the user is actually looking at being several meters away (whether in reality or VR makes no difference here), you could at best identify large objects or some sort of zones where the user is likely to be looking. Beyond 2-3 meters the errors grow fast because the same eye movement corresponds to much a larger displacement the farther the target is (think shining a torch at a distant wall - little hand movement makes it move a lot). Oh and it still totally ignores the last detail - human eye is not a camera or gun that you aim at something and what the red crosshair aligns with is what the user is looking at. That only means that the thing is in the field of view but the user could be focused on something totally different in the same FOV. Ever seen those experiments where you are asked to follow some ball or card or count how many objects of some kind are in the scene - and at the end you get asked whether you have noticed the gorilla walking through the stage. Most people won't, yet it was directly in their field of view. Our brain is a large part of the visual perception and yet we have no way to know what it is really "looking at". So unless we are talking special cases like accessibility solutions for people with disabilities, eye tracking certainly isn't a good candidate for an input device. And even for things like foveated rendering its utility is a disputable - the complexity, added latency and computational requirements may often outweigh the benefits. Foveated rendering can be done with a fixed, untracked FOV - not ideal but often good enough. Where I could see it being genuinely useful for VR is eye contact between avatars. That's a big problem that this could solve. However, whether it would justify the extra cost of the headset, the required extra user discomfort (calibration, extra setup required for good results, etc.) I doubt. Then there are obviously research and professional uses (e.g. tracking attention of a trainee during an instruction session) but those are niche application cases. And, well, lets not get into the "big brother" stuff where one will be tracked, profiled and evaluated on whether or not they looked at ads, signs or even a cute girl/boy passing by. Companies like Facebook (err Meta) being involved with this should give anyone a pause.
fnordpiglet 4y ago
I think eye tracking and facial cameras inside the goggles are key, but not as input. We will never be able to render a realistic avatar if it’s not picking up facial movements and eye focus. With a photo of the face and the ability to track muscle movements in the face around the eyes you can reproduce a realistic personal face with expressed expressions reflected in VR. As an input it feels like a limited utility thing - perhaps bring up information on the object focused on, but mental intent and eye focus are often not in line with each other - I might intend to do something with X but am monitoring Y and Z in the environment with my eyes while keeping X in partial focus.
deleted 4y ago
[deleted]
daxfohl 4y ago
I keep thinking something about spotting squirrels climbing trees
gkfasdfasdf 4y ago
Looking forward to the pop up ads right in the center of my gaze /s
jvanderbot 4y ago
Gaze tracking for adaptive resolution and frame rate makes sense, though.
wellthisisgreat 4y ago
Aren't there VR headsets that do actually perform eye-tracking? Also is projection on a retina is a thing? I thought there were prototypes at least that did that. I heard of such thing from people at MIT or maybe I misunderstood what they meant back then (some 5-6 years ago?) I did some very basic work with pupil dilation tracking using IR and it's gotta be RGB cameras or you'll just burn your eyes out (or it will feel that way anyways)
wslh 4y ago
Mark Zuckerberg says in a recent podcast [1] that the next VR headset will scan gestures perform eye contact. He says that will be available in October. [1] https://open.spotify.com/episode/51gxrAActH18RGhKNza598 https://open.spotify.com/episode/51gxrAActH18RGhKNza598
joshruby16 4y ago
Most comments here are about how painful eye-tracking as a mouse replacement is. What if instead it was just used for the 1-2 most often used buttons on a site. For example, here on HN - to go back to the main page. Or, on gmail - to press compose. Once gaze in the general area is detected, the affordance can be highlighted, at which point press Enter would cause navigation. I feel that <4 buttons are the ones that are pressed 80% of the time on most sites.
emkoemko 4y ago
PSVR 2 has eye tracking, you have to calibrate it when you first try it but once its done it works really well, stuff like when characters talk to you they look at your eyes, being able to see what other people are looking at etc but the biggest advantage to eye tracking is foveated rendering where the game engine renders in higher resolution in areas where your eyes are looking so this will increase performance and graphics.

134 points by mlajtos 4y ago

hoseja 4y ago

This would be worth it for foveated rendering alone.

moron4hire 4y ago
Turns out foveated rendering is not the panacea it was imagined to be. Eye saccades are so fast that there's just too much motion-to-photon latency to do it well. https://twitter.com/ID_AA_Carmack/status/1391869530327052291 https://twitter.com/ID_AA_Carmack/status/1391869530327052291

presentation 4y ago

Sounds tiring to me, I already get fatigued looking down to make the iPhone’s Face ID happy.

xattt 4y ago
I beg to differ. Canon marketed several film and video cameras with eye-tracking for focus. I remember it being a very intuitive interface. Considering this through a modern perspective of inclusivity, there are probably issues around accessibility for people with eye-mobility issues (amblyopia, stroke-related impairments, etc). (1) https://m.dpreview.com/articles/6531126959/looking-back-canons-eye-controlled-focus https://m.dpreview.com/articles/6531126959/looking-back-cano...
- ghaff 4y ago
  They appeared during the late days of film EOS models but went away and never returned. I've never heard a good explanation. I even asked a friend of mine who sometimes covers camera tech for one of the big online pubs and Canon would never give an explicit reason.
  - Auracle 4y ago
    They’re back now, actually.
    - ghaff 4y ago
      Yeah. Someone else mentioned that you can apparently use eye tracking to set an initial focus point in the R3. And it apparently works well for some and not at all for others.

m00dy 4y ago

I think a comfortable vr headset is the missing device for VR experiences.

PretzelPirate 4y ago
I find the HP Reverb G2 to be incredibly comfortable and can wear it for hours. I do have an air conditioning unit in my VR room which keeps me cool while moving during that time. The Quest 2 is the lest comfortable thing I've ever put on my face and due to its popularity, seems to skew perspectives around how comfortable VR can be.
mlajtos 4y ago
Agree 100%. I am curious what fruit company has in their sleeve.
- ohgodplsno 4y ago
  An overpriced device that doesn't run any games nor has developers for it ? The Valve Index showed that as long as your VR headset is $1000 and above, barely anyone will buy it. And that's without taking into account the hardware you need. If anyone has a chance to succeed this generation, it's Sony with their PSVR. And even then, it's $500, and you need another $150 for their headset.
  - mlajtos 4y ago
    Standalone headsets (like Meta Quest line) are much more interesting then PCVR which requires beefy PC, base stations and lot of cables and setting up. Paying $1000+ for a standalone headset might sound too ridiculous but when you understand that it is the whole computer, it is much more acceptable.
    - ohgodplsno 4y ago
      And you expect Apple to put out a $300 device like the Quest ? Because this is the main subject. Paying $1000+ for a standalone headset, even if you can play Half Life Alyx in 4K is still not something that will reach any proper market adoption. Unlike a phone, which you can use at all times, a VR device requires a ton of space, requires buy-in from developers (Apple's current list of games and apps that support VR is... right, zero. Planning to run your game on Rosetta + MoltenVK ?), etc. It's not the whole computer. It's a computer that you can't really bring outside, that is heavily limited because haptics are still terrible (and will stay terrible). Do you expect people to bring their VR headset to starbucks ?

oneoff786 4y ago

Rye tracking would be great. But you’ll never have ready player one quality experiences with a headset at all

solardev 4y ago
The gluten-free movement has gone too far.

lagrange77 4y ago

While thinking about it, i realized, that our vision apparatus has an interesting feature: When you move your head, your field of sight moves with that motion. But when you hold your head still and only move your eyeballs, the field of sight stays constant, while the focus point (gaze point?) changes, scanning the still standing field of sight. Imagine replacing the eyeballs along with their mounting and motion with cameras. Every motion would translate the field of sight as a whole. Anyone knows, how this mechanism is called?

baxtr 4y ago
Damn. Now that I did that experiment I was reminded of my eye floaters. Thanks for that... My day is ruined. /s https://en.wikipedia.org/wiki/Floater https://en.wikipedia.org/wiki/Floater
- lagrange77 4y ago
  Good ol' faithful friends
GistNoesis 4y ago
In the radio domain, this is called beam-forming.
xeonmc 4y ago
gimbal? And the reflex for keeping the gaze fixed is oculovestibular reflex
- lagrange77 4y ago
  As a layman, i would say the gimbal mechanism merely lets you lock focus on some global coordinates, while moving your head around, but maybe i'm wrong.
numpad0 4y ago
Mind blown. Indeed: - when I move my head to (+30, +45), my brain reports viewport.LeftTop =(0,0), and, - when I move my eyes to (+30, +15), my brain reports viewport.LeftTop = (30, -15). But our eyes are built in such way that the whole lens and sensor assembly rotates, rather than having sensor fixed to the head and lens rotating, so it makes no sense. Framebuffer should always span from (0,0) to (size.x, size.y). Perhaps the framebuffer always covers the full viewing range, and DMA start addresses are dynamically determined by current eye positions, or perhaps even by feature matching?

atemerev 4y ago

I have thought so before, but in my experience, wide field of vision works fine so you can just look at different spots. However, there is no focus/accommodation tracking in VR, which makes your experience hyperreal (everything is always in focus). Incidentally, it is very close to lucid dreaming, where you also see everything at once.

mlajtos 4y ago
I have never realized the similarity between VR and lucid dreams is because of missing depth. Thank you for this. :)

Findecanor 4y ago

I once tried a few examples of eye-tracking as an input device at a visit to Tobii some years ago. One of these was a eye-tracking shooting game. It felt completely wrong and unnatural. My eyes are my input devices. If I have to constrain my gaze and look only at the object to control then I would have to strain myself to perceive the blurry objects outside the centre of my vision just to be able to see those objects at all. IMHO, eye tracking should only be applied to make the experience feel more natural, not the opposite. For example: in combination with gesture controls to find the object that the gesture is intended for. Or in VR headsets to render at highest resolution only where the user is watching, to keep the FPS rate up.

mlajtos 4y ago
Yes, what you describe is the Golden Gaze problem (related to golden touch of the King Midas). There are several techniques mentioned in the article to mitigate the problem. Combination of gaze and hand gestures (and voice) seems like the most efficient and natural. Foveated rendering is also interesting, but that is pretty straightforward application of gaze contingency.
- macrolime 4y ago
  I think a combination of gaze and controllers (like the quest controllers) seem more natural, unless it's for AR and you're out and about.
- brobdingnagians 4y ago
  Talon allows using the Tobii with other triggers so that gaze does nothing without a trigger (a keyword or noise), which then zooms the area looked at, then allows using another trigger to perform the click.
acomjean 4y ago
Canon tried to use eye tracking cameras. Eg “look in the viewfinder where you want to focus”. It didn’t take initially (90s). But it’s back . https://m.dpreview.com/articles/6931257631/is-the-canon-eos-r3-s-eye-control-af-good-enough-for-sports-shooting https://m.dpreview.com/articles/6931257631/is-the-canon-eos-...

ElCheapo 4y ago

*look down to pay respects*

Daub 4y ago

I worked on an eye tracking project that compared the difference between the way artists and non artists look at images. Very interesting. Non artists tends to dwell on faces, hands and other evidently interesting things. Artists find interest in less obvious thing… odd angular protrusions, unexpected colors etc. It also showed me how active and (frankly) jittery all human vision is: never resting, scanning. Also... unexpectedly, we seem to evaluate objects according to their features (points, blobs, angles).

epakai 4y ago

Fove [0] built a headset with eye tracking features. I had a senior project trying to use it for visual field testing in 2017. [0] https://www.kickstarter.com/projects/fove/fove-the-worlds-first-eye-tracking-virtual-reality https://www.kickstarter.com/projects/fove/fove-the-worlds-fi...

ohgodplsno 4y ago

As said in other comments, eye tracking to aim at things would be absolutely horrendous. Try using a mouse on maximum sensitivity, and you'll see the kind of accuracy you'll need. However, eye tracking does give you one thing (that is already getting into VR devices today, and that's foveated rendering. Increasing resolution where needed and lowering it elsewhere. Couple this with various techniques to turn on fewer pixels [https://www.immersivecomputinglab.org/publication/color-perception-guided-display-power-reduction-for-virtual-reality/ https://www.immersivecomputinglab.org/publication/color-perc...] and save up even more on performance/energy usage, and you kind have pleasant, high refresh rate experiences.

thedorkknight 4y ago
The article talks about foveated rendering
- ohgodplsno 4y ago
  The article barely mentions foveated rendering, in the middle of a sea of "use your gaze to click it'll be great I promise".
  - mlajtos 4y ago
    Foveated rendering should be an invisible technology — the image you see should be perceptually indistinguishable from the fully rendered image. How to get there is an interesting engineering problem, but for me that isn’t an interesting domain. You summed up the article quite good, but I don’t claim it will be great. I just think that gaze as an input device is an underresearched topic. Ken Pfeuffer’s research [0] is beyond awesome, but we need masses of thinkerers to explore the domain more. New wave of VR headsets will enable it. [0] https://kenpfeuffer.com/publications-2/ https://kenpfeuffer.com/publications-2/

blululu 4y ago

I’ve heard a lot of these claims before and seen them fall flat when you actually start building it. Things like foveated rendering that many (the author included) claim will lead to better graphics and lower power just don’t add up. You need at least two additional cameras running a pretty compute heavy ml algorithm to track eyes and this costs more watts than just rendering the full screen as your gpu intended. As an interface it can be quick but don’t underestimate that it is one really fucking annoying to get targeting feedback on everything you look at (which sucks for dwell oriented tasks like reading). Finally there is the elephant in the room that this tech works somewhat well for people from Northern Europe, but gets super shaken for a wider swath of the global population. Maybe the author has seen some demos that were better than what I’ve seen, but this all seems like the project pitches that I’ve seen for why we need to explore this area and no the final report.

LordHeini 4y ago
That is completely wrong. For foveated rendering one could easily build dedicate hardware into the device which would require almost no additional power. And those algorithms are not that expensive to begin with. Every camera has face tracking build in (which basically is the same problem) and that does not drain the battery either. And why should northern Europeans eyes work differently? Even if that would be true one could add a bunch of settings for personalisation. Eye distance for example is a common setting on dodays devices...
- blululu 4y ago
  Eye trackers typically require 2-4 additional cameras trained on the eyes. The algorithms for eye tracking are typically pretty involved, but maybe you know some very lower power algorithm to track the eye at 120 hz? I'm not sure how dedicated hardware will reduce the power consumption for 2 cameras and an ML pipeline to almost nothing. Northern Europeans have very light irises. It makes it easier to track the pupil from a single signal-to-noise perspective. But there are a ton of weird ethnic challenges for eye tracking. Things like reflectivity and moisture of eyes varies a lot across different across different ethnic groups. Personalization add complexity and will only take you so far when you are dealing with physical sensing challenges.

sys_64738 4y ago

What if I have a lazy eye? Isn't that discrimination?

micromacrofoot 4y ago
no
thedorkknight 4y ago
At worst you're just back to a regular VR display in that case, so unless current non-eye-tracking headsets are discriminatory, no. Plus I would think it would still be able to use your non-lazy eye
lm28469 4y ago
Everything is a discrimination to someone if you start with such crazy takes
pessimizer 4y ago
What if I'm blind? Is having any visual component to VR bigotry?
- sys_64738 4y ago
  This is why you have Americans with disabilities acts forcing accessibility for those deprived of all sensory and motor function.

twirlock 4y ago

picsao 4y ago

Philip-J-Fry 4y ago

PSVR 2 has eye tracking. So, we'll be seeing more usage of it.

k__ 4y ago

Could be used for games like Silent Hill or Amnesia for effects that only happen if you don't look directly at them.

mlajtos 4y ago
Weeping angels from Doctor Who

oblak 4y ago

> Eye as a Pointing Device As a long time gamer and VR (Valve Index) owner, this is where I stopped reading. No, no, no. As others have said, we could use gaze tracking for plenty, just not pointing devices. People have been trying this approach for decades and it's always been the wrong one. It could work for disabled people/animals, but it should not be the main pointing device. Not my kind of article, that's for sure

glaslong 4y ago
This exactly. It’s terrible as a precision pointer, but great as an attention heuristic. A strong secondary intent signal.
- pwython 4y ago
  What came to mind first was sniping in games like Counter Strike. Your eye is generally within the crosshair aiming at a certain spot, but suddenly someone comes in your peripheral and you have to make a quick flick shot, which requires more muscle memory and not even focusing on the target.
seejayseesjays 4y ago
Exactly. I've got Nystagmus so I reckon that if the eye tracking is particularly accurate, it'd be like trying to control a mouse that uncontrollably for me.
sli 4y ago
I only want eye tracking on my Index for VRChat integration so my avatar's eyes move with mine. I just cannot imagine using my eyes as some kind of input device, that sounds immensely awkward for someone that has more traditional options (i.e. no physical disabilities). I'm not using VR for productivity in any case, anyway.

jrm4 4y ago

Speaking of VR, it really feels like the motion sickness part is still woefully underacknowledged. I've enjoyed quite a few VR games (I have access to most of the systems at my job) but there's no way I can spend more than e.g. 30 min a week "in there." The body REALLY REALLY REALLY hates when movement and what you see don't match up, and very few experiences deal with this well and also are really compelling.

fknorangesite 4y ago
> underacknowledged Because it doesn't affect everyone to the same degree. What you describe sounds really severe (and I don't doubt it) - but for example I'm personally way at the other end of the spectrum: I can easily spend hours in VR, and I can think of only one game that ever made me even slightly motion sick.
status200 4y ago
I had the opposite experience, my family has a history of motion sickness, but myself and other family members that have tried it have felt really natural in VR. Even my sister could use smooth movement and "advanced" settings, which I never would have predicted. Our best guess is that the 6DoF system at least matches head movement, which might be the difference that makes our brain(s) not freak out compared to reading in the car, etc.
- ShamelessC 4y ago
  The truth is that it's highly subjective to each individual and simply because it has gotten better for some, doesnt make it a solved issue (and still makes it a non-starter for people effected).
apike 4y ago
I’m susceptible to this too, but I believe the two reasons VR motion sickness is much less discussed nowadays are: 1. Often it’s triggered by frame rate hitches, which seems solvable with the usual advance of software and hardware over time. 2. Most folks can train their mind and body out of the motion sickness with practice.

status200 4y ago

Fairly certain that eye tracking will be standard on the next generation of VR devices (at least from the major players like the Quest Pro [0] and Pico 4 [1]), which is exciting and terrifying at the same time. [0] https://www.roadtovr.com/oculus-quest-pro-eye-face-tracking/ https://www.roadtovr.com/oculus-quest-pro-eye-face-tracking/ [1] https://www.roadtovr.com/pico-4-pro-enterprise-eye-tracking-face/ https://www.roadtovr.com/pico-4-pro-enterprise-eye-tracking-...

shafyy 4y ago
Yes, pretty certain the Meta Cambria (or Quest Pro), which will be announced on Oct 11, will have eye tracking. Main use is to enable foveated rendering for better performance and quality, but yeah, it's kind of terrfying what other uses there could be. Edit: Just saw you edited and updated your post mentioning the Quest Pro after posting this.
squeaky-clean 4y ago
Playstation VR 2 as well
jimmySixDOF 4y ago
According to some just leaked Quest 3 CAD files [1] it will not have eye tracking to reduce the camera costs. Project Cambria has both eyes and upper and lower facial cameras which all need real time processing. [1] https://youtu.be/tq57TPTsBQQ https://youtu.be/tq57TPTsBQQ SadleyItsBradley leak of Quest 3

barbariangrunge 4y ago

Speaking for myself, I don’t really want companies studying my eye movements, and I generally enjoy being to be able to look at things without them reacting to my glances. I don’t want to see targeted ads because of where my eyes wander, or to be profiled based on it The only thing that I want eye tracking for is maybe foveated rendering, and even then, hardware is catching up and we eventually won’t need that. Head tilt or body orientation is more useful for determining if a character in a game is being ”addressed” or not Speaking for myself at least

GuB-42 4y ago
> The only thing that I want eye tracking for is maybe foveated rendering, and even then, hardware is catching up and we eventually won’t need that. I don't like the idea of wasting tremendous amount of power on things you don't see. It may be a thing now because eye tracking with foveated rendering is more expensive then rendering the full frame at full resolution, which is currently less than ideal. Nailing down foveated rendering would allow us to greatly improve quality where it matters, while at the same time provide a wide field of view on a limited power budget. Adaptive focus is another area where eye tracking could be important. Even more important than foveated rendering. For now it is almost impossible to read a book in VR as you would do in real life, that's the problem. Unless you have severe presbyopia (yes, it is an advantage in VR!), everything close will appear blurry. Varifocal lens with eye tracking is a way to solve that problem (see Oculus "half dome")

tzs 4y ago

Heck, I'd like eye-tracking for my desktop computer experiences. In particular, I'd like eye-tracking combined with a system setting that makes closing windows with ⌘-W on my Mac put up a confirmation dialog if I'm not looking at the window that would be closed.

augasur 4y ago

One of the features I am keen to try for eye-tracking is radial menu selection in action games.

janoc 4y ago

Sorry, as someone who has worked in VR and including with eye tracking for over 20 years, no. Eye or gaze tracking is definitely *not* an input device and should never be used as one. At least not in the sense of using it as some sort of conscious, user controlled input. Using it to *passively monitor* where and what the user is looking at is fine and useful, though, but maybe less so than many think. The reason for this is simple - humans don't consciously control eye movement to the same degree as e.g. their limbs. A lot of eye movement is completely automatic. And even the little that we can consciously control would get extremely fatiguing fast - imagine not being able to move your gaze even a millimeter while driving a car without risking a crash! We have made such experiment around 2004 where the user was asked to guide a character walking on a projection screen (not HMD) by gaze. Most people got a horrible headache within few minutes and it was very uncomfortable. Another aspect is that eye tracking isn't magic. There is literally an infrared camera looking at your eye from a close distance. This camera has a finite resolution and the eyeball/iris moves relatively little. With the target the user is actually looking at being several meters away (whether in reality or VR makes no difference here), you could at best identify large objects or some sort of zones where the user is likely to be looking. Beyond 2-3 meters the errors grow fast because the same eye movement corresponds to much a larger displacement the farther the target is (think shining a torch at a distant wall - little hand movement makes it move a lot). Oh and it still totally ignores the last detail - human eye is not a camera or gun that you aim at something and what the red crosshair aligns with is what the user is looking at. That only means that the thing is in the field of view but the user could be focused on something totally different in the same FOV. Ever seen those experiments where you are asked to follow some ball or card or count how many objects of some kind are in the scene - and at the end you get asked whether you have noticed the gorilla walking through the stage. Most people won't, yet it was directly in their field of view. Our brain is a large part of the visual perception and yet we have no way to know what it is really "looking at". So unless we are talking special cases like accessibility solutions for people with disabilities, eye tracking certainly isn't a good candidate for an input device. And even for things like foveated rendering its utility is a disputable - the complexity, added latency and computational requirements may often outweigh the benefits. Foveated rendering can be done with a fixed, untracked FOV - not ideal but often good enough. Where I could see it being genuinely useful for VR is eye contact between avatars. That's a big problem that this could solve. However, whether it would justify the extra cost of the headset, the required extra user discomfort (calibration, extra setup required for good results, etc.) I doubt. Then there are obviously research and professional uses (e.g. tracking attention of a trainee during an instruction session) but those are niche application cases. And, well, lets not get into the "big brother" stuff where one will be tracked, profiled and evaluated on whether or not they looked at ads, signs or even a cute girl/boy passing by. Companies like Facebook (err Meta) being involved with this should give anyone a pause.

fnordpiglet 4y ago

I think eye tracking and facial cameras inside the goggles are key, but not as input. We will never be able to render a realistic avatar if it’s not picking up facial movements and eye focus. With a photo of the face and the ability to track muscle movements in the face around the eyes you can reproduce a realistic personal face with expressed expressions reflected in VR. As an input it feels like a limited utility thing - perhaps bring up information on the object focused on, but mental intent and eye focus are often not in line with each other - I might intend to do something with X but am monitoring Y and Z in the environment with my eyes while keeping X in partial focus.

deleted 4y ago

[deleted]

daxfohl 4y ago

I keep thinking something about spotting squirrels climbing trees

gkfasdfasdf 4y ago

Looking forward to the pop up ads right in the center of my gaze /s

jvanderbot 4y ago

Gaze tracking for adaptive resolution and frame rate makes sense, though.

wellthisisgreat 4y ago

Aren't there VR headsets that do actually perform eye-tracking? Also is projection on a retina is a thing? I thought there were prototypes at least that did that. I heard of such thing from people at MIT or maybe I misunderstood what they meant back then (some 5-6 years ago?) I did some very basic work with pupil dilation tracking using IR and it's gotta be RGB cameras or you'll just burn your eyes out (or it will feel that way anyways)

wslh 4y ago

Mark Zuckerberg says in a recent podcast [1] that the next VR headset will scan gestures perform eye contact. He says that will be available in October. [1] https://open.spotify.com/episode/51gxrAActH18RGhKNza598 https://open.spotify.com/episode/51gxrAActH18RGhKNza598

joshruby16 4y ago

Most comments here are about how painful eye-tracking as a mouse replacement is. What if instead it was just used for the 1-2 most often used buttons on a site. For example, here on HN - to go back to the main page. Or, on gmail - to press compose. Once gaze in the general area is detected, the affordance can be highlighted, at which point press Enter would cause navigation. I feel that <4 buttons are the ones that are pressed 80% of the time on most sites.

emkoemko 4y ago

PSVR 2 has eye tracking, you have to calibrate it when you first try it but once its done it works really well, stuff like when characters talk to you they look at your eyes, being able to see what other people are looking at etc but the biggest advantage to eye tracking is foveated rendering where the game engine renders in higher resolution in areas where your eyes are looking so this will increase performance and graphics.