5 ms·

I’ve heard a lot of these claims before and seen them fall flat when you actually start building it. Things like foveated rendering that many (the author includ

I’ve heard a lot of these claims before and seen them fall flat when you actually start building it. Things like foveated rendering that many (the author included) claim will lead to better graphics and lower power just don’t add up. You need at least two additional cameras running a pretty compute heavy ml algorithm to track eyes and this costs more watts than just rendering the full screen as your gpu intended. As an interface it can be quick but don’t underestimate that it is one really fucking annoying to get targeting feedback on everything you look at (which sucks for dwell oriented tasks like reading). Finally there is the elephant in the room that this tech works somewhat well for people from Northern Europe, but gets super shaken for a wider swath of the global population. Maybe the author has seen some demos that were better than what I’ve seen, but this all seems like the project pitches that I’ve seen for why we need to explore this area and no the final report.

LordHeini 4y ago
That is completely wrong. For foveated rendering one could easily build dedicate hardware into the device which would require almost no additional power. And those algorithms are not that expensive to begin with. Every camera has face tracking build in (which basically is the same problem) and that does not drain the battery either. And why should northern Europeans eyes work differently? Even if that would be true one could add a bunch of settings for personalisation. Eye distance for example is a common setting on dodays devices...
- blululu 4y ago
  Eye trackers typically require 2-4 additional cameras trained on the eyes. The algorithms for eye tracking are typically pretty involved, but maybe you know some very lower power algorithm to track the eye at 120 hz? I'm not sure how dedicated hardware will reduce the power consumption for 2 cameras and an ML pipeline to almost nothing. Northern Europeans have very light irises. It makes it easier to track the pupil from a single signal-to-noise perspective. But there are a ton of weird ethnic challenges for eye tracking. Things like reflectivity and moisture of eyes varies a lot across different across different ethnic groups. Personalization add complexity and will only take you so far when you are dealing with physical sensing challenges.
- deleted 4y ago
  [deleted]
terafo 4y ago
You are underestimating cost of rendering and overestimating cost of machine learning models. And forgetting the fact that cameras are already used in modern headsets for position tracking and more(there is Quest 2 demo where you cast spells using your hands, no controllers), and it is much much more complicated task than eye tracking.
- blululu 4y ago
  If you have some numbers please share. I've seen NVIDIA and Tobii (highly motivated parties) claim that you might get a 10% power reduction, but those studies were based on very constrained operating conditions and conveniently brush certain details under the rug. All told, it's a lot of work and a lot complexity to chase something that will at most provide modest gains. The cameras on headsets face the world. So you need 2-4 additional cameras trained on the eyes. These camera each require power just to get images. For a decent UX you need tracking at ~120hz (60hz eye tracking is slow). Exact figures are hard to come by on a Sunday morning, but off the top of my head, I've seen figures to the tune of 250mW per camera (which ignores ISP and the compute to move the data around). Computer vision algorithms running at 120 hz are typically not cheap. Pupil Labs cites 50% CPU (on an i5) per eye[1]. Tobii doesn't publish their power consumption but it is also pretty steep. I'm sure there are optimizations here but all the specs I've seen blush at the power consumption required by a general purpose eye tracker. If you have figures that suggest otherwise I would be happy to see but every quantitative analysis I've seen comes up with a muddled response. If you already commit to running eye tracking for other reasons then it makes sense to run this rendering optimization, but I'm not sure if it really pays for its self. [1] https://pupil-labs.com/products/vr-ar/tech-specs/ https://pupil-labs.com/products/vr-ar/tech-specs/
- charcircuit 4y ago
  >And forgetting the fact that cameras are already used in modern headsets for position tracking and more Just because you already are running cameras + ML models that doesn't mean adding even more cameras and ML models is free.
- jayd16 4y ago
  I would think it's actually harder to do eye tracked foveated rendering than motion tracking. They're not identical problems. Motion tracking is a feature, foveated rendering is an optimization that needs to significantly beat the naive approach of fixed foveated rendering. Part of the problem is you can throw everything you have at motion tracking. You can over render and then use the end of frame head position to space/time warp to adjust the render the display gets, maybe throwing away some of the edges. For foveated rendering, you need to know what to render at the START of the frame before you render it, there can't be any waste, and we're only talking about a thin band of rendering gain IF the user is not looking forward. I would assume the focal point can change faster because you're at least dealing with the head movement + eye movement, so it's even more work that needs to catch up at the end of the frame. I really do not think foveated rendering is easier than head tracking. If it was, we would see more of it. Dynamic foveation is really only just now coming to headsets.
moron4hire 4y ago
Eye-tracking uses simple IR reflection sensors to follow the black dot of the pupil. It has no dependence on skin or iris color. It's been done for decades in psychological studies and doesn't require complex ML algorithms to do. Eye tracking is not a particularly hard thing to implement in hardware, it's just an added expense for dubious--very dubious--utility. Probably the "best" use of eye tracking for modern VR systems is pretty much just reflecting the eye motion in avatars during VR teleconferencing sessions to give a more natural "face-to-face conversation" feel. Beyond that, things like eye-tracking as input and foveated rendering turn out to only be great on paper.
- charcircuit 4y ago
  >Eye-tracking uses simple IR reflection sensors Every VR headset with eye tracking uses actual cameras.
  - sgtnoodle 4y ago
    These days, wouldn't a camera sensor be considered a simple sensor? Optical mice use actual cameras too, they're just optimized for a very shallow focus, very high temporal resolution rather than spatial resolution, and sensitivity to IR. Do eye trackers need especially high spatial resolution, or RGB color?
    - charcircuit 4y ago
      I was mainly referring to the power requirements.
lam 4y ago
Why was this down-voted? S/he seems to be making valid comments.

LordHeini 4y ago

That is completely wrong. For foveated rendering one could easily build dedicate hardware into the device which would require almost no additional power. And those algorithms are not that expensive to begin with. Every camera has face tracking build in (which basically is the same problem) and that does not drain the battery either. And why should northern Europeans eyes work differently? Even if that would be true one could add a bunch of settings for personalisation. Eye distance for example is a common setting on dodays devices...

blululu 4y ago
Eye trackers typically require 2-4 additional cameras trained on the eyes. The algorithms for eye tracking are typically pretty involved, but maybe you know some very lower power algorithm to track the eye at 120 hz? I'm not sure how dedicated hardware will reduce the power consumption for 2 cameras and an ML pipeline to almost nothing. Northern Europeans have very light irises. It makes it easier to track the pupil from a single signal-to-noise perspective. But there are a ton of weird ethnic challenges for eye tracking. Things like reflectivity and moisture of eyes varies a lot across different across different ethnic groups. Personalization add complexity and will only take you so far when you are dealing with physical sensing challenges.
deleted 4y ago
[deleted]

terafo 4y ago

You are underestimating cost of rendering and overestimating cost of machine learning models. And forgetting the fact that cameras are already used in modern headsets for position tracking and more(there is Quest 2 demo where you cast spells using your hands, no controllers), and it is much much more complicated task than eye tracking.

blululu 4y ago
If you have some numbers please share. I've seen NVIDIA and Tobii (highly motivated parties) claim that you might get a 10% power reduction, but those studies were based on very constrained operating conditions and conveniently brush certain details under the rug. All told, it's a lot of work and a lot complexity to chase something that will at most provide modest gains. The cameras on headsets face the world. So you need 2-4 additional cameras trained on the eyes. These camera each require power just to get images. For a decent UX you need tracking at ~120hz (60hz eye tracking is slow). Exact figures are hard to come by on a Sunday morning, but off the top of my head, I've seen figures to the tune of 250mW per camera (which ignores ISP and the compute to move the data around). Computer vision algorithms running at 120 hz are typically not cheap. Pupil Labs cites 50% CPU (on an i5) per eye[1]. Tobii doesn't publish their power consumption but it is also pretty steep. I'm sure there are optimizations here but all the specs I've seen blush at the power consumption required by a general purpose eye tracker. If you have figures that suggest otherwise I would be happy to see but every quantitative analysis I've seen comes up with a muddled response. If you already commit to running eye tracking for other reasons then it makes sense to run this rendering optimization, but I'm not sure if it really pays for its self. [1] https://pupil-labs.com/products/vr-ar/tech-specs/ https://pupil-labs.com/products/vr-ar/tech-specs/
charcircuit 4y ago
>And forgetting the fact that cameras are already used in modern headsets for position tracking and more Just because you already are running cameras + ML models that doesn't mean adding even more cameras and ML models is free.
jayd16 4y ago
I would think it's actually harder to do eye tracked foveated rendering than motion tracking. They're not identical problems. Motion tracking is a feature, foveated rendering is an optimization that needs to significantly beat the naive approach of fixed foveated rendering. Part of the problem is you can throw everything you have at motion tracking. You can over render and then use the end of frame head position to space/time warp to adjust the render the display gets, maybe throwing away some of the edges. For foveated rendering, you need to know what to render at the START of the frame before you render it, there can't be any waste, and we're only talking about a thin band of rendering gain IF the user is not looking forward. I would assume the focal point can change faster because you're at least dealing with the head movement + eye movement, so it's even more work that needs to catch up at the end of the frame. I really do not think foveated rendering is easier than head tracking. If it was, we would see more of it. Dynamic foveation is really only just now coming to headsets.

moron4hire 4y ago

Eye-tracking uses simple IR reflection sensors to follow the black dot of the pupil. It has no dependence on skin or iris color. It's been done for decades in psychological studies and doesn't require complex ML algorithms to do. Eye tracking is not a particularly hard thing to implement in hardware, it's just an added expense for dubious--very dubious--utility. Probably the "best" use of eye tracking for modern VR systems is pretty much just reflecting the eye motion in avatars during VR teleconferencing sessions to give a more natural "face-to-face conversation" feel. Beyond that, things like eye-tracking as input and foveated rendering turn out to only be great on paper.

charcircuit 4y ago
>Eye-tracking uses simple IR reflection sensors Every VR headset with eye tracking uses actual cameras.
- sgtnoodle 4y ago
  These days, wouldn't a camera sensor be considered a simple sensor? Optical mice use actual cameras too, they're just optimized for a very shallow focus, very high temporal resolution rather than spatial resolution, and sensitivity to IR. Do eye trackers need especially high spatial resolution, or RGB color?
  - charcircuit 4y ago
    I was mainly referring to the power requirements.

lam 4y ago

Why was this down-voted? S/he seems to be making valid comments.