4 ms·

How Eye Tracking Will Change Gaming

51 points by johntans 11y ago

tdaltonc 11y ago
So they talk about 4 ways to use eye tracking, 1) Use gaze to control the camera pivot -- That sounds incredibly annoying. Like I wouldn't be able to just look around a screen without it constantly moving. Though if a system is built with this use case in mind it could be cool. Imagine having a whole 360* view on screen but with it kind of squished around the edges, when you look out to an area on the edge it could immediately foveate the display to that area. 2) Eye contact - Could be really cool. Monsters that can only move when you look away (or blink). Or they disappear if you look directly at them. The NPC's stop talking or get upset if they think that you're not paying attention. 3) natural targeting - I'd like to try this but I'm skeptical. What If I want to look where I'm running for a moment? Then my gun careens across the screen. 4) Immersive Graphics and Sound - This is the most interesting one and it has the worst name. This application dynamically focuses and blurs the depth of field of an image as you look around. One of the big problems with 3D immersive tech right now (Real3D, oculus) is that you can't capture this element of natural vision. They usually default to rendering everything as extremely sharp. I'm surprised they didn't take about typing with your eyes since it's something that eye-tracking is currently used for.
- nugga 11y ago
  It's not so much about mimicking natural vision at it is to reduce the render target radically, rendering only the small area where you're looking at in good detail while the rest of the image can look like quake 1. edit: specifically I was thinking about foveated rendering.
- harperlee 11y ago
  > Monsters that can only move when you look away (or blink) This could be so cool, reminds me (probably you might have been inspired by it!) to this enemies from Dr. Who, the weeping angels. I'd recommend anyone curious to look at the episode "Blink" (it seems that it is on youtube: https://www.youtube.com/watch?v=7vahAzc2l9w https://www.youtube.com/watch?v=7vahAzc2l9w)
  - CM30 11y ago
    It also reminds me of the Boos from the Super Mario Bros series. Either way, such monsters would certainly be interesting in a game with this sort of eye tracking stuff included.
  - tdaltonc 11y ago
    Ya, I was thinking of Boo's and weeping angles.
kshatrea 11y ago
I have worked and work for competitors of Tobii. The article is kind of biased. The cheapest eyetracker in the world is The Eye Tribe tracker ( I have worked there). They've not been able to solve the battery drain problem yet (running IR LEDs costs you) and therefore mobile gaming is out. I would assume Tobii has had similar issues. Tobii's EyeX is the only Tobii device cheap enough to go into the consumer market but it cannot do a high resolution head pose estimation at a high frame rate. The other awesome company for eyetracking in Sweden is Smart Eye which is targeting a different industry still has issues with high precision at low frame rates. In short, eye tracking is important and progressing fast but we aren't where we want to be in terms of using less power and also kind of supplanting the hand controlled devices for typical use cases like aiming in FPS games or changing POV based on head pose. We've used Tobii as part of competition research and the EyeX API is a bit limited also. I would give this article a 7/10. Edited for clarity.
- navaati 11y ago
  Is the precision in the current techno good enough to replace the mouse for non-gaming activities, like web browsing or using a classical GUI with buttons and text fields and stuff ? I'd love being able to never ever get my hands off my keyboard.
  - deleted 11y ago
    [deleted]
  - rdw 11y ago
    I have an eyeX and the precision is not as high as a mouse. I don't think it can ever be, the eye is just not that controlled itself. Having low latency from the device seems even more important to me.
  - deleted 11y ago
    [deleted]
  - praetorian84 11y ago
    Not really. Based on a vague recollection of working with previous model eye trackers, the margin for error is enough to make precise selection annoying. Attempts at using eye-tracking in normal GUIs generally require some form of compensation for the inaccuracy, or redesigning the UI to require less precise selection.
Terr_ 11y ago
IMO the bigger change in the short term would be that Depth of Field and "HDR" effects in games could actually work correctly, rather than being unrealistic imitations of videocamera limitations. Currently, the computer has to guess where you're attention is (e.g always on the crosshair for you weapon/hands) and any attempt to peer elsewhere is punished by unyielding blur and darkness/bloom.
hackuser 11y ago
They discuss trying to mimic the focus of real human vision, making the foreground in sharper focus than background and peripheral objects. Why is this necessary? My human vision will do that itself. For example, if I look at the room I'm in, foreground objects are in sharper focus. If I look at the screen of the computer I'm typing on, if I stare at this letter X, maybe 1-4 letters to each side are in sharp enough focus to read.[1] Why would the computer need to blur the non-foreground images further? The only guess I have is that objects on a screen are in a false perspective, based on the screen size and the typical user distance from it. Cramming the real world onto a screen that is tiny relative to the immersive real world requires some tricks. But that's just a guess - does anyone know? ---- [1] It depends on the font size and probably on resolution too, of course.
- blakecaldwell 11y ago
  It's not just sharpness of focus. There's that, because of - I believe - how densely different areas of our retinas are populated with cones and rods. That's not the same thing that's going on when you look at a tree in front of you, and then the mountains miles away. The screen is flat. If you were playing a game that occupies 3D space, the game could mimic real depth by changing how it looks depending on where you're looking. It would blur things as they'd be blurred in real life. Or maybe they'd double up things close to us when we look far away, like when you hold a finger in front of your face and then stare far away, or vice-versa. I think we probably rarely notice these things in real life. Maybe this would make it feel magically more real.
- tekromancr 11y ago
  I think you have the idea backwards a bit. The system isn't interested in making the background blurry, it wants to make your exact eye point the section of the image with the highest fidelity. Then the renderer can just sort of half ass the rest of the image. It's a way to make the apparent image better than what the hardware would allow.
- tdaltonc 11y ago
  You're confusing a few ideas here. Focus has to do with the fact that the real world can only be projected on to our retina sharply at one distance. The letters in your peripheral vision are hard to make out not because they are out of focus but because they are falling on a part of the retina that has low resolution (also see "visual crowding"). The wikipedia page on optical focus has a good explanation of focus blur. Suffice it to say real scenes have dynamic focus blur and computer screens don't and we can subliminally tell the difference. This tech will make rendered scenes look more like real ones.
lifeformed 11y ago
Wasn't there a Microsoft demo using eye tracking, where only the focused area of the screen was rendered at full resolution, and the rest at lower blurred resolution? It saved a lot of GPU power by just needing to draw a tiny fraction of the game, and it looked indistinguishable to the player. What happened to that?
- ggreer 11y ago
  You're thinking of Foveated 3D Graphics[1]. Despite being a research project, they still got a 5-6x performance improvement. VR will get foveated rendering eventually. The performance improvements are too good to ignore. 1. http://research.microsoft.com/pubs/176610/foveated_final15.pdf http://research.microsoft.com/pubs/176610/foveated_final15.p...
ggreer 11y ago
I'm really looking forward to eye tracking in VR, because it will make for the scariest games ever. Games could notice that you're looking in one direction, then cause a creature to skitter by in your peripheral vision. By the time your eyes or head have moved, it's gone. Peripheral visual acuity is terrible, so you'd never get a good look at the thing until the game wanted you to.
J_Darnley 11y ago
As I said over here https://news.ycombinator.com/item?id=10512349 https://news.ycombinator.com/item?id=10512349 I will be dead before I allow an always on camera to be watching me at my computer.
CM30 11y ago
So, this article. Read it a few days ago, and while I do like some concepts (like the below mentioned monsters that move when you're looking away thing), a lot just seems like it'd be more annoying than anything else. Camera controlled by where you look? Oh, that's good, except quite a lot of the time in games you don't want to have the camera follow your gaze your screen. For example, in a lot of platformers, you want to be able to see as much of the area as possible in order to judge where you're jumping too. If the camera tracks your eye movements... well, someone's probably falling into the abyss pretty soon. I don't think anyone would want to be trying Grandmaster Galaxy or Champion Road with this sort of tech any time soon. There's also the way a lot of games have 'lock on' mechanics for targeting enemies. Your attacks go one way (towards the enemy), but the camera focuses somewhere else to give you a better view of your surroundings. I can see that falling apart pretty quickly if eye tracking got brought into it. But even assuming it'll mostly be used for games with a first person viewpoint (which in itself might be a limiting factor for things like virtual reality, since it's a smaller percentage of games than people like to think), it just seems like it'd be a bit too... precise for a lot of titles. People playing games look all over the screen for things that might be important in the future, and they probably don't want the camera bouncing around like a pinball as a result. The focus and depth of field aspects seem the only plausible ones for most games.
rawnlq 11y ago
The top post in /r/programming this year is about a guy writing his own eye tracking software: https://www.reddit.com/r/programming/comments/3ke7ug/eye_tracking_software_for_sufferers_of_alsmnd_can/ https://www.reddit.com/r/programming/comments/3ke7ug/eye_tra... https://www.reddit.com/r/software/comments/3kdghp/eye_tracking_software_for_sufferers_of_alsmnd_can/?ref=share&ref_source=link https://www.reddit.com/r/software/comments/3kdghp/eye_tracki...
sinxoveretothex 11y ago
I do not look forward to advertisements using this technology.
tracker1 11y ago
Is it sad that all I see this turning into is REALLY annoying ads in streaming apps? I'm a little cynical, but I think that VR interaction might benefit from this a little bit, but most of the interactions in gaming are better served be a more immersive display option, with full head tracking.
andersto 11y ago
Hey, I am working with Tobii and I saw this thread so let me try to give some background info about the Tobii EyeX to answer some of the questions in this thread. We really value the community and comments at Hacker News and we know there are equal amounts of excitement and doubts about eye tracking and I am gonna have a quick giveaway of 10 EyeX sets here in the next 48 hours as well so that some of you can experience it first hand! More on that below but here is some information: First off, we are not in a position to comment on products from other eye tracking companies globally, nor would we want to speak for them. So I am picking a few key issues here to discuss. 1)"Battery Drain issue" – The battery drain issue is probably more skewed towards mobile gaming and not for desktop/laptops and we are seeing little to no difference in battery drain when playing games with/without our eye tracker. 2)"Supplanting the hand controlled devices for aiming or changing POV” - We don’t intend to replace mice, keyboards or even to compete with them, but enhance the gaming experience with as natural interactions as possible. We already play games looking at the screen, and we can make your eyes a useful component of gameplay. 3)“Cannot do a high-res head pose estimation at high frame rate” - We are focused on eye tracking and have elements of head tracking but it is mainly on the point of gaze instead. 4)“Is the precision in the current tech good enough to replace the mouse for non-gaming activities?, web browsing or using a classical GUI with buttons and text fields and stuff” - Technically it is possible for similar levels of accuracy, but our intention was always to use it in combination with other modalities. We have plenty of people in the office who have become much less reliant on a mouse with the eye tracker but natural, learned interaction is not at a hands-free stage at the moment.
- andersto 11y ago
  5)“Eye fatigue comes in pretty quickly” - we don’t want you to do anything differently, but play as naturally as possible since the eye tracker is monitoring your eye movements. So there is really NO NEED to stay perfectly still, blink or be unnaturally close to the screen. 6)"Use gaze to control the camera pivot -- That sounds incredibly annoying.” - We understand the doubts but we are pretty confident that you might have a different opinion after trying it out! A gamespot reporter did the same:http://l.gamespot.com/1Mm7rar http://l.gamespot.com/1Mm7rar 7)“Making the foreground in sharper focus than background and peripheral objects. Why is this necessary?” - We want allow your normal human vision to exist within the game based on your natural focus to allow for real depth instead of a flat screen. @teckromancr @blakecaldwell and @tdaltonc got it right! 8)"Is it sad that all I see this turning into is REALLY annoying ads in streaming apps?"/"I do not look forward to advertisements using this technology.” - We have no intentions of making ads from eye tracking! We also have SDK agreements based on not allowing infringement of privacy. Its been 15 years in eye tracking for Tobii so personal integrity is something we take extremely seriously. So if any of you want to try out the tracker, all you have to do is tweet "let me try @tobiieyex via #hackernews!” and we will DM you for your details. The first 10 individuals to tweet it within 48 hours will get our latest Tobii EyeX shipped right to you, free of charge. All we want is for you to really play/test it out for yourself. We have not done this with any other community before but hacker news is pretty special to us!

51 points by johntans 11y ago

So they talk about 4 ways to use eye tracking, 1) Use gaze to control the camera pivot -- That sounds incredibly annoying. Like I wouldn't be able to just look around a screen without it constantly moving. Though if a system is built with this use case in mind it could be cool. Imagine having a whole 360* view on screen but with it kind of squished around the edges, when you look out to an area on the edge it could immediately foveate the display to that area. 2) Eye contact - Could be really cool. Monsters that can only move when you look away (or blink). Or they disappear if you look directly at them. The NPC's stop talking or get upset if they think that you're not paying attention. 3) natural targeting - I'd like to try this but I'm skeptical. What If I want to look where I'm running for a moment? Then my gun careens across the screen. 4) Immersive Graphics and Sound - This is the most interesting one and it has the worst name. This application dynamically focuses and blurs the depth of field of an image as you look around. One of the big problems with 3D immersive tech right now (Real3D, oculus) is that you can't capture this element of natural vision. They usually default to rendering everything as extremely sharp. I'm surprised they didn't take about typing with your eyes since it's something that eye-tracking is currently used for.

nugga 11y ago
It's not so much about mimicking natural vision at it is to reduce the render target radically, rendering only the small area where you're looking at in good detail while the rest of the image can look like quake 1. edit: specifically I was thinking about foveated rendering.
harperlee 11y ago
> Monsters that can only move when you look away (or blink) This could be so cool, reminds me (probably you might have been inspired by it!) to this enemies from Dr. Who, the weeping angels. I'd recommend anyone curious to look at the episode "Blink" (it seems that it is on youtube: https://www.youtube.com/watch?v=7vahAzc2l9w https://www.youtube.com/watch?v=7vahAzc2l9w)
- CM30 11y ago
  It also reminds me of the Boos from the Super Mario Bros series. Either way, such monsters would certainly be interesting in a game with this sort of eye tracking stuff included.
- tdaltonc 11y ago
  Ya, I was thinking of Boo's and weeping angles.

kshatrea 11y ago

I have worked and work for competitors of Tobii. The article is kind of biased. The cheapest eyetracker in the world is The Eye Tribe tracker ( I have worked there). They've not been able to solve the battery drain problem yet (running IR LEDs costs you) and therefore mobile gaming is out. I would assume Tobii has had similar issues. Tobii's EyeX is the only Tobii device cheap enough to go into the consumer market but it cannot do a high resolution head pose estimation at a high frame rate. The other awesome company for eyetracking in Sweden is Smart Eye which is targeting a different industry still has issues with high precision at low frame rates. In short, eye tracking is important and progressing fast but we aren't where we want to be in terms of using less power and also kind of supplanting the hand controlled devices for typical use cases like aiming in FPS games or changing POV based on head pose. We've used Tobii as part of competition research and the EyeX API is a bit limited also. I would give this article a 7/10. Edited for clarity.

navaati 11y ago
Is the precision in the current techno good enough to replace the mouse for non-gaming activities, like web browsing or using a classical GUI with buttons and text fields and stuff ? I'd love being able to never ever get my hands off my keyboard.
- deleted 11y ago
  [deleted]
- rdw 11y ago
  I have an eyeX and the precision is not as high as a mouse. I don't think it can ever be, the eye is just not that controlled itself. Having low latency from the device seems even more important to me.
- deleted 11y ago
  [deleted]
- praetorian84 11y ago
  Not really. Based on a vague recollection of working with previous model eye trackers, the margin for error is enough to make precise selection annoying. Attempts at using eye-tracking in normal GUIs generally require some form of compensation for the inaccuracy, or redesigning the UI to require less precise selection.

Terr_ 11y ago

IMO the bigger change in the short term would be that Depth of Field and "HDR" effects in games could actually work correctly, rather than being unrealistic imitations of videocamera limitations. Currently, the computer has to guess where you're attention is (e.g always on the crosshair for you weapon/hands) and any attempt to peer elsewhere is punished by unyielding blur and darkness/bloom.

hackuser 11y ago

They discuss trying to mimic the focus of real human vision, making the foreground in sharper focus than background and peripheral objects. Why is this necessary? My human vision will do that itself. For example, if I look at the room I'm in, foreground objects are in sharper focus. If I look at the screen of the computer I'm typing on, if I stare at this letter X, maybe 1-4 letters to each side are in sharp enough focus to read.[1] Why would the computer need to blur the non-foreground images further? The only guess I have is that objects on a screen are in a false perspective, based on the screen size and the typical user distance from it. Cramming the real world onto a screen that is tiny relative to the immersive real world requires some tricks. But that's just a guess - does anyone know? ---- [1] It depends on the font size and probably on resolution too, of course.

blakecaldwell 11y ago
It's not just sharpness of focus. There's that, because of - I believe - how densely different areas of our retinas are populated with cones and rods. That's not the same thing that's going on when you look at a tree in front of you, and then the mountains miles away. The screen is flat. If you were playing a game that occupies 3D space, the game could mimic real depth by changing how it looks depending on where you're looking. It would blur things as they'd be blurred in real life. Or maybe they'd double up things close to us when we look far away, like when you hold a finger in front of your face and then stare far away, or vice-versa. I think we probably rarely notice these things in real life. Maybe this would make it feel magically more real.
tekromancr 11y ago
I think you have the idea backwards a bit. The system isn't interested in making the background blurry, it wants to make your exact eye point the section of the image with the highest fidelity. Then the renderer can just sort of half ass the rest of the image. It's a way to make the apparent image better than what the hardware would allow.
tdaltonc 11y ago
You're confusing a few ideas here. Focus has to do with the fact that the real world can only be projected on to our retina sharply at one distance. The letters in your peripheral vision are hard to make out not because they are out of focus but because they are falling on a part of the retina that has low resolution (also see "visual crowding"). The wikipedia page on optical focus has a good explanation of focus blur. Suffice it to say real scenes have dynamic focus blur and computer screens don't and we can subliminally tell the difference. This tech will make rendered scenes look more like real ones.

lifeformed 11y ago

Wasn't there a Microsoft demo using eye tracking, where only the focused area of the screen was rendered at full resolution, and the rest at lower blurred resolution? It saved a lot of GPU power by just needing to draw a tiny fraction of the game, and it looked indistinguishable to the player. What happened to that?

ggreer 11y ago
You're thinking of Foveated 3D Graphics[1]. Despite being a research project, they still got a 5-6x performance improvement. VR will get foveated rendering eventually. The performance improvements are too good to ignore. 1. http://research.microsoft.com/pubs/176610/foveated_final15.pdf http://research.microsoft.com/pubs/176610/foveated_final15.p...

ggreer 11y ago

I'm really looking forward to eye tracking in VR, because it will make for the scariest games ever. Games could notice that you're looking in one direction, then cause a creature to skitter by in your peripheral vision. By the time your eyes or head have moved, it's gone. Peripheral visual acuity is terrible, so you'd never get a good look at the thing until the game wanted you to.

J_Darnley 11y ago

As I said over here https://news.ycombinator.com/item?id=10512349 https://news.ycombinator.com/item?id=10512349 I will be dead before I allow an always on camera to be watching me at my computer.

CM30 11y ago

So, this article. Read it a few days ago, and while I do like some concepts (like the below mentioned monsters that move when you're looking away thing), a lot just seems like it'd be more annoying than anything else. Camera controlled by where you look? Oh, that's good, except quite a lot of the time in games you don't want to have the camera follow your gaze your screen. For example, in a lot of platformers, you want to be able to see as much of the area as possible in order to judge where you're jumping too. If the camera tracks your eye movements... well, someone's probably falling into the abyss pretty soon. I don't think anyone would want to be trying Grandmaster Galaxy or Champion Road with this sort of tech any time soon. There's also the way a lot of games have 'lock on' mechanics for targeting enemies. Your attacks go one way (towards the enemy), but the camera focuses somewhere else to give you a better view of your surroundings. I can see that falling apart pretty quickly if eye tracking got brought into it. But even assuming it'll mostly be used for games with a first person viewpoint (which in itself might be a limiting factor for things like virtual reality, since it's a smaller percentage of games than people like to think), it just seems like it'd be a bit too... precise for a lot of titles. People playing games look all over the screen for things that might be important in the future, and they probably don't want the camera bouncing around like a pinball as a result. The focus and depth of field aspects seem the only plausible ones for most games.

rawnlq 11y ago

The top post in /r/programming this year is about a guy writing his own eye tracking software: https://www.reddit.com/r/programming/comments/3ke7ug/eye_tracking_software_for_sufferers_of_alsmnd_can/ https://www.reddit.com/r/programming/comments/3ke7ug/eye_tra... https://www.reddit.com/r/software/comments/3kdghp/eye_tracking_software_for_sufferers_of_alsmnd_can/?ref=share&ref_source=link https://www.reddit.com/r/software/comments/3kdghp/eye_tracki...

sinxoveretothex 11y ago

I do not look forward to advertisements using this technology.

tracker1 11y ago

Is it sad that all I see this turning into is REALLY annoying ads in streaming apps? I'm a little cynical, but I think that VR interaction might benefit from this a little bit, but most of the interactions in gaming are better served be a more immersive display option, with full head tracking.

andersto 11y ago

Hey, I am working with Tobii and I saw this thread so let me try to give some background info about the Tobii EyeX to answer some of the questions in this thread. We really value the community and comments at Hacker News and we know there are equal amounts of excitement and doubts about eye tracking and I am gonna have a quick giveaway of 10 EyeX sets here in the next 48 hours as well so that some of you can experience it first hand! More on that below but here is some information: First off, we are not in a position to comment on products from other eye tracking companies globally, nor would we want to speak for them. So I am picking a few key issues here to discuss. 1)"Battery Drain issue" – The battery drain issue is probably more skewed towards mobile gaming and not for desktop/laptops and we are seeing little to no difference in battery drain when playing games with/without our eye tracker. 2)"Supplanting the hand controlled devices for aiming or changing POV” - We don’t intend to replace mice, keyboards or even to compete with them, but enhance the gaming experience with as natural interactions as possible. We already play games looking at the screen, and we can make your eyes a useful component of gameplay. 3)“Cannot do a high-res head pose estimation at high frame rate” - We are focused on eye tracking and have elements of head tracking but it is mainly on the point of gaze instead. 4)“Is the precision in the current tech good enough to replace the mouse for non-gaming activities?, web browsing or using a classical GUI with buttons and text fields and stuff” - Technically it is possible for similar levels of accuracy, but our intention was always to use it in combination with other modalities. We have plenty of people in the office who have become much less reliant on a mouse with the eye tracker but natural, learned interaction is not at a hands-free stage at the moment.

andersto 11y ago
5)“Eye fatigue comes in pretty quickly” - we don’t want you to do anything differently, but play as naturally as possible since the eye tracker is monitoring your eye movements. So there is really NO NEED to stay perfectly still, blink or be unnaturally close to the screen. 6)"Use gaze to control the camera pivot -- That sounds incredibly annoying.” - We understand the doubts but we are pretty confident that you might have a different opinion after trying it out! A gamespot reporter did the same:http://l.gamespot.com/1Mm7rar http://l.gamespot.com/1Mm7rar 7)“Making the foreground in sharper focus than background and peripheral objects. Why is this necessary?” - We want allow your normal human vision to exist within the game based on your natural focus to allow for real depth instead of a flat screen. @teckromancr @blakecaldwell and @tdaltonc got it right! 8)"Is it sad that all I see this turning into is REALLY annoying ads in streaming apps?"/"I do not look forward to advertisements using this technology.” - We have no intentions of making ads from eye tracking! We also have SDK agreements based on not allowing infringement of privacy. Its been 15 years in eye tracking for Tobii so personal integrity is something we take extremely seriously. So if any of you want to try out the tracker, all you have to do is tweet "let me try @tobiieyex via #hackernews!” and we will DM you for your details. The first 10 individuals to tweet it within 48 hours will get our latest Tobii EyeX shipped right to you, free of charge. All we want is for you to really play/test it out for yourself. We have not done this with any other community before but hacker news is pretty special to us!