4 ms·

Tobii's UI stuff is lame. But it is possible to make good UI for eye tracking. It requires rethinking how UI should work. What everyone has been doing so far i

Tobii's UI stuff is lame. But it is possible to make good UI for eye tracking. It requires rethinking how UI should work. What everyone has been doing so far is like slapping a touchscreen on a desktop operating system. Doesn't work! You need to redesign all the interactions for touch, like the iPhone did. Change how scrolling works, how selection works, how menus work. Redo the foundations. Eye tracking requires the same treatment. Touchscreens sucked before the iPhone. Eye tracking can have an iPhone moment too, but someone needs to spend years rethinking everything first.

roberttod 4y ago
Is that conjecture, or are there UIs you have in mind that work? I made an eye tracking UI for my masters thesis, with a few different components. Although they all worked, having my eyes tracked to control the UI felt very intense, it caused a lot of mental and physical strain. I'm not saying that my thesis is evidence it absolutely can't be done, but it was surprisingly uncomfortable even for what seemed like basic functionality like for example autofocusing the input boxes you are looking at. But maybe there are tweaks that can make it work, I think the main issue is the screen giving feedback on where you happen to be looking, which gives this unnatural feeling that where you look is causing side affects.
- modeless 4y ago
  I'd be interested to see your masters thesis! I made my own rudimentary eye tracker many years ago [1] which got me a job at a startup called Eyefluence, where I made a much better eye tracker. The deep learning revolution was very new at the time, and this was possibly the first deep learning based eye tracker. We did some really cool experimentation with eye controlled UI, in some ways more advanced than anything I've seen since. We were acquired by Google and I believe the technology is now shelved. But I still think that there is a path to a good eye controlled UI. Comfort is a matter of knowing the constraints of the eye and designing to them. https://james.darpinian.com/blog/eye-tracker https://james.darpinian.com/blog/eye-tracker
  - mlajtos 4y ago
    Really cool solution with the hot mirror! I always wondered if we will end up with cameras behind microdisplays. Some phones are already doing this, but doing that for VR is probably years away. The point you made about rethinking all UI interactions for eye-tracking — bull’s eye! Have you done some work along these lines in Eyefluence? I think the fruit company will gladly introduce new UI paradigm. They did some impressive stuff of improving eye-tracking accuraccy via refining synthetic images to real-looking ones with GANs. [0] [0] https://machinelearning.apple.com/research/gan https://machinelearning.apple.com/research/gan
    - modeless 4y ago
      Yes, rethinking UI for eye tracking is what Eyefluence was working on. And in fact we showed Apple all of our stuff before the acquisition. I believe we were in acquisition talks with them as well as Google. I spent a lot of time on the Apple campus tweaking our neural nets to their liking. This was prior to their publishing of that eye tracking work.
      - mlajtos 4y ago
        Is there any public record of what Eyefluence had done? Some demos maybe? I would really like to see that.
        modeless 4y ago
        There is actually an older demo still on YouTube: https://www.youtube.com/watch?v=TYcrQswVcnA&t=10s https://www.youtube.com/watch?v=TYcrQswVcnA&t=10s (full presentation: https://www.youtube.com/watch?v=iCZLll1l92g https://www.youtube.com/watch?v=iCZLll1l92g) The technique called "dual gaze" in the article has some similarity to some of the stuff we were doing. This was long before that paper was published, and I think there were several aspects of our design that were better than the one in that paper.
        mlajtos 4y ago
- nfw2 4y ago
  I would also be interested in reading that thesis. I was thinking a while ago that going to grad school to work on something like that would be cool. One thing that occurred to me is that shifting focus may work better as a gradual activation than an immediate one. In other words, focus would be an average of eye position over time, and after reaching a certain threshold on a certain element, the focus would shift. Not sure it would work in practice, but perhaps it could ameliorate the chaotic effects you were mentioning.
- towaway15463 4y ago
  Did you try having your gaze put elements into easy reach of a cursor or pointer instead of using the gaze as a pointer? Gaze is often an indicator of intent so rather than have it do something it would be more natural for it to make something else easier to do. An example, using a laser pointer style interface for VR controllers can sometimes be awkward because small motions can move the pointer a great distance making it hard to pick out a precise target. Gaze could be used as an aim mode where motion of the pointer would be limited/mapped to a smaller area surrounding the spot being looked at. It could also be used as a meta key to change the function of buttons, look to the right of a window and a button click fast forwards, look left and the same button rewinds.
deleted 4y ago
[deleted]

roberttod 4y ago

Is that conjecture, or are there UIs you have in mind that work? I made an eye tracking UI for my masters thesis, with a few different components. Although they all worked, having my eyes tracked to control the UI felt very intense, it caused a lot of mental and physical strain. I'm not saying that my thesis is evidence it absolutely can't be done, but it was surprisingly uncomfortable even for what seemed like basic functionality like for example autofocusing the input boxes you are looking at. But maybe there are tweaks that can make it work, I think the main issue is the screen giving feedback on where you happen to be looking, which gives this unnatural feeling that where you look is causing side affects.

modeless 4y ago
I'd be interested to see your masters thesis! I made my own rudimentary eye tracker many years ago [1] which got me a job at a startup called Eyefluence, where I made a much better eye tracker. The deep learning revolution was very new at the time, and this was possibly the first deep learning based eye tracker. We did some really cool experimentation with eye controlled UI, in some ways more advanced than anything I've seen since. We were acquired by Google and I believe the technology is now shelved. But I still think that there is a path to a good eye controlled UI. Comfort is a matter of knowing the constraints of the eye and designing to them. https://james.darpinian.com/blog/eye-tracker https://james.darpinian.com/blog/eye-tracker
- mlajtos 4y ago
  Really cool solution with the hot mirror! I always wondered if we will end up with cameras behind microdisplays. Some phones are already doing this, but doing that for VR is probably years away. The point you made about rethinking all UI interactions for eye-tracking — bull’s eye! Have you done some work along these lines in Eyefluence? I think the fruit company will gladly introduce new UI paradigm. They did some impressive stuff of improving eye-tracking accuraccy via refining synthetic images to real-looking ones with GANs. [0] [0] https://machinelearning.apple.com/research/gan https://machinelearning.apple.com/research/gan
  - modeless 4y ago
    Yes, rethinking UI for eye tracking is what Eyefluence was working on. And in fact we showed Apple all of our stuff before the acquisition. I believe we were in acquisition talks with them as well as Google. I spent a lot of time on the Apple campus tweaking our neural nets to their liking. This was prior to their publishing of that eye tracking work.
    - mlajtos 4y ago
      Is there any public record of what Eyefluence had done? Some demos maybe? I would really like to see that.
      - modeless 4y ago
        There is actually an older demo still on YouTube: https://www.youtube.com/watch?v=TYcrQswVcnA&t=10s https://www.youtube.com/watch?v=TYcrQswVcnA&t=10s (full presentation: https://www.youtube.com/watch?v=iCZLll1l92g https://www.youtube.com/watch?v=iCZLll1l92g) The technique called "dual gaze" in the article has some similarity to some of the stuff we were doing. This was long before that paper was published, and I think there were several aspects of our design that were better than the one in that paper.
        mlajtos 4y ago
nfw2 4y ago
I would also be interested in reading that thesis. I was thinking a while ago that going to grad school to work on something like that would be cool. One thing that occurred to me is that shifting focus may work better as a gradual activation than an immediate one. In other words, focus would be an average of eye position over time, and after reaching a certain threshold on a certain element, the focus would shift. Not sure it would work in practice, but perhaps it could ameliorate the chaotic effects you were mentioning.
towaway15463 4y ago
Did you try having your gaze put elements into easy reach of a cursor or pointer instead of using the gaze as a pointer? Gaze is often an indicator of intent so rather than have it do something it would be more natural for it to make something else easier to do. An example, using a laser pointer style interface for VR controllers can sometimes be awkward because small motions can move the pointer a great distance making it hard to pick out a precise target. Gaze could be used as an aim mode where motion of the pointer would be limited/mapped to a smaller area surrounding the spot being looked at. It could also be used as a meta key to change the function of buttons, look to the right of a window and a button click fast forwards, look left and the same button rewinds.

deleted 4y ago

[deleted]