16 ms·

Webgazer.js webcam eye tracking on the browser

138 points by lazyjeff 6y ago

natebleker 6y ago
Disclaimer, I am the primary engineer of a commercial eye tracking system. Tools like these popup every now and then are a nice tool for rough estimation of gaze. Fixation tracking is a large complicated issue that usually requires some sort of calibration to get more precise results. By sidestepping the calibration problem, much higher subject compliance can be achieved since you don't have a grad student barking confusing order at you. The downside to this is the noise you see in the tracking results. For those interested a product that produces similar results is pupil labs ambient gaze tracking "Core" research headset[0]. [0] https://pupil-labs.com/products/core/ https://pupil-labs.com/products/core/
- xamuel 6y ago
  In your estimation, how far away are we from eye-tracking software being able to detect the start of a microsaccade, estimate where the gaze is moving toward, and draw new contents on the screen there before the gaze even reaches that point? I would think that by "hacking" into human brain "vulnerabilities" like saccadic masking and chronostasis, such software could potentially yield seriously trippy and mind-altering results!
  - thaumasiotes 6y ago
    "How far away are we" is a strange question to ask about something we already do. You can read about this technique being used to investigate how we read here: https://docs.microsoft.com/en-us/typography/develop/word-recognition https://docs.microsoft.com/en-us/typography/develop/word-rec... Note the citation to a study that used the technique in 1975.
  - natebleker 6y ago
    There’s two parts to that implementation of such a system, and they’re both interesting! The first part is detection of a micro saccade which is already available in research systems I have personally worked on. You can basically crank up the camera frame rate until you’re around the micro saccade range and do some clustering analysis on the positional data to decouple the hardware moving vs the face. The second part is having a fast enough reliable commercial grade displa system to present stimulus on. Displays are very fickle in practice and getting your hands on one that can run with adequate color, contrast, and brightness at speed is currently very difficult. One angle under research currently is the perception of stimulus during saccades and micro saccades. There’s quite a bit of time and effort in the industry going into neurological assessments through saccades, the tools coming out of this are really starting to come down in price. This opens the door for a bunch of lower priced research options, such as the parent article, to enable a much more rapid pace of understanding.
    - tootie 6y ago
      That's fascinating, but I'm wondering what is the practical value of detecting microsaccades? Aren't they just involuntary twitching?
      - natebleker 6y ago
        They’re looking to be increasingly more interesting as a biomarker for MS among other things [0]. There’s a lot we still don’t know about our visual systems and the advent of lower cost hardware is allowing huge amount of interesting questions to be hypothesized and tested. It’s a great time to be in the science behind all of this, like the authors at Brown on this paper! [0] https://iovs.arvojournals.org/article.aspx?articleid=2670267 https://iovs.arvojournals.org/article.aspx?articleid=2670267
      - taneq 6y ago
        For the duration of the microsaccade, you're blind. So if something changes onscreen it's much harder to see. IIRC people have used small orientation changes during microsaccades for redirected walking in VR. You feel like you're walking straight but you're actually curving back on yourself. Edit: I think that was just detecting full-on saccades but a microsaccade version would be smoother and harder to detect. Source: https://blog.siggraph.org/2018/05/challenge-accepted-infinite-walking-in-vr.html/ https://blog.siggraph.org/2018/05/challenge-accepted-infinit...
  - trishume 6y ago
    I believe this might be available in less than 5 years, potentially less than 2, provided both http://www.adhawkmicrosystems.com/ http://www.adhawkmicrosystems.com/ and https://www.kura.tech/ https://www.kura.tech/ ship and partner with each other. If either one of those companies dies or their technology doesn't work out like I hope, then it might be a while longer.
- lazyjeff 6y ago
  Author here (I didn't use Show HN because it's a larger collaborative project). Yes this is basically a tradeoff between accuracy and flexibility+accessibility. For usability testing, physical eye trackers have to be used one person at a time (no simultaneous use), use an experimenter's time to schedule and administer, can only be used for a short period, and only work with local participants. But yes they will probably always have better accuracy, and detect saccades/fixations better which are also great for psychology studies. The other thing is if you want to make a consumer application (like browser game, or accessibility mode), then it's more practical to have people just consent to having their webcam turned on, than to go out and buy an eye tracker.
- tootie 6y ago
  It depends on your application. If you're trying to replace your mouse with gaze, then yes you need careful calibration. I have worked on some purpose-built gaze interactive experiences and coarse accuracy without calibration can be enough to do something useful.
- diesal11 6y ago
  What are your thoughts on newer ML based approaches like MITs "Eye Tracking for Everyone"? https://gazecapture.csail.mit.edu/ https://gazecapture.csail.mit.edu/
chrisweekly 6y ago
Whoa. I have a strange mix of reactions to this. It's impressive, and interesting, and worthwhile, slightly creepy, and potentially hugely useful for a11y, democratizing cutting-edge HCI, AR, gaming, etc etc. Surprised it's not getting more attention!
- chrisco255 6y ago
  One good thing about webcam access on the web is that browsers require explicit access by the user and browser tabs clearly indicate when a tab is recording with either the mic or camera.
  - chrisweekly 6y ago
    Yes! Hence the "slightly" modifier. :)
- aagd 6y ago
  Maybe because it's been around for a couple of years?
ebg13 6y ago
I tried my best on the calibration demo page in Safari and consistently get a 0% accuracy result. It calibrates sort of ok in Firefox. I guess this is heavily browser dependent? But why?
- lazyjeff 6y ago
  Hmm, not sure -- do you see the gaze dot when you're looking at the center circle?
  - ebg13 6y ago
    I see a gaze dot. It just seems kinda like Safari is accumulating some tracking error that Firefox doesn't. Both browsers find my face accurately. (side note: when the gaze dot overlaps the calibration circles, the circles become unclickable without some very careful fiddling)
cbrgm 6y ago
Can you recommend any literature to get an introduction into eye-tracking at all?
- natebleker 6y ago
  Here’s an old school high level overview of pupil tracking methods[0]. The focus recently has been moving these older computer vision methodologies to ML without introducing too many errors. [0] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.155.5186&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.155...
stared 6y ago
I had an idea for an evil-eye game, in the style of Happy Tree Friends or so. That is: a lot of cute creatures around, and nothing happens (they may giggle, jump or blink occasionally) until you look at them. If you do...
- lecarore 6y ago
  That's a really cool use case for the library ! I think the calibration stage could be made easier with some "click as fast as you can on the randomly appearing circle on screen" gamification, and just assume that people look at what they click on by default. It could be some little character changing color at random and you need to click it at that point, to make sure the user keeps his eyes on it for a few seconds before clicking.
Meph504 6y ago
The listed use cases here struck me as odd, I can't imagine why I or anyone wanting to read a news article would allow the site access to my webcam. Nor do I think most users would turn on their cameras to help with any given sites analytics. So aside from games, what is the actual use case here ?
- lazyjeff 6y ago
  Great question. There's a use for usability testing, where paid participants currently sit at an application in a lab to do eye tracking. So there can be a similar version online for if let's say, Bloomberg is testing a new version of their online terminal. Multiple large companies exist that solely develop eye trackers, so there's quite a market for this. Plus it does not have to be monetary compensation. I could imagine eye tracking as an option to get around the current newspaper paywalls. If you want to read an article, you can choose to pay (subscription) as usual, or you can read them for free if you let the New York Times see what articles you read, for how long, in which order, or what ads you looked it.
  - Meph504 6y ago
    I suppose that might be an option, but in my experience people are highly unlikely to grant a site permission to their camera if they aren't using it as part of a conference call. But I do agree if you have some A/B testing, that the users are doing it as a part of a study, it might be usable.
- jasonjayr 6y ago
  Nielson Research in years past would ask volunteers to attach special equipment to their televisions to report back what they were watching, and for how long. Even nowadays, I have a relative that carries an always-listening device to track ambient music being played/heard. A technology like this would make it easier to run a program with a subset of volunteers online, w/o them needing any special equipment. Of course, given that slightly more people (than tech communities) are aware of the dangers of an uncovered camera, the possibility of using this broadly w/o consent is nearly nil.
- 6y ago
pmlnr 6y ago
Now add this to vidconf systems by figuring out which speaker is who, and make a button to talk only to them.
awake 6y ago
In the brown undergraduate computer vision course one of the homework’s (or final project; my memory is a little hazy) was to improve the tracking of webgazer. People were pretty successful from what I remember. As an undergrad I didn’t think much of it but as a grad student I would be slightly mortified if undergraduates were improving on my current research for homework.
- dylan604 6y ago
  Why? Someone smarter will always follow
  - yjftsjthsd-h 6y ago
    Not even just smarter; we all stand on the shoulders of giants, and every year the barrier to entry falls that much more.
- samvaran 6y ago
  That's not too surprising, tbh. Building the baseline solution or platform from the ground up is often the most difficult part, and by the end of such a project, you end up with a 100 ideas about how to make it better. But either due to time constraints, a "perfect is the enemy of done" philosophy, or simply the fact that you reach a natural crossroads and there are multiple valid ways forwards, you have to draw a line in the sand and release at some point. It's natural that there my be low-hanging fruit for some people to pick up and improve on for some projects, particularly if they have different end goals than the initial project had.
gruez 6y ago
Can't wait until this makes its way to adtech. You can have sites that hold their "premium" content hostage until you allow camera access and proved (via eye tracking) that you looked at their ad.
- _31 6y ago
  "Wanna play a game? Just move the ball to the hole with your eyes. Oh and we need access to your camera."
- deleted 6y ago
  [deleted]
- A4ET8a8uTh0 6y ago
  I wonder. Obviously its technologically possible, but would average user accept it?
  - y4bvusw 6y ago
    It is both possible and being pursued actively. I believe the average user will accept it. The average user has an uncovered camera on the computer and almost everyone has an uncovered camera on the phone. Requiring a cookie banner notification did nothing but make it more annoying to use websites, with less accountability on the site administrator for all the questionable actions they can freely undertake, now with your consent! But it goes far beyond that - there will be cameras embedded in digital billboards/posters you walk past that track your gaze and provide targeted ads based on either facial ID or device ID, the ad effectively learning who you are through your leaky apps or always-on bluetooth.
- oceliker 6y ago
  “Please drink verification can”
  - ketzo 6y ago
    Boy, I love to see an r/lol reference on HN. Gotta be a lot of crossover, right?
evan_ 6y ago
This seems really cool but it doesn't seem to let me choose which webcam to use- it seems to default to a virtual device I use for streaming that isn't set up right now so I can't really try it out.
- nl 6y ago
  I've had this issue with Chrome on OSX where it won't allow camera selection. Firefox (and I think Safari) both do. It's unclear why Chrome does this - it has a preference for which camera but it seems ignored and takes the most recently installed one.
dsteinman 6y ago
I tried using this one time but unfortunately found it didn't work when I wore my reading glasses. The screen reflection off my glasses just whites out both eyes.
giraj 6y ago
Had a blast at LauzHack '16 using this library to make a Chrome extension enabling "no-hands scrolling"[1]. Sadly, reading web articles while eating with both hands is still a dream. We managed to get okay-ish results on a MacBook Pro in perfect lighting from time to time, but nothing consistent. In variable lighting and on lower-end laptops we found it impossible. I wonder if it would be possible with a better webcam and good, consistent lighting. [1] https://github.com/jarlg/lookmanohands https://github.com/jarlg/lookmanohands
bscphil 6y ago
The "move the ball around with your eyes" demo doesn't work at all for me in Safari on a MacBook. I can only move the ball with the mouse. The other demo works okay, but very low accuracy (like 50%). Firefox on the same computer is unusably slow, generates a new "data point" (the visible dot) only every few seconds. Accuracy was about 10%.
mrburton 6y ago
Sadly my eyes are too small to actually track the pupils. I remember when I got my ID, the lady taking the picture said "Sir, please open your eyes". I looked at her and asked "They are open... you don't see that?" She laughed and asked if I had been smoking. :/
paq85 6y ago
If you want to see how well webcam eye tracking works check this out https://www.realeye.io/test/f80dd676-3b55-4d2c-931c-d925b068a740/ https://www.realeye.io/test/f80dd676-3b55-4d2c-931c-d925b068... It's using WebGazer as it's core - but with many improvements done by us. BTW I'm RealEye co-founder so AMA :)
HomeDeLaPot 6y ago
I just don't see the point of this. I don't think I've ever lost track of my eyes.

138 points by lazyjeff 6y ago

Disclaimer, I am the primary engineer of a commercial eye tracking system. Tools like these popup every now and then are a nice tool for rough estimation of gaze. Fixation tracking is a large complicated issue that usually requires some sort of calibration to get more precise results. By sidestepping the calibration problem, much higher subject compliance can be achieved since you don't have a grad student barking confusing order at you. The downside to this is the noise you see in the tracking results. For those interested a product that produces similar results is pupil labs ambient gaze tracking "Core" research headset[0]. [0] https://pupil-labs.com/products/core/ https://pupil-labs.com/products/core/

xamuel 6y ago
In your estimation, how far away are we from eye-tracking software being able to detect the start of a microsaccade, estimate where the gaze is moving toward, and draw new contents on the screen there before the gaze even reaches that point? I would think that by "hacking" into human brain "vulnerabilities" like saccadic masking and chronostasis, such software could potentially yield seriously trippy and mind-altering results!
- thaumasiotes 6y ago
  "How far away are we" is a strange question to ask about something we already do. You can read about this technique being used to investigate how we read here: https://docs.microsoft.com/en-us/typography/develop/word-recognition https://docs.microsoft.com/en-us/typography/develop/word-rec... Note the citation to a study that used the technique in 1975.
- natebleker 6y ago
  There’s two parts to that implementation of such a system, and they’re both interesting! The first part is detection of a micro saccade which is already available in research systems I have personally worked on. You can basically crank up the camera frame rate until you’re around the micro saccade range and do some clustering analysis on the positional data to decouple the hardware moving vs the face. The second part is having a fast enough reliable commercial grade displa system to present stimulus on. Displays are very fickle in practice and getting your hands on one that can run with adequate color, contrast, and brightness at speed is currently very difficult. One angle under research currently is the perception of stimulus during saccades and micro saccades. There’s quite a bit of time and effort in the industry going into neurological assessments through saccades, the tools coming out of this are really starting to come down in price. This opens the door for a bunch of lower priced research options, such as the parent article, to enable a much more rapid pace of understanding.
  - tootie 6y ago
    That's fascinating, but I'm wondering what is the practical value of detecting microsaccades? Aren't they just involuntary twitching?
    - natebleker 6y ago
      They’re looking to be increasingly more interesting as a biomarker for MS among other things [0]. There’s a lot we still don’t know about our visual systems and the advent of lower cost hardware is allowing huge amount of interesting questions to be hypothesized and tested. It’s a great time to be in the science behind all of this, like the authors at Brown on this paper! [0] https://iovs.arvojournals.org/article.aspx?articleid=2670267 https://iovs.arvojournals.org/article.aspx?articleid=2670267
    - taneq 6y ago
      For the duration of the microsaccade, you're blind. So if something changes onscreen it's much harder to see. IIRC people have used small orientation changes during microsaccades for redirected walking in VR. You feel like you're walking straight but you're actually curving back on yourself. Edit: I think that was just detecting full-on saccades but a microsaccade version would be smoother and harder to detect. Source: https://blog.siggraph.org/2018/05/challenge-accepted-infinite-walking-in-vr.html/ https://blog.siggraph.org/2018/05/challenge-accepted-infinit...
- trishume 6y ago
  I believe this might be available in less than 5 years, potentially less than 2, provided both http://www.adhawkmicrosystems.com/ http://www.adhawkmicrosystems.com/ and https://www.kura.tech/ https://www.kura.tech/ ship and partner with each other. If either one of those companies dies or their technology doesn't work out like I hope, then it might be a while longer.
lazyjeff 6y ago
Author here (I didn't use Show HN because it's a larger collaborative project). Yes this is basically a tradeoff between accuracy and flexibility+accessibility. For usability testing, physical eye trackers have to be used one person at a time (no simultaneous use), use an experimenter's time to schedule and administer, can only be used for a short period, and only work with local participants. But yes they will probably always have better accuracy, and detect saccades/fixations better which are also great for psychology studies. The other thing is if you want to make a consumer application (like browser game, or accessibility mode), then it's more practical to have people just consent to having their webcam turned on, than to go out and buy an eye tracker.
tootie 6y ago
It depends on your application. If you're trying to replace your mouse with gaze, then yes you need careful calibration. I have worked on some purpose-built gaze interactive experiences and coarse accuracy without calibration can be enough to do something useful.
diesal11 6y ago
What are your thoughts on newer ML based approaches like MITs "Eye Tracking for Everyone"? https://gazecapture.csail.mit.edu/ https://gazecapture.csail.mit.edu/

chrisweekly 6y ago

Whoa. I have a strange mix of reactions to this. It's impressive, and interesting, and worthwhile, slightly creepy, and potentially hugely useful for a11y, democratizing cutting-edge HCI, AR, gaming, etc etc. Surprised it's not getting more attention!

chrisco255 6y ago
One good thing about webcam access on the web is that browsers require explicit access by the user and browser tabs clearly indicate when a tab is recording with either the mic or camera.
- chrisweekly 6y ago
  Yes! Hence the "slightly" modifier. :)
aagd 6y ago
Maybe because it's been around for a couple of years?

ebg13 6y ago

I tried my best on the calibration demo page in Safari and consistently get a 0% accuracy result. It calibrates sort of ok in Firefox. I guess this is heavily browser dependent? But why?

lazyjeff 6y ago
Hmm, not sure -- do you see the gaze dot when you're looking at the center circle?
- ebg13 6y ago
  I see a gaze dot. It just seems kinda like Safari is accumulating some tracking error that Firefox doesn't. Both browsers find my face accurately. (side note: when the gaze dot overlaps the calibration circles, the circles become unclickable without some very careful fiddling)

cbrgm 6y ago

Can you recommend any literature to get an introduction into eye-tracking at all?

natebleker 6y ago
Here’s an old school high level overview of pupil tracking methods[0]. The focus recently has been moving these older computer vision methodologies to ML without introducing too many errors. [0] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.155.5186&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.155...

stared 6y ago

I had an idea for an evil-eye game, in the style of Happy Tree Friends or so. That is: a lot of cute creatures around, and nothing happens (they may giggle, jump or blink occasionally) until you look at them. If you do...

lecarore 6y ago
That's a really cool use case for the library ! I think the calibration stage could be made easier with some "click as fast as you can on the randomly appearing circle on screen" gamification, and just assume that people look at what they click on by default. It could be some little character changing color at random and you need to click it at that point, to make sure the user keeps his eyes on it for a few seconds before clicking.

Meph504 6y ago

The listed use cases here struck me as odd, I can't imagine why I or anyone wanting to read a news article would allow the site access to my webcam. Nor do I think most users would turn on their cameras to help with any given sites analytics. So aside from games, what is the actual use case here ?

lazyjeff 6y ago
Great question. There's a use for usability testing, where paid participants currently sit at an application in a lab to do eye tracking. So there can be a similar version online for if let's say, Bloomberg is testing a new version of their online terminal. Multiple large companies exist that solely develop eye trackers, so there's quite a market for this. Plus it does not have to be monetary compensation. I could imagine eye tracking as an option to get around the current newspaper paywalls. If you want to read an article, you can choose to pay (subscription) as usual, or you can read them for free if you let the New York Times see what articles you read, for how long, in which order, or what ads you looked it.
- Meph504 6y ago
  I suppose that might be an option, but in my experience people are highly unlikely to grant a site permission to their camera if they aren't using it as part of a conference call. But I do agree if you have some A/B testing, that the users are doing it as a part of a study, it might be usable.
jasonjayr 6y ago
Nielson Research in years past would ask volunteers to attach special equipment to their televisions to report back what they were watching, and for how long. Even nowadays, I have a relative that carries an always-listening device to track ambient music being played/heard. A technology like this would make it easier to run a program with a subset of volunteers online, w/o them needing any special equipment. Of course, given that slightly more people (than tech communities) are aware of the dangers of an uncovered camera, the possibility of using this broadly w/o consent is nearly nil.
6y ago

pmlnr 6y ago

Now add this to vidconf systems by figuring out which speaker is who, and make a button to talk only to them.

awake 6y ago

In the brown undergraduate computer vision course one of the homework’s (or final project; my memory is a little hazy) was to improve the tracking of webgazer. People were pretty successful from what I remember. As an undergrad I didn’t think much of it but as a grad student I would be slightly mortified if undergraduates were improving on my current research for homework.

dylan604 6y ago
Why? Someone smarter will always follow
- yjftsjthsd-h 6y ago
  Not even just smarter; we all stand on the shoulders of giants, and every year the barrier to entry falls that much more.
samvaran 6y ago
That's not too surprising, tbh. Building the baseline solution or platform from the ground up is often the most difficult part, and by the end of such a project, you end up with a 100 ideas about how to make it better. But either due to time constraints, a "perfect is the enemy of done" philosophy, or simply the fact that you reach a natural crossroads and there are multiple valid ways forwards, you have to draw a line in the sand and release at some point. It's natural that there my be low-hanging fruit for some people to pick up and improve on for some projects, particularly if they have different end goals than the initial project had.

gruez 6y ago

Can't wait until this makes its way to adtech. You can have sites that hold their "premium" content hostage until you allow camera access and proved (via eye tracking) that you looked at their ad.

_31 6y ago
"Wanna play a game? Just move the ball to the hole with your eyes. Oh and we need access to your camera."
deleted 6y ago
[deleted]
A4ET8a8uTh0 6y ago
I wonder. Obviously its technologically possible, but would average user accept it?
- y4bvusw 6y ago
  It is both possible and being pursued actively. I believe the average user will accept it. The average user has an uncovered camera on the computer and almost everyone has an uncovered camera on the phone. Requiring a cookie banner notification did nothing but make it more annoying to use websites, with less accountability on the site administrator for all the questionable actions they can freely undertake, now with your consent! But it goes far beyond that - there will be cameras embedded in digital billboards/posters you walk past that track your gaze and provide targeted ads based on either facial ID or device ID, the ad effectively learning who you are through your leaky apps or always-on bluetooth.
oceliker 6y ago
“Please drink verification can”
- ketzo 6y ago
  Boy, I love to see an r/lol reference on HN. Gotta be a lot of crossover, right?

evan_ 6y ago

This seems really cool but it doesn't seem to let me choose which webcam to use- it seems to default to a virtual device I use for streaming that isn't set up right now so I can't really try it out.

nl 6y ago
I've had this issue with Chrome on OSX where it won't allow camera selection. Firefox (and I think Safari) both do. It's unclear why Chrome does this - it has a preference for which camera but it seems ignored and takes the most recently installed one.

dsteinman 6y ago

I tried using this one time but unfortunately found it didn't work when I wore my reading glasses. The screen reflection off my glasses just whites out both eyes.

giraj 6y ago

Had a blast at LauzHack '16 using this library to make a Chrome extension enabling "no-hands scrolling"[1]. Sadly, reading web articles while eating with both hands is still a dream. We managed to get okay-ish results on a MacBook Pro in perfect lighting from time to time, but nothing consistent. In variable lighting and on lower-end laptops we found it impossible. I wonder if it would be possible with a better webcam and good, consistent lighting. [1] https://github.com/jarlg/lookmanohands https://github.com/jarlg/lookmanohands

bscphil 6y ago

The "move the ball around with your eyes" demo doesn't work at all for me in Safari on a MacBook. I can only move the ball with the mouse. The other demo works okay, but very low accuracy (like 50%). Firefox on the same computer is unusably slow, generates a new "data point" (the visible dot) only every few seconds. Accuracy was about 10%.

mrburton 6y ago

Sadly my eyes are too small to actually track the pupils. I remember when I got my ID, the lady taking the picture said "Sir, please open your eyes". I looked at her and asked "They are open... you don't see that?" She laughed and asked if I had been smoking. :/

paq85 6y ago

If you want to see how well webcam eye tracking works check this out https://www.realeye.io/test/f80dd676-3b55-4d2c-931c-d925b068a740/ https://www.realeye.io/test/f80dd676-3b55-4d2c-931c-d925b068... It's using WebGazer as it's core - but with many improvements done by us. BTW I'm RealEye co-founder so AMA :)

HomeDeLaPot 6y ago

I just don't see the point of this. I don't think I've ever lost track of my eyes.