6 ms·

I found it notable Apple has this to say about eye tracking under the "Learn more about privacy and security" section: >Eye input is not shared with Apple, thi

I found it notable Apple has this to say about eye tracking under the "Learn more about privacy and security" section: >Eye input is not shared with Apple, third-party apps, or websites. Only your final selections are transmitted when you tap your fingers together. Along with this small video: https://www.apple.com/105/media/us/apple-vision-pro/2023/7e268c13-eb22-493d-a860-f0637bacb569/anim/drawer-privacy-eye-input/large.mp4 https://www.apple.com/105/media/us/apple-vision-pro/2023/7e2... Which is great and important for me to see, but I'm unsure how that jives with features such as foveated rendering.

reaperman 3y ago
You can get a surprising amount of information about cognitive processes from eye tracking. I worked on the industrial version of HP Reverb G2 which included a PPG sensor and Tobii eye tracking. Was really an amazing product which suffered from having to use third party platforms like Windows Mixed Reality and OpenXR which didn’t have first class support for what was possible. One notable takeaway is that eye tracking of saccades is loosely correlated with cognitive load. In fact, “cognitive load” was a product that would be licensed to developers to use in their apps so that difficulty levels could be dynamically adjusted to keep users in a sweet spot for learning. Especially when combined with a PPG sensor which can in theory detect if a man gets aroused (HP never investigated developing a model for that, but some comments from friends who were DoD-aligned contractors expressed alternating concern or interest in it, because they knew from their own career and independent research what applications might be possible with those technologies) …it could be used to infer things which someone might prefer to keep private (I.e. which categories of things they have an emotional response to). Apple definitely is aware of this and is heading off potential issues by not only not providing that data to third party app developers but also even to Apple itself. Apple does have a strong pre-existing framework for dealing with sensitive health data, so they could have decided to capture similar data and treat it the same way they deal with ECG data. HP did not have a HIPAA compatible framework so they threw away 99% of the PPG data and distilled it down to just “heart rate” which was the only data source available to app developers from the PPG sensor.
gnicholas 3y ago
I imagine there will be some developers that try to have analytics fire when certain events happen, correlated with eye-hovering. Hopefully Apple will catch these tricks before they can go live. The potential value of knowing where someone's eyes are (whether they saw an advertisement, for example) could be very high, so I envision a cat-and-mouse game.
- FabHK 3y ago
  Apple addressed that in the keynote. There’ll be an invisible canvas over the window content, and the app never learns where you look (nor Apple, ie it is not further stored or processed). Only when you click (ie tap your thumb and index together) does the app get an event. (Downside is that one loses the capability for hovering, but strikes me as the correct trade off.)
  - gnicholas 3y ago
    Sure, that works for certain contexts. But what about a game, where your vision is used to aim a weapon? The app has to know where you are looking in order to know where to shoot. If it's an online game, then the information cannot be isolated on the device since you're shooting at other players. Now imagine the game has advertisements, or loot boxes, or something else that the game company wants to know if you've looked at. How does Apple prevent them from learning that, if they're able to know what they need to know for the game to function as a regular FPS?
    - tsimionescu 3y ago
      Shooting is typically an action you take, not just a matter of the direction you're pointing in - so this fits the model well: the game gets information about motion to swing the camera, and gets the exact position when you press a button, but doesn't get to know exactly what you're looking at if you're not pressing the button. The question will be how easy it is to reconstruct that information from the last exact position + movement info...
      - 3y ago

reaperman 3y ago

You can get a surprising amount of information about cognitive processes from eye tracking. I worked on the industrial version of HP Reverb G2 which included a PPG sensor and Tobii eye tracking. Was really an amazing product which suffered from having to use third party platforms like Windows Mixed Reality and OpenXR which didn’t have first class support for what was possible. One notable takeaway is that eye tracking of saccades is loosely correlated with cognitive load. In fact, “cognitive load” was a product that would be licensed to developers to use in their apps so that difficulty levels could be dynamically adjusted to keep users in a sweet spot for learning. Especially when combined with a PPG sensor which can in theory detect if a man gets aroused (HP never investigated developing a model for that, but some comments from friends who were DoD-aligned contractors expressed alternating concern or interest in it, because they knew from their own career and independent research what applications might be possible with those technologies) …it could be used to infer things which someone might prefer to keep private (I.e. which categories of things they have an emotional response to). Apple definitely is aware of this and is heading off potential issues by not only not providing that data to third party app developers but also even to Apple itself. Apple does have a strong pre-existing framework for dealing with sensitive health data, so they could have decided to capture similar data and treat it the same way they deal with ECG data. HP did not have a HIPAA compatible framework so they threw away 99% of the PPG data and distilled it down to just “heart rate” which was the only data source available to app developers from the PPG sensor.

gnicholas 3y ago

I imagine there will be some developers that try to have analytics fire when certain events happen, correlated with eye-hovering. Hopefully Apple will catch these tricks before they can go live. The potential value of knowing where someone's eyes are (whether they saw an advertisement, for example) could be very high, so I envision a cat-and-mouse game.

FabHK 3y ago
Apple addressed that in the keynote. There’ll be an invisible canvas over the window content, and the app never learns where you look (nor Apple, ie it is not further stored or processed). Only when you click (ie tap your thumb and index together) does the app get an event. (Downside is that one loses the capability for hovering, but strikes me as the correct trade off.)
- gnicholas 3y ago
  Sure, that works for certain contexts. But what about a game, where your vision is used to aim a weapon? The app has to know where you are looking in order to know where to shoot. If it's an online game, then the information cannot be isolated on the device since you're shooting at other players. Now imagine the game has advertisements, or loot boxes, or something else that the game company wants to know if you've looked at. How does Apple prevent them from learning that, if they're able to know what they need to know for the game to function as a regular FPS?
  - tsimionescu 3y ago
    Shooting is typically an action you take, not just a matter of the direction you're pointing in - so this fits the model well: the game gets information about motion to swing the camera, and gets the exact position when you press a button, but doesn't get to know exactly what you're looking at if you're not pressing the button. The question will be how easy it is to reconstruct that information from the last exact position + movement info...
    - 3y ago