15 ms·

Apple announces new accessibility features, including eye tracking

500 points by dmd 2y ago

pcl 2y ago
Bummer the eye tracking is iOS only. I’ve been wanting focus-follows-eyes for decades.
- npongratz 2y ago
  I share the sentiment. I've long noted in situations with grids of terminal windows where focus-follows-eyes would be so much faster and present less friction than using hotkeys or focus-follows-mouse.
dagmx 2y ago
This is one major advantage to Apple sharing a foundation across all their devices. Vision Pro introduced eye tracking to their systems as a new input modality, and now it trickles down to their other platforms. I am surprised CarPlay didn’t have voice control before this though.
- callwhendone 2y ago
  It would be amazing if it gets carried over to the Mac.
- simlevesque 2y ago
  CarPlay devices aren't really powerful.
  - andrewmunsell 2y ago
    CarPlay is rendered by the phone itself, so it's not strictly a function of how powerful the car infotainment is. You've been able to talk to Siri since the beginning of CarPlay so additional voice control is really just an accessibility thing
    - joezydeco 2y ago
      Some cars already have a voice control button on the wheel for their existing system which, if done correctly, is overriden by Siri+CarPlay. Which is really nice when it works.
      - dhosek 2y ago
        I’m able to control carplay by just saying “Hey Siri.” Siri’s abilities tend to fluctuate based on what Apple is doing on the server, the phase of the moon and whether I remembered to sacrifice a chicken that morning, but otherwise, it seems to work fine.
  - Tagbert 2y ago
    CarPlay devices (car components) are essentially playing a streaming video of a hidden display generated by the phone. CarPlay also lets those devices send back touch events to trigger buttons and other interactions. Very little process is done on the vehicle. BTW if you are plugged in to CarPlay and take a screen shot, it will include the hidden CarPlay screen.
- jsheard 2y ago
  Is this really likely to be downstream of the Vision Pro implementation? I would think that eye-tracking with specialized hardware at a fixed position very close to the eye is very different to doing it at a distance with a general purpose front facing camera.
  - iloveyouocean 2y ago
    The 'tracking eyes' part is different, but once you have eye position data, the 'how the eyes interact with the interface' could be very similar.
  - dmicah 2y ago
    Typically eye-trackers work by illuminating the eyes with near infrared light and using infrared cameras. This creates a higher contrast image of the pupils, etc. I assume Apple is doing this in the Vision Pro. Eye-tracking can also be done with just visible light, though. Apple has the benefit of knowing where all the user interface elements are on screen, so eye-tracking in this on the iPhone or iPad doesn't need to be high precision. Knowledge of the position of the items can help to reduce the uncertainty of what is being fixated on.
    - neverokay 2y ago
      So there isn’t much more to it than getting a good resolution image of the eye from any distance, every millisecond. Precision is the issue because we are mostly moving our eyes in about 8 directions, there’s no precision because we don’t know how to measure focusing of our eye lens with a camera yet (unless that too is just a matter of getting a picture). Squinting would be the closest thing to physically expressing focusing. So the camera needs to know I’m looking left with my eye, followed by a squint to achieve precision. Seems stressful though. Gonna need AI just to do noise cancelling of involuntary things your eyes do like pupil dilation, blinking.
  - dagmx 2y ago
    Not the hardware side, but the software side. Implementing all the various behaviours and security models for the eye tracking and baking it into SwiftUI, means that it translates over easier once they figure out the hardware aspect. But the iPads and iPhones with FaceID have had eye tracking capabilities for a while, just not useful in UI.
devinprater 2y ago
I wonder what new voices will be added to VoiceOver? We blind people never, ever thought Eloquence, an old TTS engine from 20 years ago now, would ever come to iOS. And yet, here it is in iOS 17. I wouldn't be surprised to see DecTalk, or more Siri voices. More Braille features is amazing, and they even mentioned the Mac! VoiceOver for Mac is notoriously never given as much love as VoiceOver for iOS is, so most blind people still use Windows, even though they have iPhones. I was expecting to see much better image descriptions, but they've already announced a ton of new stuff for plenty other disabilities. Having haptic music will be awesome even for me, adding another sense to the music. There are just so many new accessibility stuff, and I can't wait to see what all is really new in VoiceOver, since there's always new things not talked about in WWDC or release notes. I'm hoping that, one day, we get a tutorial for VoiceOver, like TalkBack on Android has, since there are so many commands, gestures, and settings that a new user never learns unless they learn to learn about them.
- bombcar 2y ago
  The image description stuff is already surprisingly good - I noticed when I got a photo text while driving and it described it well enough for me to know what it was.
  - skunkworker 2y ago
    Same, a family member send a photo while I was driving and over Carplay it was described fairly accurately.
    - slau 2y ago
      It’s sometimes awesome, and often extremely basic. “Contact sent you a picture of a group of people at an airport”. Amazing. “Contact sent you a screenshot of a social media post”. Useless. We know iOS can select text in pictures, so Siri can clearly read it. It knows it’s SoMe, so why not give me the headline?
      - MBCook 2y ago
stacktrust 2y ago
iOS 17 Image Descriptions are quite good, but audio descriptions don't seem to work on non-Pro devices, even though text descriptions are being shown on the screen and the audio menu is present and activated. Is that a bug? Even on Pro devices, audio image descriptions stop working after a few cycles of switching between Image Magnifier and apps/home. This can be fixed by restarting the app and disabling/enabling audio image descriptions, but that breaks the use case when an iPhone is dedicated to running only Image Magnifier + audio descriptions, via remote MDM with no way for the local blind user to restart the app. On-device iOS image descriptions could be improved if the user could help train the local image recognition by annotating photos or videos with text descriptions. For a blind person, this would enable locally-specific audio descriptions like "bedroom door", "kitchen fridge" or specific food dishes. Are there other iOS or Android AR apps which offer audio descriptions of live video from the camera?
s3p 2y ago
I wonder if this announcement had anything to do with the bombshells OpenAI and Google dropped this week. Couldn’t this have been part of WWDC next month?
- terramex 2y ago
  Tomorrow (today in some timezones) is Global Accesibility Awareness Day: https://en.m.wikipedia.org/wiki/Global_Accessibility_Awareness_Day https://en.m.wikipedia.org/wiki/Global_Accessibility_Awarene...
- Tagbert 2y ago
  As Terramex pointed out this is tied to a particular, relevant event. It’s also pretty common for Apple to preannounce some smaller features that are too specialized to be featured in the WWDC announcements. This gives them some attention when they would be lost and buried in WWDC footnotes. It is also probably an indication that WWDC will be full of new features and only the most impactful will be part of the keynote.
- kergonath 2y ago
  They do it every year at the same time. Also, it’s a small announcement, not a keynote or the kind of fanfare we have at WWDC or the September events. This does not seem calibrated to be effective in an advertising war with another company. All this to say, probably not.
- joshstrange 2y ago
  I think it's more of "clearing the decks" for stuff that didn't make the cut for WWDC. I assume WWDC is going to be all about AI and they couldn't find a good spot to put this announcement. "Clearing the decks" isn't a very kind way to refer to this accessibility tech since Apple has always been better than almost everyone else when it comes to accessibility. I don't see this as "we don't care, just announce it early" as much as "we can't fit this in so let's announce it early".
  - dhosek 2y ago
    As noted elsewhere, Apple always does their accessibility announcements in advance of WWDC.
xnx 2y ago
I love accessibility features because they might be the last features developed solely with the benefit of the user in mind. So many other app/os features are designed to steal your attention or gradually nerf usefulness.
- stacktrust 2y ago
  > developed solely with the benefit of the user in mind Hopefully accessibility features are never artificially segmented to higher priced devices.
  - corps_and_code 2y ago
    I wonder if that would be legal, at least in the US. That feels like it'd be a violation of the ADA?
    - diebeforei485 2y ago
      It would only be a violation if it's purely software locked. If it requires a chip that supports specific operations, and entry tier devices have an older chip, that wouldn't be a violation.
  - Loughla 2y ago
    At least in the US, they kind of can't be. The disability community is pretty up front about lawsuits.
    - stacktrust 2y ago
      iOS 17 audio image descriptions for blind people via Image Magnifier should work on all iPhones, but do not work on iPhone SE3 and iPhone 11 Pro. Audio image descriptions do work in iPhone 12 Pro. Lidar in 12 Pro increases accuracy, but should not be mandatory. Hopefully this is a bug that can be fixed, since text descriptions were still functional on the lower-end devices. Source: purchased devices until finding one that worked, since Apple docs indicated the feature should work on all iPhones that can run iOS 17. Edit: audio descriptions in Magnifier are non-functional on iPad Air, working on M2 iPad Pro.
badbart14 2y ago
I definitely get a good amount of motion sickness when using my phone while in a car so I'm super interested about the motion sickness cues and if they'll work. The dots look like they may get in the way a bit but I'm willing to take that tradeoff. My current car motion sickness mitigation system is these glasses that have liquid in them that supposedly help your ears feel the motion of the car better (and make you look like Harry Potter)
Shank 2y ago
> Vehicle Motion Cues is a new experience for iPhone and iPad that can help reduce motion sickness for passengers in moving vehicles. This excites me so, so much! I can't really use my phone as a passenger in a car without getting motion sick after 1-2 minutes. This seems like it might be a promising thing to try.
- droopyEyelids 2y ago
  Have you noticed any correlation between how hungry you are and how fast motion sickness kicks in?
  - Shank 2y ago
    It’s really interesting you say this. Is this a known correlation? I feel like now that you mention it, it’s incredibly fast if I’m hungry.
    - toast0 2y ago
      I went on a cruise, and had significant (for me) motion sickness that only got better once I ate --- of course, I was avoiding eating because I didn't feel well, so that seems like the wrong choice.
    - KolmogorovComp 2y ago
      It is a known correlation.
  - astrange 2y ago
    I'm not sure why, but I feel like I only get motion sickness in the back of Priuses. It must be something about their braking curve. I don't sit in enough EVs to tell if they're the same.
    - 121789 2y ago
      Teslas are especially bad for me. I think it’s the rough suspension and fast acceleration/deceleration
tootie 2y ago
Quibble but this isn't "eye tracking" it's "gaze tracking". Eye tracking is detecting where your eyes are. Gaze tracking is what you're looking at.
- crancher 2y ago
  Well then it's both?
  - tootie 2y ago
    Could be, but not necessarily. Eye tracking usually means it's tracking the eyes of one or more people in a video. Gaze tracking usually requires your eyes stay pretty steady and close to the tracker.
- astrange 2y ago
  This is how I feel about "face recognition" (it should mean recognizing whether something is a face or not), but it is common to use eye tracking this way.
- bogtog 2y ago
  Where are you getting this language point from? If I look up any company selling "eye trackers", their products are all meant to track where you're looking, e.g., https://www.tobii.com/ https://www.tobii.com/
  - tootie 2y ago
    Interesting. I picked it up working with Tobii devices a few years ago actually. I guess they updated their vocabulary.
roughly 2y ago
Accessibility settings are really a gold mine on iOS for device customization (yes, I agree, they shouldn’t be limited to accessibility). I’m particularly interested in the motion cues and the color filters for CarPlay - I have color filters set up to enable every night as kind of a Turbo-night shift mode (deep orange-red color shift), would love to do the same for CarPlay. I also completely forgot iOS had a magnifier built in!
- ryandrake 2y ago
  Accessibility features tend to be superpowers though, and I'm glad Apple gates them behind permissions and opt-ins. We all know of applications who try to trick the user into granting them inappropriate access to the device through the Accessibility APIs. I think DropBox still begs you to grant them Accessibility access so its tendrils can do who-knows-what to your system. With great power comes great responsibility.
  - roughly 2y ago
    It varies. Things like keyboard control or that kind of thing, absolutely, but mostly I've used it for stuff like "don't make an animated transition every time I change pages like an overcaffienated George Lucas" or "actually make night shift shift enough to be useful at night". I also use the background sounds to augment noise cancellation while taking a nap. All of those are just useful things or personal settings, not necessarily attack vectors.
  - burntwater 2y ago
    Guaranteed that marketers are salivating at the idea of eye tracking on apps and website. It's an amazing feature that absolutely needs to be gatekept.
    - klausa 2y ago
      I wonder if it'll use the same architecture as visionOS; where the vision tracking events and UI affordances are processed and composited out-of-process; with the app never seeing them.
emehrkay 2y ago
Eye tracking coupled with the show grid feature would seem like using a computer the way that people do in movies https://www.youtube.com/watch?v=UxigSW9MbY8 https://www.youtube.com/watch?v=UxigSW9MbY8
- tngranados 2y ago
  It's basically how the Apple Vision Pro mainly works.
deleted 2y ago
[deleted]
myth_drannon 2y ago
Eye Gaze devices(tablet with camera + software) cost around $20K, even if it offers 1/4 of the features this is good news for those who can't afford it.
- asadotzler 2y ago
  Don't be ridiculous. Solid hardware and software combos for windows cost a small fraction of that. The convenient and decent Tobii PCEye costs like $1,250 and a very nice TMS5 mini is under $2,000. Your bullshit was off by at least an order of magnitude.
  - boneghost 2y ago
    Let's be fair and compare similar products. Do you have any examples of $2000 mobile devices that support eye tracking on the OS level? The products you mention look like they're extra hardware you strap to a Windows PC. Certainly useful, but not quite as potentially life-changing as having that built into your phone.
fitsumbelay 2y ago
my first thought regarding eye-tracking: "whose accessibility to what|whom?"
- devmor 2y ago
  Do you have another idea beyond the accessibility of Users without fine motor control to the phone's screen?
  - smegsicle 2y ago
    which ads you are looking at
devmor 2y ago
I am excited for Vocal Cues - my main frustration with Siri is how poorly it comprehends even explicit instructions. One thing I wish Apple would implement is some kind of gesture control. The camera can detect fine details of face movement, it would be nice if that were leveraged to track hands that aren't touching the screen. For an example, if I have my iPhone or iPad on the desk in front of me, and a push notification that I don't need obstructs the content on the screen, I would love to be able to swipe my hand up towards the phone to dismiss it.
dakial1 2y ago
Eye tracking is not an accessibility feature, it is an advertising optimization feature disguised as an accessibility feature.
- devmor 2y ago
  Eye tracking is absolutely an accessibility feature. Just because you don't need it, and it can be abused, does not mean it isn't an absolutely game changing feature for some people.
  - nofunsir 2y ago
    He didn't say it wasn't an accessibility feature, just that it was disguised as one. Just because it's not a game changing feature for some people, doesn't mean its primary function isn't advertising.
    - devmor 2y ago
      Something being disguised as something explicitly implies it is not that thing.
      - nofunsir 2y ago
        It absolutely does not. Easiest example in the world. Was the trojan horse two things or was it one thing?
    - yreg 2y ago
      The user did say verbatim “Eye tracking is not an accessibility feature”
- jcotton42 2y ago
  My presumption is that apps will not be able to access this data, at least without some sort of permission gate.
  - 2y ago
rado 2y ago
They keep announcing these bombastic accessibility features while simple things like tabbing remain frustratingly broken. The macOS “allow this app to access that” dialog supports shift+tab, but not tab.
- Jtsummers 2y ago
  https://support.apple.com/guide/mac-help/use-your-keyboard-like-a-mouse-mchlp1399/mac https://support.apple.com/guide/mac-help/use-your-keyboard-l... - Keyboard navigation https://support.apple.com/guide/mac-help/navigate-your-mac-using-full-keyboard-access-mchlc06d1059/mac https://support.apple.com/guide/mac-help/navigate-your-mac-u... - Full Keyboard Access (accessibility feature, goes beyond just tabbing between elements) It's annoying that tabbing between UI elements is off by default on macOS. It's one of the first things I turn on with a new mac.
- deleted 2y ago
  [deleted]
Ocha 2y ago
Music haptics can be a cool way to teach someone how to dance and “feel the beat”
- NeuroCoder 2y ago
  There's a lot of interesting things we can do with haptics since they're relatively cheap to put in stuff. Hopefully accessibility gets the software and applications further along soon
- ErigmolCt 2y ago
  Using haptics in music to enhance rhythm perception and dance skills. Sounds really cool!
- burntwater 2y ago
  I'm severely hearing impaired and enjoy going to dance classes - swing, salsa, etc. If I'm standing still, I can easily tune into the beat. But once I start moving, I quickly lose it on many songs; dance studios aren't known for having large sound systems with substantial bass. I don't know that this specific setup would fix anything -- it would need some way of syncing to the instructor's iPhone that is connected via bluetooth to the studio's little portable speaker. But it's a step in the right direction. While on the topic, I can hear music but almost never understand lyrics; at best I might catch the key chorus phrase (example: the words "born in the USA" are literally the only words I understand in that song). A few months ago I discovered the "karaoke" feature on Apple Music, in which it displays the lyrics in time with the music. This have been game changing for me. I'm catching up on decades worth of music where I never had any idea what the lyrics are (filthy, that's what they are. So many songs about sex!). It has made exercising on the treadmill, elliptical, etc actually enjoyable.
  - pests 2y ago
    > A few months ago I discovered the "karaoke" feature on Apple Music, in which it displays the lyrics in time with the music. JSYK Spotify has had this for years too, its just under the "View Lyrics" button but it does highlight the sentence/word karaoke style. It used to be a "spotify app" back in that genre of the service.
simonw 2y ago
macOS has had a version of eye tracking for a while, it's really fun to try out. System preferences -> Accessibility -> Pointer Control Then turn on the "Head pointer" option.
- kridsdale1 2y ago
  VisionOS has this too. I mapped it to a triple click of the button for when eye tracking becomes inaccurate.
- RaoulP 2y ago
  Cool! That works surprisingly well. But how do you click while using this? Clicking on the trackpad doesn't work, when it's tracking my head.
  - saithir 2y ago
    That's a separate option. The option above head tracking in those settings allows for adding your facial expressions (smiling, blinking etc) as shortcuts for clicks.
- adriancooney 2y ago
  Cool. I'm a bit unsettled that my camera's green dot didn't turn on for it though.
alphabetting 2y ago
google yesterday also open sourced their accessibility feature for android and windows that controls cursor using head movements and facial gestures https://github.com/google/project-gameface https://github.com/google/project-gameface
mdm_ 2y ago
My first thought upon seeing the Haptic Music feature is to wonder how long until they make compatible headphones and I can relive my high school years, walking around listening to nu-metal and hip-hop with a Panasonic Shockwave walkman.
- dhosek 2y ago
  Surely, this would be a headphone feature possible without having to be supported by the player.
GiorgioG 2y ago
My wife is a hospice nurse and from time to time she'll have a patient without any ability to communicate except their eyes (think ALS) - for these folks in their final days/weeks of life this will be a godsend. There are specialized eye-tracking devices, but they're expensive and good luck getting them approved by insurance in time for the folks in need near the end of their lives.
EasyMark 2y ago
how can I make sure it's off? Is it off by default?
- yreg 2y ago
  Yes
- hyping9 2y ago
  [flagged]
tallytarik 2y ago
> “We believe deeply in the transformative power of innovation to enrich lives,” said Tim Cook, Apple’s CEO. Does this line actually mean anything? Press releases are so weird.
- astrange 2y ago
  I always wonder why press releases include claims that a person said a paragraph or more of text that noone would ever say out loud, but this one actually does sound like something he'd say. In a keynote video, anyway.
- sussmannbaka 2y ago
  Sure it means something, it’s pretty low on the PR talk scale even. Let’s see: > We believe deeply in "We", in this case, refers to Apple and as a company can’t believe in anything, it specifically refers to their employees. > the transformative power of innovation A new feature will bring forth change > to enrich lives In order to make someone’s life better. So far so good, at least no synergies being leveraged for 10x buzzword transformation. Let’s see if it passes the test: There’s a bunch of new accessibility features, that certainly fits the bill for innovation. As anecdata, at least one of these will drastically change how I interact with my devices, so there’s the transformative power. I will be able to use these new accessibility features to improve both the way I work and how I privately use technology, so one could argue my life is being positively enriched by this transformative power brought about by innovation. Do they "believe deeply" in this? That is only for Tim Cook and his employees to know, but they’re one of the few companies pushing advances in accessibility past the finish line, so they might!
Envirogenomic 2y ago
[dead]
cush 2y ago
All these features look amazing! That car motion sickness feature especially. Can’t wait to try it!
- petre 2y ago
  At least I put off the phone while I was in the car. Not the case now. Thank you Apple but I'd rather be sick while looking at your phone in a car.
hprotagonist 2y ago
Very curious to see how well eye tracking works behind a motorcycle visor. Thick leather gloves, bike noise, and a touch screen and audio interface are not much fun.
yreg 2y ago
Accessibility is for everyone, including you, if you live long enough. And the alternative is worse. So your choice is death or you are going to use accessibility features. – Siracusa
- jiggawatts 2y ago
  I aimed for the upvote button but they’re so tiny that my fat finger hit the downvote button by accident and then I had to retry the action. This is what people mean by accessibility is for everyone all of the time.
  - pseudosavant 2y ago
    I have a tremor and I run into this issue on HN all the time. I need to zoom in a lot to be sure I'll hit it.
    - avtar 2y ago
      Try https://www.modernhn.com https://www.modernhn.com if you haven't already. UI elements have more spacing around them, especially if zoomed in.
      - case 2y ago
        Hadn’t heard of this before, it looks great! Need it on mobile though, and would be happy to pay a reasonable fee for it.
        llm_trw 2y ago
        You can install the addon on mobile firefox. Ironically because of addons I now use firefox exclusively on mobile.
    - datahack 2y ago
      I don’t (yet) have accessibility challenges beyond glasses but hitting the tiny arrows is incredibly difficult. How come HN doesn’t update to be more accessible? It’s been a long time… I’m surprised it hasn’t been talked about by the team there.
ChrisMarshallNY 2y ago
I'm a believer in accessibility features. The difficulty is often in testing. I use SimDaltonism, to test for color-blindness accessibility, and, in the last app I wrote, I added a "long press help" feature, that responds to long-presses on items, by opening a popover, containing the label and hint. Makes testing much easier, and doubles as user help.
gpm 2y ago
I wonder if Vision Pro has enough microphones to do the acoustic camera thing? If so you could plausibly get "speech bubbles over peoples heads", accurately identifying who said what. I imagine that could be pretty awesome for deaf people.
- stubish 2y ago
  Deaf people I know certainly wouldn't want to be wearing a headset, say, in a restaurant. But a phone app or tablet that can do live text-to-speach would be good enough in many cases if it can separate voices into streams. Anything like this available?
  - gpm 2y ago
    I don't want to say it's certainly not available, but I doubt it. Maybe someone has done something trying to recognize different voices by differences in pitch/volume/... The vision pro has six mics, which likely enables it to locate the sources of sound in space by the amount of time it takes a sound to reach each mic (i.e. acting as an "acoustic camera"). Tablets and phones aren't going to have that hardware unfortunately. And yeah, obviously wearing a headset is a pretty big compromise that's not always (or maybe even often) going to be worth it. This was more of a though of "the hardware can probably do this cool thing for people already using it" than "people should use the hardware so it can do the cool thing".
bcx 2y ago
This is a good time to remind everyone that tomorrow, May 16th is Global Accessibility Awareness Day (GAAD) (https://accessibility.day https://accessibility.day), and that there are over 176 events worldwide going on to celebrate the process we are all making at improving accessibility in our products -- any plenty of learning opportunities for beginners and experts.
C-Loftus 2y ago
This is awesome and I love the UX, although I can't help but feel a bit sad that we need to always rely upon Apple and Microsoft for consumer accessibility. It would be so great if more resources could be allocated for such things within the Linux ecosystem.
toasted-subs 2y ago
Gotta make surf they got all the eye balls then make sure they don't get to date anybody and become homeless.
chaostheory 2y ago
Nice to see features from Vision Pro make it onto other Apple products
anyfoo 2y ago
I have better than 20/20 vision (yes, really) and now mobility problems, but there are some macOS accessibility features that I love. One is Zoom: I hold down two modifier keys and scroll, and can instantly zoom in to any part of the screen. It is extremely smooth, the more you scroll the higher you zoom, instantly and with high frame rate. Great when you want to show someone something, or just zoom into a detail of something if the app doesn’t support it or is two cumbersome. Or “pseudo-fullscreen” something. The other one is three finger drag on the trackpad. Why that isn’t default is beyond me. It means that if you use three fingers, any dragging on the trackpad behaves as if you held the button down. It’s so convenient.
- eviks 2y ago
  It's better to have an easy way to hold a mouse button via keyboard with your left hand and continue to use one finger on the touchpad rather than do the whole three fingers to drag
  - anyfoo 2y ago
    Not for me, no. But sounds like something accessibility options may also provide, and I can see how it may be better for a lot of people.
    - eviks 2y ago
      Which one have you tried?
      - anyfoo 2y ago
        Honestly? Only three finger drag. But just earlier I was holding coffee with the left hand, while using the trackpad to the right hand, and was glad I had three finger draw. I like doing "basic" mouse operations with only one hand (including right click: two fingers), and three finger drag works extremely well for me. I barely even think about it, it's automatic and an easy gesture for me. In turn, I am interested why you do think that your way is (apparently universally) better?
xyst 2y ago
eye tracking feature is interesting … for ad-tech companies
syngrog66 2y ago
while we're on topic of accessibility I'd like to point out the following lovely facts: * on Android, in the builtin popup menu for copypasta, the COPY button is wedged in tightly (by mere millimeters) between the CUT and PASTE buttons on either side of it. A harmless one packed in between two destructive, lossy ones, and usually irreversible. * the size of an adult human fingertip is no secret (and in fact 99.99% of humans have at least one!) * Google's staff consists of humans * Google supposedly hires ONLY "smart" people and their interview process involves some IQ-like tests * Android has had this obvious anti-pattern for many years now. perhaps 10+?
killjoywashere 2y ago
Imagine an eye-tracking loupe function on mobile: fit the same amount of text but bubble up the part under foveolar gaze. Save on readers* everywhere. * readers are those glasses you can pick up at the drug store for $17.99.
- skygazer 2y ago
  It took me a couple read-throughs to understand what you meant, but yes, on-screen gaze-sensitive magnification would be amazing, I agree.
tap-snap-or-nap 2y ago
This will have just enough capability limits and annoyances for you to be convinced to purchase a Vision Pro. It also makes eye tracking more widely accepted by making it available to everyone by using their existing devices.
Baeocystin 2y ago
One of the most important accessibility features they could bring back are physical home buttons on at least one ipad. I am completely serious. I work with a lot of older adults, many of whom have cognitive challenges, and the lack of a physical button to press as an escape hatch when they get confused is the #1 stumbling block by a mile.
- deleted 2y ago
  [deleted]
- lowkey 2y ago
  I agree. I have a last gen base iPad with Touch ID and a home button. I am pretty tech savvy but actually prefer this form factor.
  - foooorsyth 2y ago
    Touch ID is so drastically superior to Face ID in so many common “iPad scenarios”, eg. laying in bed with face partially obscured. I don’t understand Apple’s intense internal focus on FaceID only. FaceID with TouchID as an option when a device is flat on the table or when your face is obscured is so much nicer.
    - sleepingreset 2y ago
      FaceID builds profitable user habit loops. More real-estate on the screen to show things, easier to thoughtlessly purchase things if your password is a glance at the camera, etc. I don't think this is a user-focused decision, I believe it's a profit-focused one.
      - astrange 2y ago
        You can't buy things just by looking at the camera, there are forced button presses and the App Store tends to make you enter your password.
    - diebeforei485 2y ago
rsingla 2y ago
While it's a significant step forward for accessibility, it also invites us to consider how such technologies could integrate into everyday use for all users. This could enhance ease of use and efficiency, but it also requires careful consideration of privacy safeguards.
trilorez 2y ago
Haptics in the Music app will be great but it’s not exactly “new”, considering my haptic music app has been out for several years now: https://apps.apple.com/us/app/phazr/id1472113449 https://apps.apple.com/us/app/phazr/id1472113449
abdellah123 2y ago
[flagged]
ein0p 2y ago
If accurate enough it seems like eye tracking could be useful beyond accessibility to eg control iPads without dragging one’s greasy fingers all over the screen. iPad already supports pointers.
- superb_dev 2y ago
  I was thinking the same, it would make having an iPad under my monitor much less cumbersome
sandspar 2y ago
It's funny how the post is about enhanced surveillance technology and the text sentiment of the comment thread is overwhelmingly positive. Surveillance technology: >:( Surveillance technology with the word "accessibility" in the title: :)
- kalkr 2y ago
  of course it’s surveillance technology. an iphone constantly broadcasts your location to third parties, can deduce where you work and live, understands the unique shape of your face, has a built-in microphone and multiple kinds of cameras, stores all of your conversations, stores all of your photos, stores your health information, can figure out when you go to bed.. all on a completely closed and proprietary operating system. it’s like asking “why hasn’t anyone mentioned that we’re all using a website right now”
MagicMoonlight 2y ago
This is why Apple is the best. They’ll make features like this because it’s the right thing to do, even if it won’t really make any difference to 99.99% of their customers. They really earn consumer loyalty.
- cainxinth 2y ago
  This was the right thing to do, but I doubt “the right thing to do” was the primary motivator for it. This is smart marketing that helps position them as the more caring, inclusive, and capable tech brand. It also expands their reach more than a fraction of a percent. 13% of Americans have some form of disability.
  - bdzr 2y ago
    I feel like it also helps them get an edge in the healthcare sector, which has buckets of money in the mix.
  - GarnetFloride 2y ago
    75+% of Americans become disabled (mostly temporarily) in their lifetime. So it will affect most people someday. And everyone dies in the end.
  - igornadj 2y ago
    Tim Cook in response to a shareholder proposing scrapping accessibility to improve ROI: “When we work on making our devices accessible by the blind, I don’t consider the bloody ROI”.
    - cainxinth 2y ago
      That's just more "cause marketing." The first ever $3T company definitely considers the bloody ROI in every decision -- whether the CEO is willing to admit it publicly or not.
- s_dev 2y ago
  The accessibility features are also useful for automated UI testing. >They’ll make features like this because it’s the right thing to do While the positivity is great to see -- I'd temper that expectation they're simply doing the right thing and definitely acting in their perceived interest. People can and often do the right thing -- large companies rarely do.
jfoster 2y ago
Reminder: eye tracking means an always-on camera. I'm not fully against it, but it's good to keep in mind the implications.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
elsonrodriguez 2y ago
Didn’t Mark Rober say he worked on some motion sickness stuff at Apple?
1oooqooq 2y ago
remember how google added js apis to detect active tabs. and rendering intersection. complex webapps could use less battery and have richer interaction. everyone rejoiced. nobody implemented anything useful. soon, advertisers, google included, was selling ads packages by "viewability". lol. can't wait for safari to lead with "eyed" ads.
- sharkjacobs 2y ago
  I know this is a joke, but for for anyone who was wondering, > all data used to set up and control this feature is kept securely on device, and isn’t shared with Apple. It's not an API adopted which apps can use, it's a system pointer, like a mouse or trackpad
pmichaud 2y ago
I hope this works for people with one eye. There are dozens of us—dozens!
squeral 2y ago
Can't wait for my son to try this. He has general coordination issues due to a brainstem injury but the eyes are probably the part of the body he can better control. I'm not a fan of Apple's software and didn't have a great experience with the Vision Pro but I am excited to try it out.
brunorsini 2y ago
This is the beginning of the end for the mouse — not just on phones, but on desktops, everywhere. I highly recommend everyone to at least schedule a free Apple Vision demo at the local Apple Store.

500 points by dmd 2y ago

pcl 2y ago

Bummer the eye tracking is iOS only. I’ve been wanting focus-follows-eyes for decades.

npongratz 2y ago
I share the sentiment. I've long noted in situations with grids of terminal windows where focus-follows-eyes would be so much faster and present less friction than using hotkeys or focus-follows-mouse.

dagmx 2y ago

This is one major advantage to Apple sharing a foundation across all their devices. Vision Pro introduced eye tracking to their systems as a new input modality, and now it trickles down to their other platforms. I am surprised CarPlay didn’t have voice control before this though.

callwhendone 2y ago
It would be amazing if it gets carried over to the Mac.
simlevesque 2y ago
CarPlay devices aren't really powerful.
- andrewmunsell 2y ago
  CarPlay is rendered by the phone itself, so it's not strictly a function of how powerful the car infotainment is. You've been able to talk to Siri since the beginning of CarPlay so additional voice control is really just an accessibility thing
  - joezydeco 2y ago
    Some cars already have a voice control button on the wheel for their existing system which, if done correctly, is overriden by Siri+CarPlay. Which is really nice when it works.
    - dhosek 2y ago
      I’m able to control carplay by just saying “Hey Siri.” Siri’s abilities tend to fluctuate based on what Apple is doing on the server, the phase of the moon and whether I remembered to sacrifice a chicken that morning, but otherwise, it seems to work fine.
- Tagbert 2y ago
  CarPlay devices (car components) are essentially playing a streaming video of a hidden display generated by the phone. CarPlay also lets those devices send back touch events to trigger buttons and other interactions. Very little process is done on the vehicle. BTW if you are plugged in to CarPlay and take a screen shot, it will include the hidden CarPlay screen.
jsheard 2y ago
Is this really likely to be downstream of the Vision Pro implementation? I would think that eye-tracking with specialized hardware at a fixed position very close to the eye is very different to doing it at a distance with a general purpose front facing camera.
- iloveyouocean 2y ago
  The 'tracking eyes' part is different, but once you have eye position data, the 'how the eyes interact with the interface' could be very similar.
- dmicah 2y ago
  Typically eye-trackers work by illuminating the eyes with near infrared light and using infrared cameras. This creates a higher contrast image of the pupils, etc. I assume Apple is doing this in the Vision Pro. Eye-tracking can also be done with just visible light, though. Apple has the benefit of knowing where all the user interface elements are on screen, so eye-tracking in this on the iPhone or iPad doesn't need to be high precision. Knowledge of the position of the items can help to reduce the uncertainty of what is being fixated on.
  - neverokay 2y ago
    So there isn’t much more to it than getting a good resolution image of the eye from any distance, every millisecond. Precision is the issue because we are mostly moving our eyes in about 8 directions, there’s no precision because we don’t know how to measure focusing of our eye lens with a camera yet (unless that too is just a matter of getting a picture). Squinting would be the closest thing to physically expressing focusing. So the camera needs to know I’m looking left with my eye, followed by a squint to achieve precision. Seems stressful though. Gonna need AI just to do noise cancelling of involuntary things your eyes do like pupil dilation, blinking.
- dagmx 2y ago
  Not the hardware side, but the software side. Implementing all the various behaviours and security models for the eye tracking and baking it into SwiftUI, means that it translates over easier once they figure out the hardware aspect. But the iPads and iPhones with FaceID have had eye tracking capabilities for a while, just not useful in UI.

devinprater 2y ago

I wonder what new voices will be added to VoiceOver? We blind people never, ever thought Eloquence, an old TTS engine from 20 years ago now, would ever come to iOS. And yet, here it is in iOS 17. I wouldn't be surprised to see DecTalk, or more Siri voices. More Braille features is amazing, and they even mentioned the Mac! VoiceOver for Mac is notoriously never given as much love as VoiceOver for iOS is, so most blind people still use Windows, even though they have iPhones. I was expecting to see much better image descriptions, but they've already announced a ton of new stuff for plenty other disabilities. Having haptic music will be awesome even for me, adding another sense to the music. There are just so many new accessibility stuff, and I can't wait to see what all is really new in VoiceOver, since there's always new things not talked about in WWDC or release notes. I'm hoping that, one day, we get a tutorial for VoiceOver, like TalkBack on Android has, since there are so many commands, gestures, and settings that a new user never learns unless they learn to learn about them.

bombcar 2y ago
The image description stuff is already surprisingly good - I noticed when I got a photo text while driving and it described it well enough for me to know what it was.
- skunkworker 2y ago
  Same, a family member send a photo while I was driving and over Carplay it was described fairly accurately.
  - slau 2y ago
    It’s sometimes awesome, and often extremely basic. “Contact sent you a picture of a group of people at an airport”. Amazing. “Contact sent you a screenshot of a social media post”. Useless. We know iOS can select text in pictures, so Siri can clearly read it. It knows it’s SoMe, so why not give me the headline?
    - MBCook 2y ago

stacktrust 2y ago

iOS 17 Image Descriptions are quite good, but audio descriptions don't seem to work on non-Pro devices, even though text descriptions are being shown on the screen and the audio menu is present and activated. Is that a bug? Even on Pro devices, audio image descriptions stop working after a few cycles of switching between Image Magnifier and apps/home. This can be fixed by restarting the app and disabling/enabling audio image descriptions, but that breaks the use case when an iPhone is dedicated to running only Image Magnifier + audio descriptions, via remote MDM with no way for the local blind user to restart the app. On-device iOS image descriptions could be improved if the user could help train the local image recognition by annotating photos or videos with text descriptions. For a blind person, this would enable locally-specific audio descriptions like "bedroom door", "kitchen fridge" or specific food dishes. Are there other iOS or Android AR apps which offer audio descriptions of live video from the camera?

s3p 2y ago

I wonder if this announcement had anything to do with the bombshells OpenAI and Google dropped this week. Couldn’t this have been part of WWDC next month?

terramex 2y ago
Tomorrow (today in some timezones) is Global Accesibility Awareness Day: https://en.m.wikipedia.org/wiki/Global_Accessibility_Awareness_Day https://en.m.wikipedia.org/wiki/Global_Accessibility_Awarene...
Tagbert 2y ago
As Terramex pointed out this is tied to a particular, relevant event. It’s also pretty common for Apple to preannounce some smaller features that are too specialized to be featured in the WWDC announcements. This gives them some attention when they would be lost and buried in WWDC footnotes. It is also probably an indication that WWDC will be full of new features and only the most impactful will be part of the keynote.
kergonath 2y ago
They do it every year at the same time. Also, it’s a small announcement, not a keynote or the kind of fanfare we have at WWDC or the September events. This does not seem calibrated to be effective in an advertising war with another company. All this to say, probably not.
joshstrange 2y ago
I think it's more of "clearing the decks" for stuff that didn't make the cut for WWDC. I assume WWDC is going to be all about AI and they couldn't find a good spot to put this announcement. "Clearing the decks" isn't a very kind way to refer to this accessibility tech since Apple has always been better than almost everyone else when it comes to accessibility. I don't see this as "we don't care, just announce it early" as much as "we can't fit this in so let's announce it early".
- dhosek 2y ago
  As noted elsewhere, Apple always does their accessibility announcements in advance of WWDC.

xnx 2y ago

I love accessibility features because they might be the last features developed solely with the benefit of the user in mind. So many other app/os features are designed to steal your attention or gradually nerf usefulness.

stacktrust 2y ago
> developed solely with the benefit of the user in mind Hopefully accessibility features are never artificially segmented to higher priced devices.
- corps_and_code 2y ago
  I wonder if that would be legal, at least in the US. That feels like it'd be a violation of the ADA?
  - diebeforei485 2y ago
    It would only be a violation if it's purely software locked. If it requires a chip that supports specific operations, and entry tier devices have an older chip, that wouldn't be a violation.
- Loughla 2y ago
  At least in the US, they kind of can't be. The disability community is pretty up front about lawsuits.
  - stacktrust 2y ago
    iOS 17 audio image descriptions for blind people via Image Magnifier should work on all iPhones, but do not work on iPhone SE3 and iPhone 11 Pro. Audio image descriptions do work in iPhone 12 Pro. Lidar in 12 Pro increases accuracy, but should not be mandatory. Hopefully this is a bug that can be fixed, since text descriptions were still functional on the lower-end devices. Source: purchased devices until finding one that worked, since Apple docs indicated the feature should work on all iPhones that can run iOS 17. Edit: audio descriptions in Magnifier are non-functional on iPad Air, working on M2 iPad Pro.

badbart14 2y ago

I definitely get a good amount of motion sickness when using my phone while in a car so I'm super interested about the motion sickness cues and if they'll work. The dots look like they may get in the way a bit but I'm willing to take that tradeoff. My current car motion sickness mitigation system is these glasses that have liquid in them that supposedly help your ears feel the motion of the car better (and make you look like Harry Potter)

Shank 2y ago

> Vehicle Motion Cues is a new experience for iPhone and iPad that can help reduce motion sickness for passengers in moving vehicles. This excites me so, so much! I can't really use my phone as a passenger in a car without getting motion sick after 1-2 minutes. This seems like it might be a promising thing to try.

droopyEyelids 2y ago
Have you noticed any correlation between how hungry you are and how fast motion sickness kicks in?
- Shank 2y ago
  It’s really interesting you say this. Is this a known correlation? I feel like now that you mention it, it’s incredibly fast if I’m hungry.
  - toast0 2y ago
    I went on a cruise, and had significant (for me) motion sickness that only got better once I ate --- of course, I was avoiding eating because I didn't feel well, so that seems like the wrong choice.
  - KolmogorovComp 2y ago
    It is a known correlation.
- astrange 2y ago
  I'm not sure why, but I feel like I only get motion sickness in the back of Priuses. It must be something about their braking curve. I don't sit in enough EVs to tell if they're the same.
  - 121789 2y ago
    Teslas are especially bad for me. I think it’s the rough suspension and fast acceleration/deceleration

tootie 2y ago

Quibble but this isn't "eye tracking" it's "gaze tracking". Eye tracking is detecting where your eyes are. Gaze tracking is what you're looking at.

crancher 2y ago
Well then it's both?
- tootie 2y ago
  Could be, but not necessarily. Eye tracking usually means it's tracking the eyes of one or more people in a video. Gaze tracking usually requires your eyes stay pretty steady and close to the tracker.
astrange 2y ago
This is how I feel about "face recognition" (it should mean recognizing whether something is a face or not), but it is common to use eye tracking this way.
bogtog 2y ago
Where are you getting this language point from? If I look up any company selling "eye trackers", their products are all meant to track where you're looking, e.g., https://www.tobii.com/ https://www.tobii.com/
- tootie 2y ago
  Interesting. I picked it up working with Tobii devices a few years ago actually. I guess they updated their vocabulary.

roughly 2y ago

Accessibility settings are really a gold mine on iOS for device customization (yes, I agree, they shouldn’t be limited to accessibility). I’m particularly interested in the motion cues and the color filters for CarPlay - I have color filters set up to enable every night as kind of a Turbo-night shift mode (deep orange-red color shift), would love to do the same for CarPlay. I also completely forgot iOS had a magnifier built in!

ryandrake 2y ago
Accessibility features tend to be superpowers though, and I'm glad Apple gates them behind permissions and opt-ins. We all know of applications who try to trick the user into granting them inappropriate access to the device through the Accessibility APIs. I think DropBox still begs you to grant them Accessibility access so its tendrils can do who-knows-what to your system. With great power comes great responsibility.
- roughly 2y ago
  It varies. Things like keyboard control or that kind of thing, absolutely, but mostly I've used it for stuff like "don't make an animated transition every time I change pages like an overcaffienated George Lucas" or "actually make night shift shift enough to be useful at night". I also use the background sounds to augment noise cancellation while taking a nap. All of those are just useful things or personal settings, not necessarily attack vectors.
- burntwater 2y ago
  Guaranteed that marketers are salivating at the idea of eye tracking on apps and website. It's an amazing feature that absolutely needs to be gatekept.
  - klausa 2y ago
    I wonder if it'll use the same architecture as visionOS; where the vision tracking events and UI affordances are processed and composited out-of-process; with the app never seeing them.

emehrkay 2y ago

Eye tracking coupled with the show grid feature would seem like using a computer the way that people do in movies https://www.youtube.com/watch?v=UxigSW9MbY8 https://www.youtube.com/watch?v=UxigSW9MbY8

tngranados 2y ago
It's basically how the Apple Vision Pro mainly works.

deleted 2y ago

[deleted]

myth_drannon 2y ago

Eye Gaze devices(tablet with camera + software) cost around $20K, even if it offers 1/4 of the features this is good news for those who can't afford it.

asadotzler 2y ago
Don't be ridiculous. Solid hardware and software combos for windows cost a small fraction of that. The convenient and decent Tobii PCEye costs like $1,250 and a very nice TMS5 mini is under $2,000. Your bullshit was off by at least an order of magnitude.
- boneghost 2y ago
  Let's be fair and compare similar products. Do you have any examples of $2000 mobile devices that support eye tracking on the OS level? The products you mention look like they're extra hardware you strap to a Windows PC. Certainly useful, but not quite as potentially life-changing as having that built into your phone.

fitsumbelay 2y ago

my first thought regarding eye-tracking: "whose accessibility to what|whom?"

devmor 2y ago
Do you have another idea beyond the accessibility of Users without fine motor control to the phone's screen?
- smegsicle 2y ago
  which ads you are looking at

devmor 2y ago

I am excited for Vocal Cues - my main frustration with Siri is how poorly it comprehends even explicit instructions. One thing I wish Apple would implement is some kind of gesture control. The camera can detect fine details of face movement, it would be nice if that were leveraged to track hands that aren't touching the screen. For an example, if I have my iPhone or iPad on the desk in front of me, and a push notification that I don't need obstructs the content on the screen, I would love to be able to swipe my hand up towards the phone to dismiss it.

dakial1 2y ago

Eye tracking is not an accessibility feature, it is an advertising optimization feature disguised as an accessibility feature.

devmor 2y ago
Eye tracking is absolutely an accessibility feature. Just because you don't need it, and it can be abused, does not mean it isn't an absolutely game changing feature for some people.
- nofunsir 2y ago
  He didn't say it wasn't an accessibility feature, just that it was disguised as one. Just because it's not a game changing feature for some people, doesn't mean its primary function isn't advertising.
  - devmor 2y ago
    Something being disguised as something explicitly implies it is not that thing.
    - nofunsir 2y ago
      It absolutely does not. Easiest example in the world. Was the trojan horse two things or was it one thing?
  - yreg 2y ago
    The user did say verbatim “Eye tracking is not an accessibility feature”
jcotton42 2y ago
My presumption is that apps will not be able to access this data, at least without some sort of permission gate.
- 2y ago

rado 2y ago

They keep announcing these bombastic accessibility features while simple things like tabbing remain frustratingly broken. The macOS “allow this app to access that” dialog supports shift+tab, but not tab.

Jtsummers 2y ago
https://support.apple.com/guide/mac-help/use-your-keyboard-like-a-mouse-mchlp1399/mac https://support.apple.com/guide/mac-help/use-your-keyboard-l... - Keyboard navigation https://support.apple.com/guide/mac-help/navigate-your-mac-using-full-keyboard-access-mchlc06d1059/mac https://support.apple.com/guide/mac-help/navigate-your-mac-u... - Full Keyboard Access (accessibility feature, goes beyond just tabbing between elements) It's annoying that tabbing between UI elements is off by default on macOS. It's one of the first things I turn on with a new mac.
deleted 2y ago
[deleted]

Ocha 2y ago

Music haptics can be a cool way to teach someone how to dance and “feel the beat”

NeuroCoder 2y ago
There's a lot of interesting things we can do with haptics since they're relatively cheap to put in stuff. Hopefully accessibility gets the software and applications further along soon
ErigmolCt 2y ago
Using haptics in music to enhance rhythm perception and dance skills. Sounds really cool!
burntwater 2y ago
I'm severely hearing impaired and enjoy going to dance classes - swing, salsa, etc. If I'm standing still, I can easily tune into the beat. But once I start moving, I quickly lose it on many songs; dance studios aren't known for having large sound systems with substantial bass. I don't know that this specific setup would fix anything -- it would need some way of syncing to the instructor's iPhone that is connected via bluetooth to the studio's little portable speaker. But it's a step in the right direction. While on the topic, I can hear music but almost never understand lyrics; at best I might catch the key chorus phrase (example: the words "born in the USA" are literally the only words I understand in that song). A few months ago I discovered the "karaoke" feature on Apple Music, in which it displays the lyrics in time with the music. This have been game changing for me. I'm catching up on decades worth of music where I never had any idea what the lyrics are (filthy, that's what they are. So many songs about sex!). It has made exercising on the treadmill, elliptical, etc actually enjoyable.
- pests 2y ago
  > A few months ago I discovered the "karaoke" feature on Apple Music, in which it displays the lyrics in time with the music. JSYK Spotify has had this for years too, its just under the "View Lyrics" button but it does highlight the sentence/word karaoke style. It used to be a "spotify app" back in that genre of the service.

simonw 2y ago

macOS has had a version of eye tracking for a while, it's really fun to try out. System preferences -> Accessibility -> Pointer Control Then turn on the "Head pointer" option.

kridsdale1 2y ago
VisionOS has this too. I mapped it to a triple click of the button for when eye tracking becomes inaccurate.
RaoulP 2y ago
Cool! That works surprisingly well. But how do you click while using this? Clicking on the trackpad doesn't work, when it's tracking my head.
- saithir 2y ago
  That's a separate option. The option above head tracking in those settings allows for adding your facial expressions (smiling, blinking etc) as shortcuts for clicks.
adriancooney 2y ago
Cool. I'm a bit unsettled that my camera's green dot didn't turn on for it though.

alphabetting 2y ago

google yesterday also open sourced their accessibility feature for android and windows that controls cursor using head movements and facial gestures https://github.com/google/project-gameface https://github.com/google/project-gameface

mdm_ 2y ago

My first thought upon seeing the Haptic Music feature is to wonder how long until they make compatible headphones and I can relive my high school years, walking around listening to nu-metal and hip-hop with a Panasonic Shockwave walkman.

dhosek 2y ago
Surely, this would be a headphone feature possible without having to be supported by the player.

GiorgioG 2y ago

My wife is a hospice nurse and from time to time she'll have a patient without any ability to communicate except their eyes (think ALS) - for these folks in their final days/weeks of life this will be a godsend. There are specialized eye-tracking devices, but they're expensive and good luck getting them approved by insurance in time for the folks in need near the end of their lives.

EasyMark 2y ago

how can I make sure it's off? Is it off by default?

yreg 2y ago
Yes
hyping9 2y ago
[flagged]

tallytarik 2y ago

> “We believe deeply in the transformative power of innovation to enrich lives,” said Tim Cook, Apple’s CEO. Does this line actually mean anything? Press releases are so weird.

astrange 2y ago
I always wonder why press releases include claims that a person said a paragraph or more of text that noone would ever say out loud, but this one actually does sound like something he'd say. In a keynote video, anyway.
sussmannbaka 2y ago
Sure it means something, it’s pretty low on the PR talk scale even. Let’s see: > We believe deeply in "We", in this case, refers to Apple and as a company can’t believe in anything, it specifically refers to their employees. > the transformative power of innovation A new feature will bring forth change > to enrich lives In order to make someone’s life better. So far so good, at least no synergies being leveraged for 10x buzzword transformation. Let’s see if it passes the test: There’s a bunch of new accessibility features, that certainly fits the bill for innovation. As anecdata, at least one of these will drastically change how I interact with my devices, so there’s the transformative power. I will be able to use these new accessibility features to improve both the way I work and how I privately use technology, so one could argue my life is being positively enriched by this transformative power brought about by innovation. Do they "believe deeply" in this? That is only for Tim Cook and his employees to know, but they’re one of the few companies pushing advances in accessibility past the finish line, so they might!

Envirogenomic 2y ago

[dead]

cush 2y ago

All these features look amazing! That car motion sickness feature especially. Can’t wait to try it!

petre 2y ago
At least I put off the phone while I was in the car. Not the case now. Thank you Apple but I'd rather be sick while looking at your phone in a car.

hprotagonist 2y ago

Very curious to see how well eye tracking works behind a motorcycle visor. Thick leather gloves, bike noise, and a touch screen and audio interface are not much fun.

yreg 2y ago

Accessibility is for everyone, including you, if you live long enough. And the alternative is worse. So your choice is death or you are going to use accessibility features. – Siracusa

jiggawatts 2y ago
I aimed for the upvote button but they’re so tiny that my fat finger hit the downvote button by accident and then I had to retry the action. This is what people mean by accessibility is for everyone all of the time.
- pseudosavant 2y ago
  I have a tremor and I run into this issue on HN all the time. I need to zoom in a lot to be sure I'll hit it.
  - avtar 2y ago
    Try https://www.modernhn.com https://www.modernhn.com if you haven't already. UI elements have more spacing around them, especially if zoomed in.
    - case 2y ago
      Hadn’t heard of this before, it looks great! Need it on mobile though, and would be happy to pay a reasonable fee for it.
      - llm_trw 2y ago
        You can install the addon on mobile firefox. Ironically because of addons I now use firefox exclusively on mobile.
  - datahack 2y ago
    I don’t (yet) have accessibility challenges beyond glasses but hitting the tiny arrows is incredibly difficult. How come HN doesn’t update to be more accessible? It’s been a long time… I’m surprised it hasn’t been talked about by the team there.

ChrisMarshallNY 2y ago

I'm a believer in accessibility features. The difficulty is often in testing. I use SimDaltonism, to test for color-blindness accessibility, and, in the last app I wrote, I added a "long press help" feature, that responds to long-presses on items, by opening a popover, containing the label and hint. Makes testing much easier, and doubles as user help.

gpm 2y ago

I wonder if Vision Pro has enough microphones to do the acoustic camera thing? If so you could plausibly get "speech bubbles over peoples heads", accurately identifying who said what. I imagine that could be pretty awesome for deaf people.

stubish 2y ago
Deaf people I know certainly wouldn't want to be wearing a headset, say, in a restaurant. But a phone app or tablet that can do live text-to-speach would be good enough in many cases if it can separate voices into streams. Anything like this available?
- gpm 2y ago
  I don't want to say it's certainly not available, but I doubt it. Maybe someone has done something trying to recognize different voices by differences in pitch/volume/... The vision pro has six mics, which likely enables it to locate the sources of sound in space by the amount of time it takes a sound to reach each mic (i.e. acting as an "acoustic camera"). Tablets and phones aren't going to have that hardware unfortunately. And yeah, obviously wearing a headset is a pretty big compromise that's not always (or maybe even often) going to be worth it. This was more of a though of "the hardware can probably do this cool thing for people already using it" than "people should use the hardware so it can do the cool thing".

bcx 2y ago

This is a good time to remind everyone that tomorrow, May 16th is Global Accessibility Awareness Day (GAAD) (https://accessibility.day https://accessibility.day), and that there are over 176 events worldwide going on to celebrate the process we are all making at improving accessibility in our products -- any plenty of learning opportunities for beginners and experts.

C-Loftus 2y ago

This is awesome and I love the UX, although I can't help but feel a bit sad that we need to always rely upon Apple and Microsoft for consumer accessibility. It would be so great if more resources could be allocated for such things within the Linux ecosystem.

toasted-subs 2y ago

Gotta make surf they got all the eye balls then make sure they don't get to date anybody and become homeless.

chaostheory 2y ago

Nice to see features from Vision Pro make it onto other Apple products

anyfoo 2y ago

I have better than 20/20 vision (yes, really) and now mobility problems, but there are some macOS accessibility features that I love. One is Zoom: I hold down two modifier keys and scroll, and can instantly zoom in to any part of the screen. It is extremely smooth, the more you scroll the higher you zoom, instantly and with high frame rate. Great when you want to show someone something, or just zoom into a detail of something if the app doesn’t support it or is two cumbersome. Or “pseudo-fullscreen” something. The other one is three finger drag on the trackpad. Why that isn’t default is beyond me. It means that if you use three fingers, any dragging on the trackpad behaves as if you held the button down. It’s so convenient.

eviks 2y ago
It's better to have an easy way to hold a mouse button via keyboard with your left hand and continue to use one finger on the touchpad rather than do the whole three fingers to drag
- anyfoo 2y ago
  Not for me, no. But sounds like something accessibility options may also provide, and I can see how it may be better for a lot of people.
  - eviks 2y ago
    Which one have you tried?
    - anyfoo 2y ago
      Honestly? Only three finger drag. But just earlier I was holding coffee with the left hand, while using the trackpad to the right hand, and was glad I had three finger draw. I like doing "basic" mouse operations with only one hand (including right click: two fingers), and three finger drag works extremely well for me. I barely even think about it, it's automatic and an easy gesture for me. In turn, I am interested why you do think that your way is (apparently universally) better?

xyst 2y ago

eye tracking feature is interesting … for ad-tech companies

syngrog66 2y ago

while we're on topic of accessibility I'd like to point out the following lovely facts: * on Android, in the builtin popup menu for copypasta, the COPY button is wedged in tightly (by mere millimeters) between the CUT and PASTE buttons on either side of it. A harmless one packed in between two destructive, lossy ones, and usually irreversible. * the size of an adult human fingertip is no secret (and in fact 99.99% of humans have at least one!) * Google's staff consists of humans * Google supposedly hires ONLY "smart" people and their interview process involves some IQ-like tests * Android has had this obvious anti-pattern for many years now. perhaps 10+?

killjoywashere 2y ago

Imagine an eye-tracking loupe function on mobile: fit the same amount of text but bubble up the part under foveolar gaze. Save on readers* everywhere. * readers are those glasses you can pick up at the drug store for $17.99.

skygazer 2y ago
It took me a couple read-throughs to understand what you meant, but yes, on-screen gaze-sensitive magnification would be amazing, I agree.

tap-snap-or-nap 2y ago

This will have just enough capability limits and annoyances for you to be convinced to purchase a Vision Pro. It also makes eye tracking more widely accepted by making it available to everyone by using their existing devices.

Baeocystin 2y ago

One of the most important accessibility features they could bring back are physical home buttons on at least one ipad. I am completely serious. I work with a lot of older adults, many of whom have cognitive challenges, and the lack of a physical button to press as an escape hatch when they get confused is the #1 stumbling block by a mile.

deleted 2y ago
[deleted]
lowkey 2y ago
I agree. I have a last gen base iPad with Touch ID and a home button. I am pretty tech savvy but actually prefer this form factor.
- foooorsyth 2y ago
  Touch ID is so drastically superior to Face ID in so many common “iPad scenarios”, eg. laying in bed with face partially obscured. I don’t understand Apple’s intense internal focus on FaceID only. FaceID with TouchID as an option when a device is flat on the table or when your face is obscured is so much nicer.
  - sleepingreset 2y ago
    FaceID builds profitable user habit loops. More real-estate on the screen to show things, easier to thoughtlessly purchase things if your password is a glance at the camera, etc. I don't think this is a user-focused decision, I believe it's a profit-focused one.
    - astrange 2y ago
      You can't buy things just by looking at the camera, there are forced button presses and the App Store tends to make you enter your password.
  - diebeforei485 2y ago

rsingla 2y ago

While it's a significant step forward for accessibility, it also invites us to consider how such technologies could integrate into everyday use for all users. This could enhance ease of use and efficiency, but it also requires careful consideration of privacy safeguards.

trilorez 2y ago

Haptics in the Music app will be great but it’s not exactly “new”, considering my haptic music app has been out for several years now: https://apps.apple.com/us/app/phazr/id1472113449 https://apps.apple.com/us/app/phazr/id1472113449

abdellah123 2y ago

[flagged]

ein0p 2y ago

If accurate enough it seems like eye tracking could be useful beyond accessibility to eg control iPads without dragging one’s greasy fingers all over the screen. iPad already supports pointers.

superb_dev 2y ago
I was thinking the same, it would make having an iPad under my monitor much less cumbersome

sandspar 2y ago

It's funny how the post is about enhanced surveillance technology and the text sentiment of the comment thread is overwhelmingly positive. Surveillance technology: >:( Surveillance technology with the word "accessibility" in the title: :)

kalkr 2y ago
of course it’s surveillance technology. an iphone constantly broadcasts your location to third parties, can deduce where you work and live, understands the unique shape of your face, has a built-in microphone and multiple kinds of cameras, stores all of your conversations, stores all of your photos, stores your health information, can figure out when you go to bed.. all on a completely closed and proprietary operating system. it’s like asking “why hasn’t anyone mentioned that we’re all using a website right now”

MagicMoonlight 2y ago

This is why Apple is the best. They’ll make features like this because it’s the right thing to do, even if it won’t really make any difference to 99.99% of their customers. They really earn consumer loyalty.

cainxinth 2y ago
This was the right thing to do, but I doubt “the right thing to do” was the primary motivator for it. This is smart marketing that helps position them as the more caring, inclusive, and capable tech brand. It also expands their reach more than a fraction of a percent. 13% of Americans have some form of disability.
- bdzr 2y ago
  I feel like it also helps them get an edge in the healthcare sector, which has buckets of money in the mix.
- GarnetFloride 2y ago
  75+% of Americans become disabled (mostly temporarily) in their lifetime. So it will affect most people someday. And everyone dies in the end.
- igornadj 2y ago
  Tim Cook in response to a shareholder proposing scrapping accessibility to improve ROI: “When we work on making our devices accessible by the blind, I don’t consider the bloody ROI”.
  - cainxinth 2y ago
    That's just more "cause marketing." The first ever $3T company definitely considers the bloody ROI in every decision -- whether the CEO is willing to admit it publicly or not.
s_dev 2y ago
The accessibility features are also useful for automated UI testing. >They’ll make features like this because it’s the right thing to do While the positivity is great to see -- I'd temper that expectation they're simply doing the right thing and definitely acting in their perceived interest. People can and often do the right thing -- large companies rarely do.

jfoster 2y ago

Reminder: eye tracking means an always-on camera. I'm not fully against it, but it's good to keep in mind the implications.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

elsonrodriguez 2y ago

Didn’t Mark Rober say he worked on some motion sickness stuff at Apple?

1oooqooq 2y ago

remember how google added js apis to detect active tabs. and rendering intersection. complex webapps could use less battery and have richer interaction. everyone rejoiced. nobody implemented anything useful. soon, advertisers, google included, was selling ads packages by "viewability". lol. can't wait for safari to lead with "eyed" ads.

sharkjacobs 2y ago
I know this is a joke, but for for anyone who was wondering, > all data used to set up and control this feature is kept securely on device, and isn’t shared with Apple. It's not an API adopted which apps can use, it's a system pointer, like a mouse or trackpad

pmichaud 2y ago

I hope this works for people with one eye. There are dozens of us—dozens!

squeral 2y ago

Can't wait for my son to try this. He has general coordination issues due to a brainstem injury but the eyes are probably the part of the body he can better control. I'm not a fan of Apple's software and didn't have a great experience with the Vision Pro but I am excited to try it out.

brunorsini 2y ago

This is the beginning of the end for the mouse — not just on phones, but on desktops, everywhere. I highly recommend everyone to at least schedule a free Apple Vision demo at the local Apple Store.