3 ms·

Honest question out of curiosity- Are there possibly AI-based solutions possible that can add assistance at a more generic level that doesn't require software

Honest question out of curiosity- Are there possibly AI-based solutions possible that can add assistance at a more generic level that doesn't require software that has "deep" knowledge of the window architecture (and the actual text in it etc.)? My understanding is that this is how tech like VoiceOver works- it knows the actual window definition and all the elements of it at a programmatic level and can take advantage of that. I'm not asking if they're already available (although that would be a nice option), but for projects like this that wanted the speed of rendering on the GPU (at the possible cost of, as you said, VoiceOver seeing an "empty window"), it would be at least a fallback position. (So does this mean that ALL content that renders through the GPU, such as games, are inaccessible to you? If so, I'm sorry...) Not sure if this might help you but I have an Apple shortcut defined on my iPhone called "GPT Explains" that is activated by a double-tap on the back of the phone (which you can assign, as you probably know, in Accessibility settings)- it takes a screenshot, ships it off to OpenAI and returns with a description of what it's seeing, any to-English translation of non-English text, and any counterarguments to any claims made in a meme, etc. (yeah, the prompt for this is kinda wicked, lol). If this is helpful to you, I can give you a link after I remove my OpenAI key (you'd have to provide your own). EDIT: I made a copy of it without the API key: https://www.icloud.com/shortcuts/0d063c6810d74a35a017e5a5f692776b https://www.icloud.com/shortcuts/0d063c6810d74a35a017e5a5f69...

miki123211 3y ago
iOS already has such a feature, but it's obviously not 100% accurate, not real time, and not great for battery life. It's good enough if you have to click a broken "I accept your terms and conditions" checkbox, but nowhere near good enough to daily drive your phone with. In other words, a band-aid solution for when everything else fails, mostly for situations where the app you're trying to use is mostly usable, but has an accessibility barrier preventing you from carrying out a crucial step somewhere. There's also vocr on Mac (and equivalent solutions on Windows), which recently got some AI features, but it doesn't even recognize control types (a very basic feature of any screen reader), just text. Again, good enough to get you through most installers where all you're doing is clicking "next" ten times, probably good enough to get you through the first-run experience in a VM that doesn't yet have a screen reader installed, at one tenth the speed of a sighted person, but that's about it.
- pmarreck 3y ago
  Adding control type recognition seems like THE most trivial add-on feature to implement with an AI-powered solution. I mean, at that point it's just about training data... "Here are 100 different button looks. Here are 100 different radio button looks. Here are 100 different checkboxes. Here are 100 different dropdown menus." etc.

miki123211 3y ago

iOS already has such a feature, but it's obviously not 100% accurate, not real time, and not great for battery life. It's good enough if you have to click a broken "I accept your terms and conditions" checkbox, but nowhere near good enough to daily drive your phone with. In other words, a band-aid solution for when everything else fails, mostly for situations where the app you're trying to use is mostly usable, but has an accessibility barrier preventing you from carrying out a crucial step somewhere. There's also vocr on Mac (and equivalent solutions on Windows), which recently got some AI features, but it doesn't even recognize control types (a very basic feature of any screen reader), just text. Again, good enough to get you through most installers where all you're doing is clicking "next" ten times, probably good enough to get you through the first-run experience in a VM that doesn't yet have a screen reader installed, at one tenth the speed of a sighted person, but that's about it.

pmarreck 3y ago
Adding control type recognition seems like THE most trivial add-on feature to implement with an AI-powered solution. I mean, at that point it's just about training data... "Here are 100 different button looks. Here are 100 different radio button looks. Here are 100 different checkboxes. Here are 100 different dropdown menus." etc.