3 ms·

Is there a case for ML based advanced screen readers which do not need assistance from the application ? The problem seems fairly tacklable . Learning what is

Is there a case for ML based advanced screen readers which do not need assistance from the application ? The problem seems fairly tacklable . Learning what is on a display screen is relatively easier than most computer vision problem spaces. There are many repetitive patterns in typical application UX. For example let say there is a label for Save Icon that is an image (a Floppy Disk in most apps) and not alt tagged. By visually reading the image of the screen the model should not have to much difficulty in tagging it that as Save button ? Most consumer / biz app UX do follow many standard conventions if only out of convenience and lack of imagination, so building a learning algorithm around these components should be possible ?

mwcampbell 5y ago
This is being worked on. AFAIK, Apple is the first to incorporate this approach into a released product, with the Screen Recognition feature of VoiceOver starting in iOS 14.
- manquer 5y ago
  Thanks for the response Matt. I leave the link here for others to look into [1]. Their effort seems currently limited to iOS based Phone screens. iOS is perhaps easier to solve given the strong Apple design guidelines for apps to pass the App Store review process. Perhaps a community supported distributed approach to help build the database of annotated screens for the model to learn from, combined with open source models for all kinds of screens and applications( not just Apple) would be interesting project to work on. [1] https://machinelearning.apple.com/research/creating-accessibility-metadata https://machinelearning.apple.com/research/creating-accessib...
  - SLJ7 5y ago
    Interestingly, iOS screen recognition also allows exploration of screenshots and remote desktop screens. I've heard of people using it to remotely install Windows as well. It would surprise me if Apple didn't have plans to put the same feature into its M1 mac.
peterkos 5y ago
This paper[0] takes a look at something like this, but it's notable that this is seen as a springboard for more accessible-focused design, rather than the beginning and the end (See "Discussion & Future Work"). [0] https://dl.acm.org/doi/abs/10.1145/3411764.3445186 https://dl.acm.org/doi/abs/10.1145/3411764.3445186 Edit: I realize I've just linked to the same paper as the comment below. Oh well!
neurostimulant 5y ago
Heck, a desktop application that launched from a hot key and immediately highlighted any possible buttons and input fields it can recognize on a screen/window and simulate click on selection is already very useful.

mwcampbell 5y ago

This is being worked on. AFAIK, Apple is the first to incorporate this approach into a released product, with the Screen Recognition feature of VoiceOver starting in iOS 14.

manquer 5y ago
Thanks for the response Matt. I leave the link here for others to look into [1]. Their effort seems currently limited to iOS based Phone screens. iOS is perhaps easier to solve given the strong Apple design guidelines for apps to pass the App Store review process. Perhaps a community supported distributed approach to help build the database of annotated screens for the model to learn from, combined with open source models for all kinds of screens and applications( not just Apple) would be interesting project to work on. [1] https://machinelearning.apple.com/research/creating-accessibility-metadata https://machinelearning.apple.com/research/creating-accessib...
- SLJ7 5y ago
  Interestingly, iOS screen recognition also allows exploration of screenshots and remote desktop screens. I've heard of people using it to remotely install Windows as well. It would surprise me if Apple didn't have plans to put the same feature into its M1 mac.

peterkos 5y ago

This paper[0] takes a look at something like this, but it's notable that this is seen as a springboard for more accessible-focused design, rather than the beginning and the end (See "Discussion & Future Work"). [0] https://dl.acm.org/doi/abs/10.1145/3411764.3445186 https://dl.acm.org/doi/abs/10.1145/3411764.3445186 Edit: I realize I've just linked to the same paper as the comment below. Oh well!

neurostimulant 5y ago

Heck, a desktop application that launched from a hot key and immediately highlighted any possible buttons and input fields it can recognize on a screen/window and simulate click on selection is already very useful.