3 ms·
iPhones can now automatically recognize and label buttons UI for blind users
- gostsamo 6y agoThis is actually rather cool. There are some templates for making accessible some desktop apps, though they are made on case by case bases. I had an idea for using ml to substitute the templates for some really inaccessible desktop apps but my ML skills and money for dataset creation are a bit smaller than Apple's. Hopefully, the approach becomes a standard, because with the advance of AR and VR accessibility could be really screwed in the not so far future.
- Someone 6y ago“The idea is not a new one, exactly; Bigham mentioned a screen reader, Outspoken, which years ago attempted to use pixel-level data to identify UI elements” I don’t think that feature, if implemented at all, ever made it into production. Outspoken looked at system calls (sometimes system call sequences), not the resulting pixels. I think Apple itself, at some time, had an advanced version of the monkey (https://www.folklore.org/StoryView.py?story=Monkey_Lives.txt https://www.folklore.org/StoryView.py?story=Monkey_Lives.txt) that did look at pixel data (so that, for example, one could script it to click an “OK” button without knowing where that button was, or to shift-click between two letters in a given font/font size combination)