3 ms·
“The idea is not a new one, exactly; Bigham mentioned a screen reader, Outspoken, which years ago attempted to use pixel-level data to identify UI elements” I
by Someone 6y ago
“The idea is not a new one, exactly; Bigham mentioned a screen reader, Outspoken, which years ago attempted to use pixel-level data to identify UI elements”
I don’t think that feature, if implemented at all, ever made it into production. Outspoken looked at system calls (sometimes system call sequences), not the resulting pixels.
I think Apple itself, at some time, had an advanced version of the monkey (https://www.folklore.org/StoryView.py?story=Monkey_Lives.txt https://www.folklore.org/StoryView.py?story=Monkey_Lives.txt) that did look at pixel data (so that, for example, one could script it to click an “OK” button without knowing where that button was, or to shift-click between two letters in a given font/font size combination)