3 ms·

Fascinating! What are some of the limitations of screen scraping that are addressed by accessibility APIs? Is it primarily the unreliability? Do you envision us

Fascinating! What are some of the limitations of screen scraping that are addressed by accessibility APIs? Is it primarily the unreliability? Do you envision using an accessibility API as the primary source of information, with screen scraping as a fallback?

DonHopkins 9y ago
Accessibility APIs expose lots of rich high level information like associating a label with a widget (to name one simple example, but it gets MUCH more complex than that). And of course the position and meaning and state of each object on the screen. There needs to be a higher level scriptable way to get a handle on all that complexity, like jQuery helps automate the creation and manipulation and abstraction of HTML DOM and events and handlers. PhoneGap/Cordova plugins and NativeScript both take a similar approach: they have a generic Objective C / Java / Whatever <=> JavaScript bridge that lets you directly call all (or most) of the native APIs, including accessibility, directly on each platform. Then you write higher level platform independent APIs on top of the native APIs in JavaScript, a different one for each platform, but the same high level interface to JavaScript programmers. https://docs.nativescript.org/core-concepts/accessing-native-apis-with-javascript https://docs.nativescript.org/core-concepts/accessing-native... I would try to use the native API as much as possible since they're precise yet tedious. It should have a platform-independent accessibility-specific selector language like xpath or jQuery selectors, which should be implemented efficiently in native code like querySelector. The pattern matching engine would send asynchronous events back to JavaScript. You could write JavaScript handlers for patterns of pixels as well as accessibility path patterns appearing and disappearing, like jQuery lets you bind handlers to patterns of DOM elements that don't exist yet, instead of concrete existing elements. Those handlers would create higher level widgets that could manage those existing pixels and widgets. (Like recognizing a youtube video player in any web browser window, and wrapping it in an aQuery widget implementing an abstract VideoPlayer interface, for example.) Of course calling the native accessibility API is lighter weight than screen scraping and pixel matching, and you could use the native API to drill down to just the region of pixels you want to match or screencast, to minimize the amount of screen scraping and pixel matching required.

DonHopkins 9y ago

Accessibility APIs expose lots of rich high level information like associating a label with a widget (to name one simple example, but it gets MUCH more complex than that). And of course the position and meaning and state of each object on the screen. There needs to be a higher level scriptable way to get a handle on all that complexity, like jQuery helps automate the creation and manipulation and abstraction of HTML DOM and events and handlers. PhoneGap/Cordova plugins and NativeScript both take a similar approach: they have a generic Objective C / Java / Whatever <=> JavaScript bridge that lets you directly call all (or most) of the native APIs, including accessibility, directly on each platform. Then you write higher level platform independent APIs on top of the native APIs in JavaScript, a different one for each platform, but the same high level interface to JavaScript programmers. https://docs.nativescript.org/core-concepts/accessing-native-apis-with-javascript https://docs.nativescript.org/core-concepts/accessing-native... I would try to use the native API as much as possible since they're precise yet tedious. It should have a platform-independent accessibility-specific selector language like xpath or jQuery selectors, which should be implemented efficiently in native code like querySelector. The pattern matching engine would send asynchronous events back to JavaScript. You could write JavaScript handlers for patterns of pixels as well as accessibility path patterns appearing and disappearing, like jQuery lets you bind handlers to patterns of DOM elements that don't exist yet, instead of concrete existing elements. Those handlers would create higher level widgets that could manage those existing pixels and widgets. (Like recognizing a youtube video player in any web browser window, and wrapping it in an aQuery widget implementing an abstract VideoPlayer interface, for example.) Of course calling the native accessibility API is lighter weight than screen scraping and pixel matching, and you could use the native API to drill down to just the region of pixels you want to match or screencast, to minimize the amount of screen scraping and pixel matching required.