2 ms·

IMO, now that I think about this some more, letting a sandboxed app directly implement native accessibility APIs is risky. First, there's the fact that on Windo

IMO, now that I think about this some more, letting a sandboxed app directly implement native accessibility APIs is risky. First, there's the fact that on Windows, macOS, and Linux (or more accurately, Unix desktop environments that use AT-SPI), the native accessibility API uses screen coordinates. So a provider exposing screen coordinates outside of its window seems like a vulnerability. AT-SPI is very trusting of the toolkit/application in general, with no verifiable link between the accessibility tree and the X11/Wayland windows/surfaces. And while I'm not aware of other specific vulnerabilities in UI Automation on Windows, I wouldn't be surprised to find that they exist. I'm the least familiar with the macOS accessibility API. > I suspect a11y APIs will gain significant importance this decade, once people realize they can hook AI up to accessibility trees more easily than working with pixels and the like. I hadn't considered that programmatic accessibility might be helpful for AI applications. Within the accessibility community, AI is more often seen as a provider, i.e. filling in accessibility gaps, through OCR and now more advanced applications of machine learning such as iOS's Screen Recognition. But I hope you're right.

mike_hearn 4y ago
Insightful, thanks. Yes - native operating system APIs are often not designed with sandboxing in mind, unlike on mobile. A big part of why people rely on the web so heavily can be summed up as the Chrome guys methodically working through native APIs and writing lots of code to make them sandbox-safe. I guess in this case it'd be the same thing. Someone would need to write a validation layer. Chrome's Mojo IPC is really nice, I've been playing with it lately. It's not available outside their source tree but isn't too hard to extract. Their sandbox API is ... less nice, barely an API at all actually, but nonetheless the two integrate with each other well. I think accessibility becomes a lot more exciting to developers once they re-envision it as a general purpose API to expose application interfaces without being drowned in markup or other incidental complexity. The 90s concept was that apps should expose a GUI widget tree to end users and an OOP RPC API to other applications, with MS Office being the cheerleader for this approach. But it never caught on because exposing an API feels too much like a tax for most apps - it's hard to know what other devs might want, making it hard to design an API for them, API design is itself quite taxing and then finally what is the stability level like? You also have the question of what RPC system to use, what type system ... it's complex. Exporting a generic RPC API in the form of a11y trees is an interesting middle ground. It's clear that it's not a stability commitment, but at the same time it's more stable and easier to work with than raw screen scraping.
- mwcampbell 4y ago
  It might interest you to know that Windows screen reader developers used Office's COM object models to make the Office apps (especially the primary document views) accessible for a long time, going back to the early 2000s or late 90s. The third-party Windows screen readers (i.e. everyone but Narrator) are just now transitioning to using UI Automation for Office documents. Microsoft's original accessibility API, Active Accessibility, was way too simplistic, and even UIA only recently became rich and extensible enough to expose everything that screen readers got out of the Office object models.

mike_hearn 4y ago

Insightful, thanks. Yes - native operating system APIs are often not designed with sandboxing in mind, unlike on mobile. A big part of why people rely on the web so heavily can be summed up as the Chrome guys methodically working through native APIs and writing lots of code to make them sandbox-safe. I guess in this case it'd be the same thing. Someone would need to write a validation layer. Chrome's Mojo IPC is really nice, I've been playing with it lately. It's not available outside their source tree but isn't too hard to extract. Their sandbox API is ... less nice, barely an API at all actually, but nonetheless the two integrate with each other well. I think accessibility becomes a lot more exciting to developers once they re-envision it as a general purpose API to expose application interfaces without being drowned in markup or other incidental complexity. The 90s concept was that apps should expose a GUI widget tree to end users and an OOP RPC API to other applications, with MS Office being the cheerleader for this approach. But it never caught on because exposing an API feels too much like a tax for most apps - it's hard to know what other devs might want, making it hard to design an API for them, API design is itself quite taxing and then finally what is the stability level like? You also have the question of what RPC system to use, what type system ... it's complex. Exporting a generic RPC API in the form of a11y trees is an interesting middle ground. It's clear that it's not a stability commitment, but at the same time it's more stable and easier to work with than raw screen scraping.

mwcampbell 4y ago
It might interest you to know that Windows screen reader developers used Office's COM object models to make the Office apps (especially the primary document views) accessible for a long time, going back to the early 2000s or late 90s. The third-party Windows screen readers (i.e. everyone but Narrator) are just now transitioning to using UI Automation for Office documents. Microsoft's original accessibility API, Active Accessibility, was way too simplistic, and even UIA only recently became rich and extensible enough to expose everything that screen readers got out of the Office object models.