3 ms·

> to abstract and sandbox the platform's native accessibility APIs I wonder if it's important to expose something close to each platform's accessibility API to

> to abstract and sandbox the platform's native accessibility APIs I wonder if it's important to expose something close to each platform's accessibility API to sandboxed applications, or only the cross-platform abstraction over those APIs. For the latter, my AccessKit [1] project might be worth looking at. [1]: https://github.com/AccessKit/accesskit https://github.com/AccessKit/accesskit

mike_hearn 4y ago
Why not both? The idea behind allowing the sandbox to be pluggable is the sandbox authors can decide how and what to expose. If the sandbox author thinks it's safe to expose the native accessibility APIs directly it can, if they don't then they may wish to expose a platform-specific proxy emulation with extra validation (in the way that Chrome exports OpenGL into the sandbox), or they may wish to expose only a platform abstraction, and that platform abstraction could also be used outside of any sandboxes as well if the app is code signed. AccessKit certainly looks highly competent - putting an Mojo RPC interface over it would be a good way to expose an abstraction over platform specific a11y interfaces. And of course there's no rule that says an abstraction must always provide just the lowest common denominator. Abstractions can also expose platform specific features for the unique stuff that can't be properly abstracted. I suspect a11y APIs will gain significant importance this decade, once people realize they can hook AI up to accessibility trees more easily than working with pixels and the like.
- mwcampbell 4y ago
  IMO, now that I think about this some more, letting a sandboxed app directly implement native accessibility APIs is risky. First, there's the fact that on Windows, macOS, and Linux (or more accurately, Unix desktop environments that use AT-SPI), the native accessibility API uses screen coordinates. So a provider exposing screen coordinates outside of its window seems like a vulnerability. AT-SPI is very trusting of the toolkit/application in general, with no verifiable link between the accessibility tree and the X11/Wayland windows/surfaces. And while I'm not aware of other specific vulnerabilities in UI Automation on Windows, I wouldn't be surprised to find that they exist. I'm the least familiar with the macOS accessibility API. > I suspect a11y APIs will gain significant importance this decade, once people realize they can hook AI up to accessibility trees more easily than working with pixels and the like. I hadn't considered that programmatic accessibility might be helpful for AI applications. Within the accessibility community, AI is more often seen as a provider, i.e. filling in accessibility gaps, through OCR and now more advanced applications of machine learning such as iOS's Screen Recognition. But I hope you're right.
  - mike_hearn 4y ago
    Insightful, thanks. Yes - native operating system APIs are often not designed with sandboxing in mind, unlike on mobile. A big part of why people rely on the web so heavily can be summed up as the Chrome guys methodically working through native APIs and writing lots of code to make them sandbox-safe. I guess in this case it'd be the same thing. Someone would need to write a validation layer. Chrome's Mojo IPC is really nice, I've been playing with it lately. It's not available outside their source tree but isn't too hard to extract. Their sandbox API is ... less nice, barely an API at all actually, but nonetheless the two integrate with each other well. I think accessibility becomes a lot more exciting to developers once they re-envision it as a general purpose API to expose application interfaces without being drowned in markup or other incidental complexity. The 90s concept was that apps should expose a GUI widget tree to end users and an OOP RPC API to other applications, with MS Office being the cheerleader for this approach. But it never caught on because exposing an API feels too much like a tax for most apps - it's hard to know what other devs might want, making it hard to design an API for them, API design is itself quite taxing and then finally what is the stability level like? You also have the question of what RPC system to use, what type system ... it's complex. Exporting a generic RPC API in the form of a11y trees is an interesting middle ground. It's clear that it's not a stability commitment, but at the same time it's more stable and easier to work with than raw screen scraping.
    - mwcampbell 4y ago
      It might interest you to know that Windows screen reader developers used Office's COM object models to make the Office apps (especially the primary document views) accessible for a long time, going back to the early 2000s or late 90s. The third-party Windows screen readers (i.e. everyone but Narrator) are just now transitioning to using UI Automation for Office documents. Microsoft's original accessibility API, Active Accessibility, was way too simplistic, and even UIA only recently became rich and extensible enough to expose everything that screen readers got out of the Office object models.

mike_hearn 4y ago

Why not both? The idea behind allowing the sandbox to be pluggable is the sandbox authors can decide how and what to expose. If the sandbox author thinks it's safe to expose the native accessibility APIs directly it can, if they don't then they may wish to expose a platform-specific proxy emulation with extra validation (in the way that Chrome exports OpenGL into the sandbox), or they may wish to expose only a platform abstraction, and that platform abstraction could also be used outside of any sandboxes as well if the app is code signed. AccessKit certainly looks highly competent - putting an Mojo RPC interface over it would be a good way to expose an abstraction over platform specific a11y interfaces. And of course there's no rule that says an abstraction must always provide just the lowest common denominator. Abstractions can also expose platform specific features for the unique stuff that can't be properly abstracted. I suspect a11y APIs will gain significant importance this decade, once people realize they can hook AI up to accessibility trees more easily than working with pixels and the like.

mwcampbell 4y ago
IMO, now that I think about this some more, letting a sandboxed app directly implement native accessibility APIs is risky. First, there's the fact that on Windows, macOS, and Linux (or more accurately, Unix desktop environments that use AT-SPI), the native accessibility API uses screen coordinates. So a provider exposing screen coordinates outside of its window seems like a vulnerability. AT-SPI is very trusting of the toolkit/application in general, with no verifiable link between the accessibility tree and the X11/Wayland windows/surfaces. And while I'm not aware of other specific vulnerabilities in UI Automation on Windows, I wouldn't be surprised to find that they exist. I'm the least familiar with the macOS accessibility API. > I suspect a11y APIs will gain significant importance this decade, once people realize they can hook AI up to accessibility trees more easily than working with pixels and the like. I hadn't considered that programmatic accessibility might be helpful for AI applications. Within the accessibility community, AI is more often seen as a provider, i.e. filling in accessibility gaps, through OCR and now more advanced applications of machine learning such as iOS's Screen Recognition. But I hope you're right.
- mike_hearn 4y ago
  Insightful, thanks. Yes - native operating system APIs are often not designed with sandboxing in mind, unlike on mobile. A big part of why people rely on the web so heavily can be summed up as the Chrome guys methodically working through native APIs and writing lots of code to make them sandbox-safe. I guess in this case it'd be the same thing. Someone would need to write a validation layer. Chrome's Mojo IPC is really nice, I've been playing with it lately. It's not available outside their source tree but isn't too hard to extract. Their sandbox API is ... less nice, barely an API at all actually, but nonetheless the two integrate with each other well. I think accessibility becomes a lot more exciting to developers once they re-envision it as a general purpose API to expose application interfaces without being drowned in markup or other incidental complexity. The 90s concept was that apps should expose a GUI widget tree to end users and an OOP RPC API to other applications, with MS Office being the cheerleader for this approach. But it never caught on because exposing an API feels too much like a tax for most apps - it's hard to know what other devs might want, making it hard to design an API for them, API design is itself quite taxing and then finally what is the stability level like? You also have the question of what RPC system to use, what type system ... it's complex. Exporting a generic RPC API in the form of a11y trees is an interesting middle ground. It's clear that it's not a stability commitment, but at the same time it's more stable and easier to work with than raw screen scraping.
  - mwcampbell 4y ago
    It might interest you to know that Windows screen reader developers used Office's COM object models to make the Office apps (especially the primary document views) accessible for a long time, going back to the early 2000s or late 90s. The third-party Windows screen readers (i.e. everyone but Narrator) are just now transitioning to using UI Automation for Office documents. Microsoft's original accessibility API, Active Accessibility, was way too simplistic, and even UIA only recently became rich and extensible enough to expose everything that screen readers got out of the Office object models.