5 ms·

This is an interesting idea for app-interoperability in general. It brings to mind Apple Automator, which I have no idea how works, but can sometimes be used to

This is an interesting idea for app-interoperability in general. It brings to mind Apple Automator, which I have no idea how works, but can sometimes be used to make Apps interoperate. It'd be pretty cool if every app did have a sitemap-like API spec that that mapped out the core of the app, and the UI sat on top of that.

mitemte 5y ago
A lot of the UI scripting on macOS is accomplished via accessibility APIs. If you’re interested, check out Accessibility Inspector[0], which ships with Xcode. You can inspect macOS applications in a similar, albeit much more limited way to a web browser’s interactive inspect element function. You can also accomplish UI scripting via C/ObjC/Swift using the Application Services framework [1]. [0] https://developer.apple.com/library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/OSXAXTestingApps.html https://developer.apple.com/library/archive/documentation/Ac... [1] https://developer.apple.com/documentation/applicationservices https://developer.apple.com/documentation/applicationservice...
csmpltn 5y ago
Exactly. We should decouple the core of the application from the UI. Have a standardized interface which exposes all available functionality to the screen reader and allows it to talk to the core directly, bypassing the UI. Stop doing what everybody does today: stop having screen reader depend on the UI (ie. aria-labels, etc). This problem can't be solved by adding "decorations" to existing UI components. It's best solved by exposing an entirely separate interface which lets screen readers (and other hardware) interact with the core of the application directly. This sounds like a spec, a set of language frameworks and a design pattern waiting to be written.
- autoexec 5y ago
  If it's done right and standardized it'd be worth it for everybody since even people who don't need accessibility features could then create their own interfaces to the programs they use customized for their preferences and workflow
jcelerier 5y ago
These APIs have existed for like three or more decades for desktop apps. That's partly why most desktop UI toolkits are based on tree of widgets: Mac and Windows's native accessibility APIs expect a tree
- mwcampbell 5y ago
  FWIW, the oldest accessibility API was Microsoft Active Accessibility (MSAA), which was added to Windows 95 in a service pack in 1997. Classic MacOS never had an accessibility API; the Outspoken screen reader for that OS had to rely exclusively on ugly hacks (e.g. patching Toolbox functions). OS X introduced accessibility APIs for Carbon and Cocoa in an early version; not sure which one. Unix-based desktop environments got an accessibility API, AT-SPI (thank you Sun), in the early 2000s.
  - jcelerier 5y ago
    well TIL, I believed accessibility to already be there by the late 80s, in NeXT workstations and the first windows versions but it looks like that was not the case.

mitemte 5y ago

A lot of the UI scripting on macOS is accomplished via accessibility APIs. If you’re interested, check out Accessibility Inspector[0], which ships with Xcode. You can inspect macOS applications in a similar, albeit much more limited way to a web browser’s interactive inspect element function. You can also accomplish UI scripting via C/ObjC/Swift using the Application Services framework [1]. [0] https://developer.apple.com/library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/OSXAXTestingApps.html https://developer.apple.com/library/archive/documentation/Ac... [1] https://developer.apple.com/documentation/applicationservices https://developer.apple.com/documentation/applicationservice...

csmpltn 5y ago

Exactly. We should decouple the core of the application from the UI. Have a standardized interface which exposes all available functionality to the screen reader and allows it to talk to the core directly, bypassing the UI. Stop doing what everybody does today: stop having screen reader depend on the UI (ie. aria-labels, etc). This problem can't be solved by adding "decorations" to existing UI components. It's best solved by exposing an entirely separate interface which lets screen readers (and other hardware) interact with the core of the application directly. This sounds like a spec, a set of language frameworks and a design pattern waiting to be written.

autoexec 5y ago
If it's done right and standardized it'd be worth it for everybody since even people who don't need accessibility features could then create their own interfaces to the programs they use customized for their preferences and workflow

jcelerier 5y ago

These APIs have existed for like three or more decades for desktop apps. That's partly why most desktop UI toolkits are based on tree of widgets: Mac and Windows's native accessibility APIs expect a tree

mwcampbell 5y ago
FWIW, the oldest accessibility API was Microsoft Active Accessibility (MSAA), which was added to Windows 95 in a service pack in 1997. Classic MacOS never had an accessibility API; the Outspoken screen reader for that OS had to rely exclusively on ugly hacks (e.g. patching Toolbox functions). OS X introduced accessibility APIs for Carbon and Cocoa in an early version; not sure which one. Unix-based desktop environments got an accessibility API, AT-SPI (thank you Sun), in the early 2000s.
- jcelerier 5y ago
  well TIL, I believed accessibility to already be there by the late 80s, in NeXT workstations and the first windows versions but it looks like that was not the case.