3 ms·

If you're on macOS and interested in this space, I highly recommend you open up the system-provided Accessibility Inspector.app and play around with apps and br

by btown 5mo ago

If you're on macOS and interested in this space, I highly recommend you open up the system-provided Accessibility Inspector.app and play around with apps and browsers. See how the green cells might guide an LLM to only need to read/OCR specific parts of a screen, how much text is already natively available to the accessibility engine, and how this could lead to really effective hybrid systems - not just MCPs, but code generators that can build and run their own scripts to crawl your accessibility hierarchy for your workflow! I think this is very fertile ground - big labs need to use approaches that can work on multiple platforms and arbitrary workflows, and full-page vision is the lowest common denominator. Platform-specific approaches are a really exciting open space!

merlindru 5mo ago
That's how I got into this thing in the first place, hah. Golden advice. It's incredibly cool to see what some apps offer. More of them have great accessibility support than you think (or at least than I thought!)
drob518 5mo ago
Great idea.
willwade 5mo ago
take a peek at https://github.com/willwade/app-automate?tab=readme-ov-file#app-automate https://github.com/willwade/app-automate?tab=readme-ov-file#... - its early and needs some work -but this is the idea behind this.. (my use case is not agents but actual real disabled people..who need tooling to provide better access to the desktop)
jasomill 5mo ago
Windows has similar APIs and tools, see, e.g., https://accessibilityinsights.io/ https://accessibilityinsights.io/ https://learn.microsoft.com/en-us/windows/win32/winauto/inspect-objects https://learn.microsoft.com/en-us/windows/win32/winauto/insp... https://github.com/FlaUI/FlaUInspect https://github.com/FlaUI/FlaUInspect and for WPF applications specifically, https://github.com/snoopwpf/snoopwpf https://github.com/snoopwpf/snoopwpf
Noumenon72 5mo ago
According to ChatGPT Accessibility Inspector is a developer tool in Xcode now.

by btown 5mo ago

merlindru 5mo ago

That's how I got into this thing in the first place, hah. Golden advice. It's incredibly cool to see what some apps offer. More of them have great accessibility support than you think (or at least than I thought!)

drob518 5mo ago

Great idea.

willwade 5mo ago

take a peek at https://github.com/willwade/app-automate?tab=readme-ov-file#app-automate https://github.com/willwade/app-automate?tab=readme-ov-file#... - its early and needs some work -but this is the idea behind this.. (my use case is not agents but actual real disabled people..who need tooling to provide better access to the desktop)

jasomill 5mo ago

Windows has similar APIs and tools, see, e.g., https://accessibilityinsights.io/ https://accessibilityinsights.io/ https://learn.microsoft.com/en-us/windows/win32/winauto/inspect-objects https://learn.microsoft.com/en-us/windows/win32/winauto/insp... https://github.com/FlaUI/FlaUInspect https://github.com/FlaUI/FlaUInspect and for WPF applications specifically, https://github.com/snoopwpf/snoopwpf https://github.com/snoopwpf/snoopwpf

Noumenon72 5mo ago

According to ChatGPT Accessibility Inspector is a developer tool in Xcode now.