3 ms·

I have always wanted a browser automation tool that taps directly into the accessibility tree. Plenty do supporting querying based on accessibility features, bu

I have always wanted a browser automation tool that taps directly into the accessibility tree. Plenty do supporting querying based on accessibility features, but unless I'm mistaken none go directly to the same underlying accessibility tree used by screen readers and similar. Happy to be wrong here if anyone can correct me. The idea of all tests confirming both functionality and accessibility in one go would be much nicer than testing against hard coded test IDs and separately writing a few a11y tests if I'm offered the time.

jahewson 2y ago
It depends on what you’re testing. Much of a typical page is visual noise that is invisible to the accessibility tree but is often still something you’ll want tests for. It’s also not uncommon for accessible ui paths to differ from regular ones via invisible screen-reader only content, eg in a complex dropdown list. So you can end up with a situation where you test that accessible path works but not regular clicks! If you really want gold standard screen reader testing, there’s no substitute for testing with actual screen readers. Each uses the accessibility tree in its own way. Remember also that each browser has its own accessibility tree.
- _heimdall 2y ago
  Yeah those are interesting corner cases for sure. When UI is only visual noise and has no impact on functionality, I don't see much value in automated testing for it. In my experience these cases are often related to animations and notoriously difficult to automate tests for anyway. When UX diverges between UI and the accessibility tree, I'd really expect that to be the exception rather than the rule. There would need to be a way to test both in isolation, but when one use case diverges down two separate code paths it's begging for hard to find bugs and regressions. Totally agree on testing with screen readers directly though. I can't count how many weird differences I've come across between Windows (IE or Edge) and Mac over the years. If I remember right, there was a proposed spec for unifying the accessibility tree and related APIs but I don't think it went anywhere yet.
regularfry 2y ago
Guidepup looks like it's a decent stab in that direction: https://www.guidepup.dev/ https://www.guidepup.dev/ Only Windows and MacOS though, which is a problem for build pipelines. I too would very much like the page descriptions and the accessibility inputs to be the primary way of driving a page. It would make accessible access the default, rather than something you have to argue for.
- _heimdall 2y ago
  That's an interesting one, thanks! Skimming through their getting started, I wonder how translations would be handled. It looks like the tests expect to validate what the actual screen reader says rather than just the tree, for example their first test shows finding the Guidepup header in their readme my waiting for the screen reader to say "Guidepup heading level 1". If you need to test different languages, you'd have to match the phrasing used by each specific screen reader when reading the heading descriptor and text. All your tests are also actually vulnerable to any phrasing changes made to each screen reader. If VoiceOver changed something it could break all your test values. I bet they could hide that behind abstractions though, `expectHeading("Guidepup", 1)` or similar. Ideally it really would just be a check in the tree though, avoiding any particular implementation of a screen reader all together.

jahewson 2y ago

It depends on what you’re testing. Much of a typical page is visual noise that is invisible to the accessibility tree but is often still something you’ll want tests for. It’s also not uncommon for accessible ui paths to differ from regular ones via invisible screen-reader only content, eg in a complex dropdown list. So you can end up with a situation where you test that accessible path works but not regular clicks! If you really want gold standard screen reader testing, there’s no substitute for testing with actual screen readers. Each uses the accessibility tree in its own way. Remember also that each browser has its own accessibility tree.

_heimdall 2y ago
Yeah those are interesting corner cases for sure. When UI is only visual noise and has no impact on functionality, I don't see much value in automated testing for it. In my experience these cases are often related to animations and notoriously difficult to automate tests for anyway. When UX diverges between UI and the accessibility tree, I'd really expect that to be the exception rather than the rule. There would need to be a way to test both in isolation, but when one use case diverges down two separate code paths it's begging for hard to find bugs and regressions. Totally agree on testing with screen readers directly though. I can't count how many weird differences I've come across between Windows (IE or Edge) and Mac over the years. If I remember right, there was a proposed spec for unifying the accessibility tree and related APIs but I don't think it went anywhere yet.

regularfry 2y ago

Guidepup looks like it's a decent stab in that direction: https://www.guidepup.dev/ https://www.guidepup.dev/ Only Windows and MacOS though, which is a problem for build pipelines. I too would very much like the page descriptions and the accessibility inputs to be the primary way of driving a page. It would make accessible access the default, rather than something you have to argue for.

_heimdall 2y ago
That's an interesting one, thanks! Skimming through their getting started, I wonder how translations would be handled. It looks like the tests expect to validate what the actual screen reader says rather than just the tree, for example their first test shows finding the Guidepup header in their readme my waiting for the screen reader to say "Guidepup heading level 1". If you need to test different languages, you'd have to match the phrasing used by each specific screen reader when reading the heading descriptor and text. All your tests are also actually vulnerable to any phrasing changes made to each screen reader. If VoiceOver changed something it could break all your test values. I bet they could hide that behind abstractions though, `expectHeading("Guidepup", 1)` or similar. Ideally it really would just be a check in the tree though, avoiding any particular implementation of a screen reader all together.