4 ms·

I feel that this is where accessibility needs to go. We cannot rely on toolkits to implement support for accessibility standards because there will always be pe

by suby 4y ago

I feel that this is where accessibility needs to go. We cannot rely on toolkits to implement support for accessibility standards because there will always be people building apps in niche toolkits / homemade gui's which don't support accessibility, especially as wasm becomes more common on the web. We need to treat the screen like a PNG which an AI can parse to determine what the text is, what text is relevant to read to the user, where and what the buttons are, how to emphasize reading the text based on the visuals, etc. It's the only thing that will scale and while difficult it seems that creating something like this is within reach.

layer8 4y ago
I agree in principle, but good luck with AI interpreting flat-UI controls visually, it’s confusing enough for sighted users. This actually gives me a little hope that AI-interpreted UI could trigger a trend back to visually obvious and unambiguous controls.
madeofpalk 4y ago
No. That’s like saying OS should use AI to try and figure out what buttons are to make them clickable, rather that developers actually building an interface for users. All operating systems and platforms have pretty fluent and in-depth APIs for exposing an “accessibility tree” for assistive technologies to use. It’s your responsibility as a developer to build interfaces for actual users, and there’s very little reason not to.
- ronsor 4y ago
 > That’s like saying OS should use AI to try and figure out what buttons are to make them clickable, rather that developers actually building an interface for users. That's actually a really good idea.
 - maxwelljoslyn 4y ago
 Especially now that designers are making buttons/clickable areas hard to distinguish ... I could totally use AI to highlight all the things that are actually, you know, interactive.
 - nirimda 4y ago
 You don't need an AI to do this, just make your own custom build of whatever operating system you use with styles that aren't crap. It's almost certainly a more tractable task than trying to make an AI work out what the interactive bits are - considering it seems to be a business objective for many companies nowadays to make it hard to work out what the interactive bits are.
- hollasch 4y ago
 I've seen systems at Boeing that understand CAD drawings semantically, like how arrows work and what regions are alternate views of what other regions, and whether a drawing is of the nose landing gear or the flap control, all based on scanned drawings from the 50's. So ... yes. Absolutely.
 - cratermoon 4y ago
 > all based on scanned drawings from the 50's It's 2059. We're using browsers that render pages created with Dreamweaver and Frontpage using advanced machine learning techniques to determine that <blockquote> is an entry in the table of contents, while is a level 1 header.
 - maxwelljoslyn 4y ago
 That sounds SO COOL. Can you tell me more? Or do I have to go ask my friends at Boeing about it? :)
hollasch 4y ago
YES. I have seen layout prototypes for this kind of thinking, and they're magical. It's not trivial, but will come more and more into reach. All of visual layout delivers a spatial visual language that all humans understand. We know what headers _are_ because we know what headers _look like_. And not for a single style of article, but for all articles. I also used to believe that rigorous semantic tagging was The Way To Do Documents, and it was certainly a useful crutch, but we absolutely need to move beyond that.

by suby 4y ago

layer8 4y ago

I agree in principle, but good luck with AI interpreting flat-UI controls visually, it’s confusing enough for sighted users. This actually gives me a little hope that AI-interpreted UI could trigger a trend back to visually obvious and unambiguous controls.

madeofpalk 4y ago

No. That’s like saying OS should use AI to try and figure out what buttons are to make them clickable, rather that developers actually building an interface for users. All operating systems and platforms have pretty fluent and in-depth APIs for exposing an “accessibility tree” for assistive technologies to use. It’s your responsibility as a developer to build interfaces for actual users, and there’s very little reason not to.

ronsor 4y ago
> That’s like saying OS should use AI to try and figure out what buttons are to make them clickable, rather that developers actually building an interface for users. That's actually a really good idea.
- maxwelljoslyn 4y ago
  Especially now that designers are making buttons/clickable areas hard to distinguish ... I could totally use AI to highlight all the things that are actually, you know, interactive.
  - nirimda 4y ago
    You don't need an AI to do this, just make your own custom build of whatever operating system you use with styles that aren't crap. It's almost certainly a more tractable task than trying to make an AI work out what the interactive bits are - considering it seems to be a business objective for many companies nowadays to make it hard to work out what the interactive bits are.
hollasch 4y ago
I've seen systems at Boeing that understand CAD drawings semantically, like how arrows work and what regions are alternate views of what other regions, and whether a drawing is of the nose landing gear or the flap control, all based on scanned drawings from the 50's. So ... yes. Absolutely.
- cratermoon 4y ago
 > all based on scanned drawings from the 50's It's 2059. We're using browsers that render pages created with Dreamweaver and Frontpage using advanced machine learning techniques to determine that <blockquote> is an entry in the table of contents, while is a level 1 header.
- maxwelljoslyn 4y ago
 That sounds SO COOL. Can you tell me more? Or do I have to go ask my friends at Boeing about it? :)

hollasch 4y ago

YES. I have seen layout prototypes for this kind of thinking, and they're magical. It's not trivial, but will come more and more into reach. All of visual layout delivers a spatial visual language that all humans understand. We know what headers _are_ because we know what headers _look like_. And not for a single style of article, but for all articles. I also used to believe that rigorous semantic tagging was The Way To Do Documents, and it was certainly a useful crutch, but we absolutely need to move beyond that.