7 ms·

I know that this project is still in very early stages, however as I often do, I've come here to mention accessibility. I've run the examples on both Mac and Wi

I know that this project is still in very early stages, however as I often do, I've come here to mention accessibility. I've run the examples on both Mac and Windows, and as expected, I can't use either of them. This is not a strict criticism yet and I'm aware how difficult it is to interface with accessibility API's, but I do hope that if this project does make it far enough that this will be considered. Pretty much everything that draws their UI directly to the screen using OpenGL or similar immediately makes me think that I probably don't even need to try the program to know that it's inaccessible, and more often than not I'm right. If you're building an app using these gui toolkits, please do keep this in mind.

ClumsyPilot 5y ago
This is the one real (and legal) reason why people should actually use OS UI. To replicate this functionality is a tall order. The 'looks native' thing is mostly noise
pqb 5y ago
This, hundreds times this. I can't say how many years in a row I was experimenting with various GUI toolkits, libraries and concepts of an application like rxi/lite editor [0]. While I saw many good looking implementations but the very first what caught my attention was lack of any kind support to screen-reading software or problem with displaying right-click context menus. I even remember previous HN discussion when I pointed out few projects that could provide some screen-readers support [1]. Sadly, I would say 95% of all GUIs I know do not care about the accessibility. From the positive side, I am glad more and more hackers on HN focus on that matter. Also worth noting big players on the market really do take that point seriously, e.g. Flutter has a Semantics tag [2]. [0]: https://github.com/rxi/lite https://github.com/rxi/lite [1]: https://news.ycombinator.com/item?id=26223380 https://news.ycombinator.com/item?id=26223380 [2]: https://api.flutter.dev/flutter/widgets/Semantics-class.html https://api.flutter.dev/flutter/widgets/Semantics-class.html
- amelius 5y ago
  In my opinion, accessibility should be implemented at the OS level. The OS can, for example, run OCR on the entire screen and turn bitmaps into selectable text, read text out loud, etc. This kind of functionality shouldn't be replicated on a per-app basis. Accessibility of browsers is usually considered to be very good, but browsers are almost OSes, so why not lift this one more level into the OS?
  - zxzax 5y ago
    That is not really going to work, at all. To have a good experience, a screen reader needs more contextual information than just the text.
  - masklinn 5y ago
    > In my opinion, accessibility should be implemented at the OS level. Which it is. Apple has been praised for the accessibility of both macOS and iOS for years if not decades, accessibility concerns have been baked into Cocoa forever. And accessibility is literally a top level category of the Settings app. And I know that Windows has been improving by leaps and bounds. > The OS can, for example, run OCR on the entire screen and turn bitmaps into selectable text, read text out loud, etc That exists (look up VoiceOver Recognition). However it can not be reliable, and will never be anywhere near as good as actual semantic annotation. Image recognition has no way to understand that the physical UI layout has no relation to its logical setup, nor does it have any way to differentiate between semantic and decorative content, or to see through invisibility to know that a button is a menu versus an action.
  - p_l 5y ago
    OSes do provide accessibility APIs - and usually if you use one of the more supported graphic toolkits there are somewhat easy ways to integrate directly in semantic ways, without needing to OCR and do fortune telling. Usually it means that you need a deferred mode semantic tree of the application for accessibility UI to walk through, though.
- slimsag 5y ago
  If we want more accessible UIs, we need to demand better standard accessibility APIs from OS vendors Microsoft and Apple. Unity has no official support for screenreaders or colorblind modes. Unreal has both. If an major game engine developer like Unity cannot be bothered to add support, can we really expect the little developer hacking away on some OpenGL side project to? Most people working with OpenGL use GLFW (or sometimes SDL), neither have any cross-platform API for supporting screen readers. Why? Because not only do Windows and MacOS have different APIs, but different screen readers, braille displays, etc. have different APIs. But okay - let's look at the web. It's the most accessible platform of them all, right? What have Google/Apple/Mozilla done to support developers adding screenreader support from WebGL and Canvas-based applications? -> not a whole lot. You need to inject text into a hidden div, which also has performance implications so you need to build a UI to toggle it on/off or detect accessibility the way Flutter Web does. We should have better accessibility, but as long as we expect developers to run through several hours/days of hoops to get even something basic working - it's just not going to happen, and that is mostly the fault of extremely poor platform APIs.
  - iudqnolq 5y ago
    > Unity has no official support for screenreaders or colorblind modes. Unreal has both. If an major game engine developer like Unity cannot be bothered to add support, can we really expect the little developer hacking away on some OpenGL side project to? Games are special because the means of interaction is often a core part of the experience. An accessibility mode for a game can be a huge amount of work. Not to say that some people don't do that huge amount of work (which is great) but games lacking accessibility is no excuse to use game-like development practices to write apps that could otherwise be relatively easily made accessibile.
  - mwcampbell 5y ago
    > extremely poor platform APIs As a former member of the Windows accessibility team at Microsoft, I'd appreciate your thoughts on what makes the platform accessibility APIs extremely poor. I have my own thoughts on what makes them difficult to implement, but I'd like to hear your perspective first.
Ennea 5y ago
As someone who occasionally writes tiny tools with some GUI, how can I test how well they work with accessibility tools?
- supercheetah 5y ago
  On Windows, test it with NVDA[1] or JAWS[2]. On Linux, in Gnome, there's a screen reader that can be enabled in the settings under accessibility. I'm not sure what there is for Mac, Android, or iOS offhand. Also, test in high contrast modes, and screen magnifiers. 1. https://www.nvaccess.org/ https://www.nvaccess.org/ 2. https://www.freedomscientific.com/products/software/jaws/ https://www.freedomscientific.com/products/software/jaws/
  - Ennea 5y ago
    Thank you, that's very helpful! I'll be saving these for later :)
  - mwcampbell 5y ago
    To expand on this: Windows also has the Narrator screen reader built in, and in Windows 10, Narrator is a decent screen reader IMO. (Disclosure: I used to be on the Windows accessibility team at Microsoft, where I worked on Narrator.) To enable Narrator, press Ctrl+Win+Enter. Narrator has a built-in tutorial (not written by me) to help you get started. Mac and iOS has VoiceOver built in. To enable it on Mac, press Command+F5. On iOS, you may be able to triple-press the home button, on devices that have one. Failing that, you can find it in Settings. Android has TalkBack built in. You can find it somewhere in Settings; the specifics are device-dependent.
- gjvnq 5y ago
  Hacky/Unixy solution: Make a CLI version.
tapirl 5y ago
It is very helpful for the developers if you could provide more info, such as error messages.
iamgopal 5y ago
Ultimately browser as a gui, ( even without JavaScript ) will win, no other gui efforts are at par with that. So a WebKit with api to plug any language backend would be a good idea ( electron without js ). Is there any effort in that direction ?
mtzet 5y ago
This gets brought up pretty much every time immediate mode GUIs are discussed on HN. I think your comment is reasonable, and I don't disagree, but I also think it's important to not throw the baby out with the bathwater and dismiss immediate mode GUIs for accessibility reasons. It's not at all clear to me that the right approach is to have both the accessibility interface and the GUI be generated from the same components. It seems to me that it's just approaching the lowest common denominator of a passable GUI and a passable accessibility interface. The traditional widget-style GUI paradigm asks you to essentially copy all your application data into their format and to keep this format in sync with your own data. This is tedious, and I believe hinders the creation of better GUIs. On the flip-side, owning your data structure means it can indeed hook into the accessibility API for you. Immediate mode GUIs provide an alternative approach, where they only handle rendering stuff on screen. Since all the data structures are now properly controlled by the programmer, it is much easier to create dynamic GUIs. Taking a step back like this provides an opportunity to further the state of the art. The solution I see to providing accessibility in immediate mode GUIs is that the programmer has to hook into the accessibility API themselves -- if it makes sense. It's not clear to me that certain advanced GUI programs can ever be very accessible. Yes, this is more work, but there's also an opportunity to create richer experiences for people with disabilities.
- mwcampbell 5y ago
  > This gets brought up pretty much every time immediate mode GUIs are discussed on HN. I think your comment is reasonable, and I don't disagree, but I also think it's important to not throw the baby out with the bathwater and dismiss immediate mode GUIs for accessibility reasons. I can only talk about my own motivation for repeatedly bringing this up. If a developer is unaware that they're choosing an inaccessible GUI toolkit for their application, and that application is then required for a particular job, then that developer may end up unwittingly preventing some people from doing that job. And this isn't just hypothetical for me; I know of a blind person who lost his job (luckily only temporarily) because of an inaccessible application. So I think it's entirely reasonable for an application developer to dismiss a GUI toolkit because it's inaccessible. I wish more developers would research this on their own before choosing a toolkit for their applications. But since many don't, I feel obligated to draw attention to it. I'm glad I'm not the only one. As for your suggestion that applications should implement the platform accessibility APIs themselves: First, it's impractical to expect application-level code to do this directly. The Windows accessibility APIs are notoriously hard to implement correctly. And my understanding is that the Mac, iOS, and Android accessibility APIs are only easy to implement if the application is written in the platform's native programming language. Now, these problems could be solved with an open-source wrapper library -- the SDL or GLFW of accessibility. And I'm planning to implement such a library myself sometime. But my plan is for that library to be used by toolkits, not applications. Because if every application has to directly implement accessibility in parallel with its GUI, then it's a safe bet that even fewer applications will be accessible. We want to get to a place where as many applications as possible can be accessible by default, with minimal effort on the part of the application developers.
tenaciousDaniel 5y ago
As a front-end web developer who's looking to get into lower-level UI tech, I'm really curious to learn more about this. So novice question - what's the connection between immediate mode/openGL implementations and accessibility?
- mwcampbell 5y ago
  There's no reason in principle why an OpenGL-based GUI must be inaccessible. Qt heavily uses OpenGL, at least on some platforms, but is still more or less accessible. It's more accurate to say that the vast majority of GUI toolkits that have ever been written are inaccessible, and most OpenGL-based toolkits fall into this category. Accessibility is tedious to implement at the toolkit level, especially in a cross-platform toolkit, so generally only big toolkits with corporate backing get it. Also, on the web platform, if you abandon semantic HTML and use canvas or WebGL, then you're forfeiting the accessibility that your UI would normally have by default. As far as I know, the only way to make a canvas or WebGL-based UI accessible is to construct a parallel HTML DOM tree (and presumably use CSS to hide it behind the canvas). Thanks for being curious about this.
twobitshifter 5y ago
https://news.ycombinator.com/item?id=26212787 https://news.ycombinator.com/item?id=26212787 The same valid comment was made when nuklear was discussed a few months back. Many would love to unshackle themselves from the limitations of working with UI toolkits and it’s increasingly difficult/impossible to fulfill all the requirements we place on software, especially for solo developers. I don’t know how many one off tools I’ve written where the only interface is the command line over the years. It would be nice to have a simple step up from that as offered by Imgui tools.

ClumsyPilot 5y ago

This is the one real (and legal) reason why people should actually use OS UI. To replicate this functionality is a tall order. The 'looks native' thing is mostly noise

pqb 5y ago

This, hundreds times this. I can't say how many years in a row I was experimenting with various GUI toolkits, libraries and concepts of an application like rxi/lite editor [0]. While I saw many good looking implementations but the very first what caught my attention was lack of any kind support to screen-reading software or problem with displaying right-click context menus. I even remember previous HN discussion when I pointed out few projects that could provide some screen-readers support [1]. Sadly, I would say 95% of all GUIs I know do not care about the accessibility. From the positive side, I am glad more and more hackers on HN focus on that matter. Also worth noting big players on the market really do take that point seriously, e.g. Flutter has a Semantics tag [2]. [0]: https://github.com/rxi/lite https://github.com/rxi/lite [1]: https://news.ycombinator.com/item?id=26223380 https://news.ycombinator.com/item?id=26223380 [2]: https://api.flutter.dev/flutter/widgets/Semantics-class.html https://api.flutter.dev/flutter/widgets/Semantics-class.html

amelius 5y ago
In my opinion, accessibility should be implemented at the OS level. The OS can, for example, run OCR on the entire screen and turn bitmaps into selectable text, read text out loud, etc. This kind of functionality shouldn't be replicated on a per-app basis. Accessibility of browsers is usually considered to be very good, but browsers are almost OSes, so why not lift this one more level into the OS?
- zxzax 5y ago
  That is not really going to work, at all. To have a good experience, a screen reader needs more contextual information than just the text.
- masklinn 5y ago
  > In my opinion, accessibility should be implemented at the OS level. Which it is. Apple has been praised for the accessibility of both macOS and iOS for years if not decades, accessibility concerns have been baked into Cocoa forever. And accessibility is literally a top level category of the Settings app. And I know that Windows has been improving by leaps and bounds. > The OS can, for example, run OCR on the entire screen and turn bitmaps into selectable text, read text out loud, etc That exists (look up VoiceOver Recognition). However it can not be reliable, and will never be anywhere near as good as actual semantic annotation. Image recognition has no way to understand that the physical UI layout has no relation to its logical setup, nor does it have any way to differentiate between semantic and decorative content, or to see through invisibility to know that a button is a menu versus an action.
- p_l 5y ago
  OSes do provide accessibility APIs - and usually if you use one of the more supported graphic toolkits there are somewhat easy ways to integrate directly in semantic ways, without needing to OCR and do fortune telling. Usually it means that you need a deferred mode semantic tree of the application for accessibility UI to walk through, though.
slimsag 5y ago
If we want more accessible UIs, we need to demand better standard accessibility APIs from OS vendors Microsoft and Apple. Unity has no official support for screenreaders or colorblind modes. Unreal has both. If an major game engine developer like Unity cannot be bothered to add support, can we really expect the little developer hacking away on some OpenGL side project to? Most people working with OpenGL use GLFW (or sometimes SDL), neither have any cross-platform API for supporting screen readers. Why? Because not only do Windows and MacOS have different APIs, but different screen readers, braille displays, etc. have different APIs. But okay - let's look at the web. It's the most accessible platform of them all, right? What have Google/Apple/Mozilla done to support developers adding screenreader support from WebGL and Canvas-based applications? -> not a whole lot. You need to inject text into a hidden div, which also has performance implications so you need to build a UI to toggle it on/off or detect accessibility the way Flutter Web does. We should have better accessibility, but as long as we expect developers to run through several hours/days of hoops to get even something basic working - it's just not going to happen, and that is mostly the fault of extremely poor platform APIs.
- iudqnolq 5y ago
  > Unity has no official support for screenreaders or colorblind modes. Unreal has both. If an major game engine developer like Unity cannot be bothered to add support, can we really expect the little developer hacking away on some OpenGL side project to? Games are special because the means of interaction is often a core part of the experience. An accessibility mode for a game can be a huge amount of work. Not to say that some people don't do that huge amount of work (which is great) but games lacking accessibility is no excuse to use game-like development practices to write apps that could otherwise be relatively easily made accessibile.
- mwcampbell 5y ago
  > extremely poor platform APIs As a former member of the Windows accessibility team at Microsoft, I'd appreciate your thoughts on what makes the platform accessibility APIs extremely poor. I have my own thoughts on what makes them difficult to implement, but I'd like to hear your perspective first.

Ennea 5y ago

As someone who occasionally writes tiny tools with some GUI, how can I test how well they work with accessibility tools?

supercheetah 5y ago
On Windows, test it with NVDA[1] or JAWS[2]. On Linux, in Gnome, there's a screen reader that can be enabled in the settings under accessibility. I'm not sure what there is for Mac, Android, or iOS offhand. Also, test in high contrast modes, and screen magnifiers. 1. https://www.nvaccess.org/ https://www.nvaccess.org/ 2. https://www.freedomscientific.com/products/software/jaws/ https://www.freedomscientific.com/products/software/jaws/
- Ennea 5y ago
  Thank you, that's very helpful! I'll be saving these for later :)
- mwcampbell 5y ago
  To expand on this: Windows also has the Narrator screen reader built in, and in Windows 10, Narrator is a decent screen reader IMO. (Disclosure: I used to be on the Windows accessibility team at Microsoft, where I worked on Narrator.) To enable Narrator, press Ctrl+Win+Enter. Narrator has a built-in tutorial (not written by me) to help you get started. Mac and iOS has VoiceOver built in. To enable it on Mac, press Command+F5. On iOS, you may be able to triple-press the home button, on devices that have one. Failing that, you can find it in Settings. Android has TalkBack built in. You can find it somewhere in Settings; the specifics are device-dependent.
gjvnq 5y ago
Hacky/Unixy solution: Make a CLI version.

tapirl 5y ago

It is very helpful for the developers if you could provide more info, such as error messages.

iamgopal 5y ago

Ultimately browser as a gui, ( even without JavaScript ) will win, no other gui efforts are at par with that. So a WebKit with api to plug any language backend would be a good idea ( electron without js ). Is there any effort in that direction ?

mtzet 5y ago

This gets brought up pretty much every time immediate mode GUIs are discussed on HN. I think your comment is reasonable, and I don't disagree, but I also think it's important to not throw the baby out with the bathwater and dismiss immediate mode GUIs for accessibility reasons. It's not at all clear to me that the right approach is to have both the accessibility interface and the GUI be generated from the same components. It seems to me that it's just approaching the lowest common denominator of a passable GUI and a passable accessibility interface. The traditional widget-style GUI paradigm asks you to essentially copy all your application data into their format and to keep this format in sync with your own data. This is tedious, and I believe hinders the creation of better GUIs. On the flip-side, owning your data structure means it can indeed hook into the accessibility API for you. Immediate mode GUIs provide an alternative approach, where they only handle rendering stuff on screen. Since all the data structures are now properly controlled by the programmer, it is much easier to create dynamic GUIs. Taking a step back like this provides an opportunity to further the state of the art. The solution I see to providing accessibility in immediate mode GUIs is that the programmer has to hook into the accessibility API themselves -- if it makes sense. It's not clear to me that certain advanced GUI programs can ever be very accessible. Yes, this is more work, but there's also an opportunity to create richer experiences for people with disabilities.

mwcampbell 5y ago
> This gets brought up pretty much every time immediate mode GUIs are discussed on HN. I think your comment is reasonable, and I don't disagree, but I also think it's important to not throw the baby out with the bathwater and dismiss immediate mode GUIs for accessibility reasons. I can only talk about my own motivation for repeatedly bringing this up. If a developer is unaware that they're choosing an inaccessible GUI toolkit for their application, and that application is then required for a particular job, then that developer may end up unwittingly preventing some people from doing that job. And this isn't just hypothetical for me; I know of a blind person who lost his job (luckily only temporarily) because of an inaccessible application. So I think it's entirely reasonable for an application developer to dismiss a GUI toolkit because it's inaccessible. I wish more developers would research this on their own before choosing a toolkit for their applications. But since many don't, I feel obligated to draw attention to it. I'm glad I'm not the only one. As for your suggestion that applications should implement the platform accessibility APIs themselves: First, it's impractical to expect application-level code to do this directly. The Windows accessibility APIs are notoriously hard to implement correctly. And my understanding is that the Mac, iOS, and Android accessibility APIs are only easy to implement if the application is written in the platform's native programming language. Now, these problems could be solved with an open-source wrapper library -- the SDL or GLFW of accessibility. And I'm planning to implement such a library myself sometime. But my plan is for that library to be used by toolkits, not applications. Because if every application has to directly implement accessibility in parallel with its GUI, then it's a safe bet that even fewer applications will be accessible. We want to get to a place where as many applications as possible can be accessible by default, with minimal effort on the part of the application developers.

tenaciousDaniel 5y ago

As a front-end web developer who's looking to get into lower-level UI tech, I'm really curious to learn more about this. So novice question - what's the connection between immediate mode/openGL implementations and accessibility?

mwcampbell 5y ago
There's no reason in principle why an OpenGL-based GUI must be inaccessible. Qt heavily uses OpenGL, at least on some platforms, but is still more or less accessible. It's more accurate to say that the vast majority of GUI toolkits that have ever been written are inaccessible, and most OpenGL-based toolkits fall into this category. Accessibility is tedious to implement at the toolkit level, especially in a cross-platform toolkit, so generally only big toolkits with corporate backing get it. Also, on the web platform, if you abandon semantic HTML and use canvas or WebGL, then you're forfeiting the accessibility that your UI would normally have by default. As far as I know, the only way to make a canvas or WebGL-based UI accessible is to construct a parallel HTML DOM tree (and presumably use CSS to hide it behind the canvas). Thanks for being curious about this.

twobitshifter 5y ago

https://news.ycombinator.com/item?id=26212787 https://news.ycombinator.com/item?id=26212787 The same valid comment was made when nuklear was discussed a few months back. Many would love to unshackle themselves from the limitations of working with UI toolkits and it’s increasingly difficult/impossible to fulfill all the requirements we place on software, especially for solo developers. I don’t know how many one off tools I’ve written where the only interface is the command line over the years. It would be nice to have a simple step up from that as offered by Imgui tools.