5 ms·

Egui commit: Implement accessibility APIs via AccessKit

96 points by mwcampbell 4y ago

mwcampbell 4y ago
This isn't published to crates.io yet, but I'm excited to share this work I've been doing. This makes egui, as far as I know, the first pure-Rust GUI toolkit to implement platform-native accessibility APIs, and possibly the first immediate-mode GUI to do so as well (albeit with an intermediate retained tree).
- ogoffart 4y ago
  Slint has had accessibility support before. (But it was arguably done by calling into a C++ library, so technically not pure rust.) https://github.com/slint-ui/slint/pull/1294 https://github.com/slint-ui/slint/pull/1294
  - mwcampbell 4y ago
    Ah yes, I didn't mean to dismiss the work your team has already done. Just got over-excited, I guess. Anyway, thanks for the work you've already done on accessibility. It should now be straightforward to connect Slint's existing accessibility support to AccessKit, if you wish.
raphlinus 4y ago
I'm really happy to see this work, and am not even mad that the first Rust GUI toolkit to get the integration is egui. We intend to integrate AccessKit deeply in the UI work we're undertaking (the Xilem project, which is essentially a pivot to address the shortcomings in Druid). Disclosure: Google Fonts funded Matt's work and I was involved in the decision to do so.
- treuks 4y ago
  Is Xilem going to be an independent UI toolkit, or is it going to be a "modification" of Druid?
  - raphlinus 4y ago
    It's a new repo (and currently early days still) but we expect to build up the widget collection by doing a lot of adapting of the existing Druid widgets with generally minimal adaptation.
    - treuks 4y ago
      Sounds good to me! Thanks!
alice-i-cecile 4y ago
AccessKit looks great, and I'm thrilled to see a working integration for it. Egui is an awesome tool for building quick-and-dirty GUIs: we use it at work for our game's developer tools.
juancampa 4y ago
Egui keeps getting better and better! For those unfamiliar you can run the web demo on your phone or desktop: https://egui.rs https://egui.rs I’ve personally built relatively complex UIs with egui. Both a Chrome extension and a VSCode extension for https://membrane.io https://membrane.io and couldn’t be happier with it.
- mwcampbell 4y ago
  Do note that this AccessKit integration doesn't yet help with the web backend of egui, only the winit-based backend on Windows and macOS. A web platform adapter for AccessKit is planned, but not a top priority.
  - juancampa 4y ago
    Good to know. Thanks for clarifying!
- chrismorgan 4y ago
  egui’s web implementation is purely canvas with no DOM (apart from how it handles text input), so for now it’s completely invisible to screen readers. But even once AccessKit supports the web (and even pretending it wouldn’t be a performance sink that undermines half of the purpose of the canvas rendering approach in web apps), it will still be heavily inferior to something done with DOM for almost all types of content. Some examples: text rendering is awful on some common browser configurations (mine included—Firefox on Linux/Wayland, rendered at 2× scaling and downsampled to 1.5×; the text is extraordinarily fuzzy, and it’s rendering in its own font because I suppose it’s bundling a complete font, shaper and renderer—since I force my own font selections so only the generic font-families work, and I also block all web fonts at fetch time); the font rendering technique means no fallback which means things like emoji would only render if they provided a font that supported that (at present they don’t, and it’s just a nasty technique all round, you want to be able to use local fonts); links are fake, so various interactions can never work correctly (e.g. Ctrl+click and middle-click should mostly open a background tab, and right-clicking should give you the normal context menu); scrolling can’t be done properly and will thus inevitably be extremely obnoxious on most devices; mobile-style pinch zoom cannot be supported. ⸻And I’m just mentioning some of the unfixable problems with this approach; there are more things that are wonky or painful apart from that: things that could be fixed, but where it’s not surprising that they’re broken because that’s a common problem when you throw away almost all of the stuff the platform gives you and try to implement it all from scratch. I very strongly recommend against using this general approach at all on the web. The web platform just doesn’t provide the primitives required to make it anything other than miserable. (I’m not speaking against egui for platforms other than web, only for web. The fact of the matter is that functionally you’re fine building a native app with native tech, a native app with web tech, or a web app with web tech, but that building a web app with native tech is a terrible idea that will always be inferior, in important, unfixable ways.)
mwcampbell 4y ago
And now a quick demo video, on Windows using Narrator: https://mwcampbell.us/video/egui-accesskit-win-demo-2022-12-04.mp4 https://mwcampbell.us/video/egui-accesskit-win-demo-2022-12-...
- modernerd 4y ago
  Impressive progress, thank you so much for your work on this. For the live area, do desktop APIs support something like the HTML 'role' attribute? https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Live_Regions#roles_with_implicit_live_region_attributes https://developer.mozilla.org/en-US/docs/Web/Accessibility/A... As an app developer it would be helpful to be able to denote the live region in your example a 'status' (the user can read it but it's not read automatically on change). And there may be other times when the same area should be an 'alert' (changes trigger 'assertive' messages to inform the user that their change results in an error or issue).
Grimburger 4y ago
I really like egui, having always viewed immediate mode as only for games or small basic programs was blown away how little resources it actually consumes even for some fairly complicated gui's. Coming from a retained mode world it's so much easier to read, write and reason about. Don't find the wasm/web compilation as useful though, it does seem to hit performance quite badly, not sure if that's the nature of the beast in general or something else to do with optimisations for the library.
harriet1 4y ago
[dead]
ushakov 4y ago
How do i write a long commit message like that?
- palotasb 4y ago
  Don't pass the -m argument to git commit. It will open you $EDITOR an you can edit the commit message there.
- mwcampbell 4y ago
  This particular commit was a squash merge of a GitHub pull request. So GitHub generated the commit message from the commits on the branch that was merged.
astrange 4y ago
Funny name. Makes sense, but egui means something like “harsh, bitter, nasty” in Japanese.
api 4y ago
Game changing. EGUI just graduated to “able to be considered for serious projects.”
FloatArtifact 4y ago
Excited to see AccessKit. Finally a performant backend among other things for screen readers and voice control. I'm curious to see how subscribing to events and watching for changes in specific controls will be defined. In addition, python bindings would make it much more accessible as the code base solidifies towards production.
- mwcampbell 4y ago
  The way that screen readers and other assistive technologies subscribe to events is defined by the platform accessibility APIs. AccessKit simply sends the right events to whatever clients are listening. Note, though, that on both Windows and macOS, AccessKit doesn't send any events until the first client makes an accessibility request. I want to provide Python bindings at some point. That would enable Kivy to implement accessibility, for example. Just haven't gotten to it yet. If anyone wants to step up and work on that, let me know.

96 points by mwcampbell 4y ago

This isn't published to crates.io yet, but I'm excited to share this work I've been doing. This makes egui, as far as I know, the first pure-Rust GUI toolkit to implement platform-native accessibility APIs, and possibly the first immediate-mode GUI to do so as well (albeit with an intermediate retained tree).

ogoffart 4y ago
Slint has had accessibility support before. (But it was arguably done by calling into a C++ library, so technically not pure rust.) https://github.com/slint-ui/slint/pull/1294 https://github.com/slint-ui/slint/pull/1294
- mwcampbell 4y ago
  Ah yes, I didn't mean to dismiss the work your team has already done. Just got over-excited, I guess. Anyway, thanks for the work you've already done on accessibility. It should now be straightforward to connect Slint's existing accessibility support to AccessKit, if you wish.

raphlinus 4y ago

I'm really happy to see this work, and am not even mad that the first Rust GUI toolkit to get the integration is egui. We intend to integrate AccessKit deeply in the UI work we're undertaking (the Xilem project, which is essentially a pivot to address the shortcomings in Druid). Disclosure: Google Fonts funded Matt's work and I was involved in the decision to do so.

treuks 4y ago
Is Xilem going to be an independent UI toolkit, or is it going to be a "modification" of Druid?
- raphlinus 4y ago
  It's a new repo (and currently early days still) but we expect to build up the widget collection by doing a lot of adapting of the existing Druid widgets with generally minimal adaptation.
  - treuks 4y ago
    Sounds good to me! Thanks!

alice-i-cecile 4y ago

AccessKit looks great, and I'm thrilled to see a working integration for it. Egui is an awesome tool for building quick-and-dirty GUIs: we use it at work for our game's developer tools.

juancampa 4y ago

Egui keeps getting better and better! For those unfamiliar you can run the web demo on your phone or desktop: https://egui.rs https://egui.rs I’ve personally built relatively complex UIs with egui. Both a Chrome extension and a VSCode extension for https://membrane.io https://membrane.io and couldn’t be happier with it.

mwcampbell 4y ago
Do note that this AccessKit integration doesn't yet help with the web backend of egui, only the winit-based backend on Windows and macOS. A web platform adapter for AccessKit is planned, but not a top priority.
- juancampa 4y ago
  Good to know. Thanks for clarifying!
chrismorgan 4y ago
egui’s web implementation is purely canvas with no DOM (apart from how it handles text input), so for now it’s completely invisible to screen readers. But even once AccessKit supports the web (and even pretending it wouldn’t be a performance sink that undermines half of the purpose of the canvas rendering approach in web apps), it will still be heavily inferior to something done with DOM for almost all types of content. Some examples: text rendering is awful on some common browser configurations (mine included—Firefox on Linux/Wayland, rendered at 2× scaling and downsampled to 1.5×; the text is extraordinarily fuzzy, and it’s rendering in its own font because I suppose it’s bundling a complete font, shaper and renderer—since I force my own font selections so only the generic font-families work, and I also block all web fonts at fetch time); the font rendering technique means no fallback which means things like emoji would only render if they provided a font that supported that (at present they don’t, and it’s just a nasty technique all round, you want to be able to use local fonts); links are fake, so various interactions can never work correctly (e.g. Ctrl+click and middle-click should mostly open a background tab, and right-clicking should give you the normal context menu); scrolling can’t be done properly and will thus inevitably be extremely obnoxious on most devices; mobile-style pinch zoom cannot be supported. ⸻And I’m just mentioning some of the unfixable problems with this approach; there are more things that are wonky or painful apart from that: things that could be fixed, but where it’s not surprising that they’re broken because that’s a common problem when you throw away almost all of the stuff the platform gives you and try to implement it all from scratch. I very strongly recommend against using this general approach at all on the web. The web platform just doesn’t provide the primitives required to make it anything other than miserable. (I’m not speaking against egui for platforms other than web, only for web. The fact of the matter is that functionally you’re fine building a native app with native tech, a native app with web tech, or a web app with web tech, but that building a web app with native tech is a terrible idea that will always be inferior, in important, unfixable ways.)

mwcampbell 4y ago

And now a quick demo video, on Windows using Narrator: https://mwcampbell.us/video/egui-accesskit-win-demo-2022-12-04.mp4 https://mwcampbell.us/video/egui-accesskit-win-demo-2022-12-...

modernerd 4y ago
Impressive progress, thank you so much for your work on this. For the live area, do desktop APIs support something like the HTML 'role' attribute? https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Live_Regions#roles_with_implicit_live_region_attributes https://developer.mozilla.org/en-US/docs/Web/Accessibility/A... As an app developer it would be helpful to be able to denote the live region in your example a 'status' (the user can read it but it's not read automatically on change). And there may be other times when the same area should be an 'alert' (changes trigger 'assertive' messages to inform the user that their change results in an error or issue).

Grimburger 4y ago

I really like egui, having always viewed immediate mode as only for games or small basic programs was blown away how little resources it actually consumes even for some fairly complicated gui's. Coming from a retained mode world it's so much easier to read, write and reason about. Don't find the wasm/web compilation as useful though, it does seem to hit performance quite badly, not sure if that's the nature of the beast in general or something else to do with optimisations for the library.

harriet1 4y ago

[dead]

ushakov 4y ago

How do i write a long commit message like that?

palotasb 4y ago
Don't pass the -m argument to git commit. It will open you $EDITOR an you can edit the commit message there.
mwcampbell 4y ago
This particular commit was a squash merge of a GitHub pull request. So GitHub generated the commit message from the commits on the branch that was merged.

astrange 4y ago

Funny name. Makes sense, but egui means something like “harsh, bitter, nasty” in Japanese.

api 4y ago

Game changing. EGUI just graduated to “able to be considered for serious projects.”

FloatArtifact 4y ago

Excited to see AccessKit. Finally a performant backend among other things for screen readers and voice control. I'm curious to see how subscribing to events and watching for changes in specific controls will be defined. In addition, python bindings would make it much more accessible as the code base solidifies towards production.

mwcampbell 4y ago
The way that screen readers and other assistive technologies subscribe to events is defined by the platform accessibility APIs. AccessKit simply sends the right events to whatever clients are listening. Note, though, that on both Windows and macOS, AccessKit doesn't send any events until the first client makes an accessibility request. I want to provide Python bindings at some point. That would enable Kivy to implement accessibility, for example. Just haven't gotten to it yet. If anyone wants to step up and work on that, let me know.