This is still somewhat far from what I was hoping to see, but nonetheless a great inspiration and a good start. I worked on an a small Firefox plugin whereby w

This is still somewhat far from what I was hoping to see, but nonetheless a great inspiration and a good start. I worked on an a small Firefox plugin whereby would enable visually impaired users to use firefox and interact with websites using voice. It was a small attempt and it was not an easy task... The two biggest challenges were (1) understand the semantics of the website and (2) interacting with browser. For challenge one, we can look at the case whereby a div/span/css clickable button is used instead of using the <button> tag. Traditional reader has tried its best to figure out how to find these buttons, but nonetheless it still posts a challenge. “alt” attribute is missing in <imh> tag so no description when image goes 404. So wouldn’t it be awesome if we can be more responsible and also provide a standardized markup API so screen readers can use? So here comes ARIA (Accessible Rich Internet Application) [1]. But even so a lot pf popular websites are not compliant woth ARIA. Furthermore, dynamic contents, single page application, and javascript makes screen reader more difficult to use. As part of the thesis, I worked woth a few visually impaired users. Whether screen readers have evolved I do not know, so I’s love to hear feedback from those on HN. Since this project is intended to run in a browser, so the native choicenis an add-on (which mostly means Javascript). Furtheremore, to protect privacy, I want to do all the speech recognition locally, without leaving the user’s local computer. So I tackled on Gmail and my college’s website first. Parsing Gmail’s DOM was proven to be very difficult, but fortunately, Gmail.js [2] exists - I owe the author so much (although I did help fix a bug I think). Next comes the interaction. The idea is 1. user says to the computer “find the latest email” 2. the computer repeats back to the user 3. a small wait time is given to user a chance to correct the commmand before too late. 4.when time expires the code executes the command Because there is no AI and because we neednto consider states, the best choice (esp for such small experiement), I created a finite state machine. However, the number of states have exploded as expected for just 5-6 (??) commands for single website; clearly neither scalable nOr maintainable. It is unacceptable to limit a user to just a couple actions right? What would be awesome is if every website can expose APIs - not just any API, but follows standard. To start small, we will take sitemap.xml as an inspiration. There should be a manifest file to describe how to login, what are needed to login, is there a “About us” page, how to do search, how to post something and etc. This manifest file is generated on demand and is dynamic since we cannot write out 10,000 different interactions on Facebook righr? Then screen reader or my plugin can read the manifest file, and let user tell us what he/she desires to do, and we look up the manifest on how to make the call. Toapi seems to be heading a similar direction, which is great. The second challenge addon can only do so much. I did have to use the more low-level APIs to do certain thing, and if I remembered correctly service workers weren’t exactly easy back in 2013. Content script has limited capability. Also, there was no speech recognition API in Firefox in 2013, so I turned to speak.js [3] which was the only one that worked offline and has no extra dependencies like nodeJS. There is some support for Web Speech API in Firefox which is encouraging [4]. The take aways are: - we suck at designing a usable website that respects semantics (I am guilty of that on my own silly homepage) - those don’t use ARIA should try. ARIA does not solve everything but it gives a good start - DOM is a horrible place to use to deduce and find contrnts, and a nightmare to interact with - addon has restricted capability for good security reason so not a great idea to implement a complex software - javascript can hinder screen reader from doing its job - honestlt, please stop single page application and please stop hijacking the god damn back button - we hate ads espeically those overlay and popup ads (they cannot be fully blocked even with Chrome’s new feature). But also consider we’d have to unblock ads to load some of the popular websites (ironically HN search will not work on mobile unless Focus is disabled) - finally, would be awesome if there is a way for machine to consime a website or a webpage’s functionality 1. https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA https://developer.mozilla.org/en-US/docs/Web/Accessibility/A... 2. https://github.com/KartikTalwar/gmail.js/tree/master https://github.com/KartikTalwar/gmail.js/tree/master 3. https://github.com/kripken/speak.js/ https://github.com/kripken/speak.js/ 4. https://hacks.mozilla.org/2016/01/firefox-and-the-web-speech-api/ https://hacks.mozilla.org/2016/01/firefox-and-the-web-speech...