4 ms·

The challenges that remain in speech coding is not generating code as much as it is navigating through existing code or an application. There's only two ways t

The challenges that remain in speech coding is not generating code as much as it is navigating through existing code or an application. There's only two ways to do this effectively and unfortunately no one has taken the true path to accessibility. The more common way is plugins/extensions to grab a information from the editor. Accessibility is more than just one editor. It's the OS and all the applications. Microsoft needs to take the hard route to make an accessibility UI automation server to grab that information and only make up the difference through plugins as needed. It's all about grabbing information from the application and generating on the fly commands, not just parsing free dictation in order to get the best accuracy. It takes a lot of expertise to make any sort of UI automation, fast and efficient for navigating and selecting text or out of focus menu items. I've fussed around and managed to get tree sitter to navigate across code. For example generic commands are like 'next function'. Code simply isn't pronounceable when it's written by others. Therefore, navigating across generic tokens is really the best method. Then other methods can be used for fine navigation if needed. My hope is that they develop a grammar system that is open source and integrates with accessibility frameworks focused on performance. I wish I could have a phone call with the development team.

skydhash 4y ago
I think an accessibility a la vim or with something like tree sitter, would help immensely like: “Top of file Down 5 lines Modify import source to … inside first class Down 5 methods Insert new method after Inside arg list Append arg named … of type … … …” And add a way to indentify types and parameters with special pronunciation.
- FloatArtifact 4y ago
  I recognize that all applications are not accessible through accessibility APIs. However, there is no high level access to accessibility APIs. There are quite a few for automated testing UI. However, none of them are performant enough for speech to code or screen readers. Testing automation frameworks don't really require high performance. Accessibility accessing the content of the application and the context is what's important. It's more important than the speech recognition backend. Speech recognition shines work best with a narrow context. (when those commands are available) The type of performance we need as a speech recognition community and screen reader community is quite high. By the beginning of speech and just before decode time information needs to be available to be parsed for navigation/editing. That way these tokens can be weighted as commands for recognition. Commands could be modeled after vim functionality though. Outside of tree sitter it would be interesting to hook into hooking into as a client a language protocol server. However, I think they only expect one client. In addition, I still see that as a lesser approach without dedicated support for high performance UI automation server for speech recognition engine to leverage.
- FloatArtifact 4y ago
  Yes, minimizing number of command and specificity as much as possible for navigation by understanding the context of where the user is optimizes the user's time in navigation. Imagine even more precise commands 'next function' followed by a letter. That allows you to navigate to only a function with that letter defined. Really the possibilities are endless when we have complete context of the screen and the structure of the code. Someday I hope for the release of something like stable diffusion for voice coding. An open complete pipeline that users can illiterate fast and innovate!

skydhash 4y ago

I think an accessibility a la vim or with something like tree sitter, would help immensely like: “Top of file Down 5 lines Modify import source to … inside first class Down 5 methods Insert new method after Inside arg list Append arg named … of type … … …” And add a way to indentify types and parameters with special pronunciation.

FloatArtifact 4y ago
I recognize that all applications are not accessible through accessibility APIs. However, there is no high level access to accessibility APIs. There are quite a few for automated testing UI. However, none of them are performant enough for speech to code or screen readers. Testing automation frameworks don't really require high performance. Accessibility accessing the content of the application and the context is what's important. It's more important than the speech recognition backend. Speech recognition shines work best with a narrow context. (when those commands are available) The type of performance we need as a speech recognition community and screen reader community is quite high. By the beginning of speech and just before decode time information needs to be available to be parsed for navigation/editing. That way these tokens can be weighted as commands for recognition. Commands could be modeled after vim functionality though. Outside of tree sitter it would be interesting to hook into hooking into as a client a language protocol server. However, I think they only expect one client. In addition, I still see that as a lesser approach without dedicated support for high performance UI automation server for speech recognition engine to leverage.
FloatArtifact 4y ago
Yes, minimizing number of command and specificity as much as possible for navigation by understanding the context of where the user is optimizes the user's time in navigation. Imagine even more precise commands 'next function' followed by a letter. That allows you to navigate to only a function with that letter defined. Really the possibilities are endless when we have complete context of the screen and the structure of the code. Someday I hope for the release of something like stable diffusion for voice coding. An open complete pipeline that users can illiterate fast and innovate!