7 ms·

This article is skewed towards the use of hands-free programming interfaces for people who find typing inconvenient. Also, it appears that the tools that have b

This article is skewed towards the use of hands-free programming interfaces for people who find typing inconvenient. Also, it appears that the tools that have been designed for the voice coding part have been designed in a generic way so as to make them extensible and applicable in broader contexts. Two things could be done differently. First, voice coding could provide a far more efficient interface than typing based because the inherent difference in speed between being able to type out instructions and speak them out. All programming languages to date have been designed for typing based input. However, it should not be difficult to design one for sppech based input. In other words, it would make sense to design such interfaces for all users instead of only such users that have some problems using typing based interfaces. Second, instead of creating generic, one size fits all solutions for the voice to code interface, it would make more sense to design such interfaces on a per language and platform basis. From a coding perspective, the language and the platform or the system software you plan to use is what contributes to the palette of programming instructions. For example, Ruby on Rails. A voice interface could be designed better keeping this in mind.

lunixbochs 8y ago
I'm the creator of Talon. You're exactly right about context sensitivity, and you're wrong that I'm not designing for it. One of Talon's secret missions is to make extreme context sensitivity possible. I've done a lot of things at a low technical level to make this possible eventually, such as: - You can change the entire voice grammar from anything to anything in a couple of milliseconds. - I'm building a scope system, kinda like syntax scopes in a text editor like Sublime, but for arbitrary systemwide stuff. So you'll be able to say "this command activates on scope 'code.ruby.method' and 'filename matches test_*.rb" (which takes advantage of the fast grammar swap feature). Getting really specific with commands and action implementations means heavy lifting can be done for you in a lot of really cool cases, and recognition accuracy goes way up. It's also hard to remember all this stuff, one fix is to use floating UI that shows you the most relevant context-specific commands. As far as "generally accessible", another goal of Talon is to preempt RSI and find ways to convince people who can type fine to use it anyway. Two ways are "make many things way faster/smoother than typing" and "lots of cool eye tracking features".
- khitchdee 8y ago
  I think the scope system is a step in a useful direction. The speed at which you change a grammar is not really that relevant because typically, you don't program in more than one language. Being able to do it is. When you start with a generic system that can be configured down to specialized use-cases, sometimes, your focus on the end user's perspective of your tool gets clouded by your own desire to engineer in lots of bells and whistles since you have a system that can do that. From the end user's perspective, the specifics matter first. Sounds like you're making it easy to create the equivalent of macros to abbreviate the input of common tasks. It would be interesting to see if you could make this system function without a screen so even a blind person could produce code.
  - lunixbochs 8y ago
    I've definitely considered blind UI, it's a different sort of paradigm. It should be nice for other cases, like using a computer with just an earpiece. It turns out some of my approach will make fully blind voice UI much easier, even on existing apps / environments. You definitely don't understand the implications of highly dynamic grammars then. A couple instances where rapid grammar changes matter are: "I want to switch between my text editor, a terminal, and a browser, and have exactly the most specific and relevant commands available at all times with no delay", and "I want to have syntax-aware voice grammars that are very in tune with how my programming language and framework operate" You appear to be talking about this without any significant research, sources, or knowledge of my software (which you have multiple times made fairly confident statements about that are not remotely true), please stop generalizing about what I'm doing without some examples to back it up.
    - khitchdee 8y ago
      Re: grammar changes: I assumed each language was associated with its grammar and it was not more fine grained than that. This makes sense though in the context where you're simply trying to replicate the keyboard-window-mouse based tool design with a speech-window-eye-tracking one. I guess this makes it easy for a user on the former UI to switch over. You will note my comments were not about your software, but hands-free programming interfaces in general. They continue to be this way, you will note. I am addressing the subject of this topic at a high level. I also have a fairly good idea on where your software stands within that context. Don't need to do any research for that.
brandonjm 8y ago
> However, it should not be difficult to design one for sppech based input. Personally I feel that this shouldn't even be necessary. In theory all you need to be able to vocalise is what you want the code to achieve, then that could be mapped to something similar to a snippet based on what language you are using. Most programming languages share the same core abilities, it's just the syntax to achieve the task that matters once you know what the task is. So this shouldn't need to be handled by a dedicated programming language for voice coding, it could just be handled by a wrapper (A text editor for your voice with an advanced NLP based snippet system). Something like: "Check if A is equal to B, if it is, return A, if not, return B" would be much more efficient than saying "if parenthesis capital A equals equals captial B..." and so on.
- khitchdee 8y ago
  I agree. I think a wrapper would be a good way to go about it with the voice based programming interface left only for a high level description of what the programmer is trying to do. Sort of akin to how many of today's new high level languages are actually implmented by translating to C and then using a C complier

lunixbochs 8y ago

I'm the creator of Talon. You're exactly right about context sensitivity, and you're wrong that I'm not designing for it. One of Talon's secret missions is to make extreme context sensitivity possible. I've done a lot of things at a low technical level to make this possible eventually, such as: - You can change the entire voice grammar from anything to anything in a couple of milliseconds. - I'm building a scope system, kinda like syntax scopes in a text editor like Sublime, but for arbitrary systemwide stuff. So you'll be able to say "this command activates on scope 'code.ruby.method' and 'filename matches test_*.rb" (which takes advantage of the fast grammar swap feature). Getting really specific with commands and action implementations means heavy lifting can be done for you in a lot of really cool cases, and recognition accuracy goes way up. It's also hard to remember all this stuff, one fix is to use floating UI that shows you the most relevant context-specific commands. As far as "generally accessible", another goal of Talon is to preempt RSI and find ways to convince people who can type fine to use it anyway. Two ways are "make many things way faster/smoother than typing" and "lots of cool eye tracking features".

khitchdee 8y ago
I think the scope system is a step in a useful direction. The speed at which you change a grammar is not really that relevant because typically, you don't program in more than one language. Being able to do it is. When you start with a generic system that can be configured down to specialized use-cases, sometimes, your focus on the end user's perspective of your tool gets clouded by your own desire to engineer in lots of bells and whistles since you have a system that can do that. From the end user's perspective, the specifics matter first. Sounds like you're making it easy to create the equivalent of macros to abbreviate the input of common tasks. It would be interesting to see if you could make this system function without a screen so even a blind person could produce code.
- lunixbochs 8y ago
  I've definitely considered blind UI, it's a different sort of paradigm. It should be nice for other cases, like using a computer with just an earpiece. It turns out some of my approach will make fully blind voice UI much easier, even on existing apps / environments. You definitely don't understand the implications of highly dynamic grammars then. A couple instances where rapid grammar changes matter are: "I want to switch between my text editor, a terminal, and a browser, and have exactly the most specific and relevant commands available at all times with no delay", and "I want to have syntax-aware voice grammars that are very in tune with how my programming language and framework operate" You appear to be talking about this without any significant research, sources, or knowledge of my software (which you have multiple times made fairly confident statements about that are not remotely true), please stop generalizing about what I'm doing without some examples to back it up.
  - khitchdee 8y ago
    Re: grammar changes: I assumed each language was associated with its grammar and it was not more fine grained than that. This makes sense though in the context where you're simply trying to replicate the keyboard-window-mouse based tool design with a speech-window-eye-tracking one. I guess this makes it easy for a user on the former UI to switch over. You will note my comments were not about your software, but hands-free programming interfaces in general. They continue to be this way, you will note. I am addressing the subject of this topic at a high level. I also have a fairly good idea on where your software stands within that context. Don't need to do any research for that.

brandonjm 8y ago

> However, it should not be difficult to design one for sppech based input. Personally I feel that this shouldn't even be necessary. In theory all you need to be able to vocalise is what you want the code to achieve, then that could be mapped to something similar to a snippet based on what language you are using. Most programming languages share the same core abilities, it's just the syntax to achieve the task that matters once you know what the task is. So this shouldn't need to be handled by a dedicated programming language for voice coding, it could just be handled by a wrapper (A text editor for your voice with an advanced NLP based snippet system). Something like: "Check if A is equal to B, if it is, return A, if not, return B" would be much more efficient than saying "if parenthesis capital A equals equals captial B..." and so on.

khitchdee 8y ago
I agree. I think a wrapper would be a good way to go about it with the voice based programming interface left only for a high level description of what the programmer is trying to do. Sort of akin to how many of today's new high level languages are actually implmented by translating to C and then using a C complier