6 ms·
I started something like this long ago but translated into Javascript rather than Python. It was fun to work on but I pretty quickly ran into issues with ambigu
by drusepth 7y ago
I started something like this long ago but translated into Javascript rather than Python. It was fun to work on but I pretty quickly ran into issues with ambiguity in spoken word.
For example, say you want to loop from 1 to 10, print each number, and then print "done". The spoken-code equivalent might be something like "from i equals 1 to 10, print i, then print 'done'". When you translate that into code, it could reasonably end up as either of the two following snippets:
for(i = 0; i < 10; i++) {
print(i)
}
print("done")
or
for(i = 0; i < 10; i++) {
print(i)
print("done")
}
The worst part about designing a grammar for spoken language is figuring out intuitive natural language to substitute for the "silent" parts of code -- braces, semicolons, whitespace, caret position, etc. Obviously Python doesn't have those symbols specifically, but the concepts still exist -- if you start defining a function (with "create function [function_name]" in your code), what extra language do you have/need to say "okay, we're done defining this function and ready to go back to writing code wherever we were before defining this function".
Say you line up "create function foo", "set variable x 5", and "set variable y 8" voice commands. Will y get set to 8 within that function you created? If so, how do you signify the end of the function? How do you know where your caret jumps back to after finishing a function?
This is a neat project and I think it has a lot of potential for use if done correctly, but it's also insanely difficult to get something both powerful and intuitive when you're lossily translating spoken code to executable code because you're toeing a balance between saying something natural and saying literally just every character you would otherwise write.
In case it's helpful, here's my now-seven-years-old attempt and proof-of-concept in Javascript: https://github.com/drusepth/voice2code https://github.com/drusepth/voice2code
- sonofgod 7y agovoicecode.io looked particularly interesting in this regard -- essentially creating a language to describe keypresses and mouseclicks (using some headtracking gear for mouse movement). It sounds like gibberish: SENSLAM tiny test DOT add PREN BING CAMEL example test that fails SWIPE function PREN test but it renders as Tinytest.add('exampleTestThatFails', function(test)) due to where the cursor is placed between punctuation. There's also commands for cursor movement and selection modification (expand to containing bracket/quotes, deselect the last word) There's presumably some underlying scheme to what they're using, but it's not entirely obvious. It's also not obvious how available this is, with a "send an email for further info" link. :(
- fouc 7y agoYou make me think that a whole new syntax for spoken programming would be key. Would be interesting to get a bunch of people to read out loud various programs in various languages and in pseudocode and hammer down what might be the most natural way to do spoken programming.
- harperlee 7y agoTwo avenues that I think are interesting: 1. Try to mimic english grammar. For this, Attempto controlled english (http://attempto.ifi.uzh.ch/site/ http://attempto.ifi.uzh.ch/site/) is something to review. 2. Go simpler and reduce grammar as much as possible. Prolog would be a good fit, as a program is just a set of Horn clauses.
- shakna 7y agoI've actually found that Lua is one of the least painful languages to write with voice control, as it has so few symbols - they tend to be only operators so you can switch them out easily enough in your engine. (And a few extras for moving around in your text editor, like "com jump $n upwards"). But you still do need to replace all sorts of brackets, like any language. At the moment I've got: "bracket" == ( "brace" == { "index" == [ (And "end $x" for the other side of it.) Clearly not ideal, but works well enough for now, certainly a lot less painful than trying to maintain indentation in Python with voice only. (com select 20 lines upwards, reverse tab... Followed by swearing as it tabs instead of untabs.)
- Cogito 7y agoFor what it's worth, I've always known them as: ( parentheses (parens), alt. round bracket { brace, alt. curly bracket [ bracket, alt. square bracket The biggest confusion is between bracket and parentheses, in that people will use bracket to mean both. If I think there might be confusion I use the alternatives as they seem to remove all ambiguity.
- shakna 7y agoYeah, unfortunately "parenthesis" is a hard word for most of the Speech-To-Text engines I tried, at least with my accent. ("parent thesis") And it is somewhat of a long word to be using all the time.
- scottlawson 7y agoThat is so interesting! It sounds very difficult to do speech to code for many of the most popular languages. Do you think an intermediate language is necessary? For example, instead of trying to implement speech to code for JavaScript, implement it for "VoiceScript", some purpose built JS variant which is intended to be unambiguous to process speech, and which transpiles to Javascript