10 ms·

Voice Coder – Alexa skill to code by voice

oulipo 8y ago
Really cool! Perhaps you could also take a look at what we are building at https://snips.ai https://snips.ai (disclaimer: I'm a co-founder) to make a 100% on-device and private-by-design Voice AI which runs even on a RaspberryPi 3 it would be awesome to do this on a raspi, without requiring internet connection!
- jamesfdickinson 8y ago
  Great idea. I really like idea of not requiring an internet connection. Self contained projects are more reliable and portable. Also this project is aimed towards students so privacy is a real concern. The way I built it should be easy to use another voice ai input. The Alexa module of the code just passes the commands to the Core logic. It's all built in node.js. I built this for Alexa Skills Challenge: Tech for Good. https://devpost.com/software/code-by-voice https://devpost.com/software/code-by-voice . More details about how it was built can be found there.
  - j88439h84 8y ago
    https://mycroft.ai/ https://mycroft.ai/ is a very popular assistant too. (No affiliation)
- dovemint 8y ago
  I'm curious to know how the device runs everything on device without excessive storage and also, can you create a skill to ask snips to fetch information on the internet without sacrificing privacy?
- spacehome 8y ago
  This looks neat! Is it possible to get started without creating an account on your website? I couldn't figure out how, but I figured there must be a way if it's "100% on-device".
  - jamesfdickinson 8y ago
    You do not create an account on the website. Right now the only way is to open the Alexa skill and it will give you a 4 digit code for your game. After you open your game on the website you can continue adding rules to your game via Alexa. New game logic will update in the game in real time. I would like to make a version 100% on device. A few have suggest how.
    - spacehome 8y ago
      Thanks, but I was really asking my question about Snips.
  - oulipo 8y ago
    Hi spacehome, you need to create an account in order to use the webconsole to build the assistant (create intents and queries), then once this is done you can use it 100% on-device!
    - doogan 8y ago
      You guys are are also incentivizing for app work done via the console / added to platform right? For a serious dev with experience in app deployment what could I look forward to? Most of my background is in data security so this is definitely up my alley.
- MindSpaceGAL 8y ago
  This would be so awesome! I'm wondering how Snips.ai functions with no internet connection? I'm an early stage programmer with not much RaspPi experience unfortunately. Is it just a closed circuit for home automation. How does it fetch data for tasks that need external servers to update?
- TrinaryWorksToo 8y ago
  I met you at maker fair! I was impressed by the demo and how well it could recognize my voice even when I was far away or facing hard surfaces.
sleepychu 8y ago
Seems like the skill needs a lot of semantic knowledge. This is less coding than a meta game.
- jamesfdickinson 8y ago
  The game template supplies the available events/action/value. The Game Creator guides you through the available options supplied by the template. It then tries to use a human speech like interface to receive the input, the game logic. Currently there is only one game template. Each new game template would need to supply it's own possible events/actions/values to suit it's needs. A game template looks something like this: https://gist.github.com/jamesfdickinson/65b9f5a05f4902468c464b09235572d0 https://gist.github.com/jamesfdickinson/65b9f5a05f4902468c46...
  - sleepychu 8y ago
    Is this the template for the example? If so where does the logic for columns of boxes and stars come from?
    - jamesfdickinson 8y ago
      This game template was design for students with zero experience. So the create new objects command was omitted in the user defined rules. The game itself controls the box placement for this game. The goal of that game was to learn about events and actions. I plan to add more game templates with the create object command for more advanced users. You can set it's properties, game physic values and so on. The game itself will need to implement and execute that command. The game template just tells the Game Creator what that current game supports. So the magic happens in the game code. Hit view source on the game if want to see it. The project was built in a 2 weeks; server, Alexa integration, mini-game, and game creator logic. If the project gets any traction, I'd love to keep working on it.
- leowoo91 8y ago
  This is more like changing exposed parameters in Unity during gameplay for tweaking.
js8 8y ago
If you are interested in state-of-the-art voice coding, I recommend this video: https://www.youtube.com/watch?v=YRyYIIFKsdU https://www.youtube.com/watch?v=YRyYIIFKsdU
- melling 8y ago
  I have that and several other resources on a Github page: https://github.com/melling/ErgonomicNotes/blob/master/programming_by_voice.org https://github.com/melling/ErgonomicNotes/blob/master/progra...
- lunixbochs 8y ago
  Silvius isn't really state of the art. It's just newish. On their website it even says the project that inspired them has better performance. It references Aenea (which is a convoluted system of using Dragon on Linux in a VM) and calls out to Tavis Rudd's 5-year-old presentation and Dragonfly, none of which are state of the art. Silvius is unique in being the only project I know of to use Kaldi, which is very cool because it's open-source and doesn't require a GPU, but the recognition accuracy of Silvius isn't great and the voice model is fixed and has no feedback from the grammar.
nsomaru 8y ago
What’s the IP status of recordings sent to Amazon? Might that affect the IP of the code produced?
- jamesfdickinson 8y ago
  The project is designed to help students to understand the basic of coding logic using their voice. Think of it more like a coding lesson for beginners. Many at this level use drag and drop coding. Code.org has many drag and drop coding lessons. The difference here is it uses voice. The recordings are not part of the code produced, just plain text. This version cannot make production like code, just add game logic to an already created game template.
FundThrowaway 8y ago
I guess things have moved on from this Perl-Vista nightmare, I was personally looking forward to having all the coders around me screaming at their desktops in a fit of frustration and rage. https://www.youtube.com/watch?v=MzJ0CytAsec https://www.youtube.com/watch?v=MzJ0CytAsec
- melling 8y ago
  That’s 11 years old.
  - FundThrowaway 8y ago
    It was a joke, relax.
    - melling 8y ago
      This isn’t Reddit
hanoz 8y ago
Me: "Alexa, open Voice Coder" Alexa: "Sorry, I don't know that"
- jamesfdickinson 8y ago
  go to https://www.amazon.com/dp/B07HFWQPKN https://www.amazon.com/dp/B07HFWQPKN and enable the skill.
  - hanoz 8y ago
    Ah, that's why. It's not available in the UK.
    - jamesfdickinson 8y ago
      I put it in the catagory kids education. Alexa only supports the kids section in the US right now. I may change that so it can be accessed worldwide.
brtknr 8y ago
I think this approach will need a great more deal of maturity before I start programming in the way demonstrated in the video.
StavrosK 8y ago
I ask it for my code it and Alexa says, oddly and quickly, "0irk.". That code is not accepted.
- jamesfdickinson 8y ago
  I tested your code and it works for me. Make sure you type 0IRK and nothing else like spaces or periods. or click this link https://voicecoder.net/game.html?id=0IRK https://voicecoder.net/game.html?id=0IRK What browser and device are you using?
  - StavrosK 8y ago
    I guess the problem was that it said "dot", so I included a period. It should probably not say that. It works now, thanks!
    - jamesfdickinson 8y ago
      Thank you for the feedback, I'll fix that.
rpjones7 8y ago
I have personally been playing with Snips. It has been an exciting project to work with. I do need some help with certain things, but reddit has been a good source of information.
jt3 8y ago
See https://talonvoice.com https://talonvoice.com for an extensible open source alternative to voice code. The creator/main developer is a friend of mine and is one of the smartest people I know. The project can be used to control IDA for reverse engineering, play video games, program, and more.
- lunixbochs 8y ago
  Thanks for the plug! Couple notes: Talon is free, but there are closed source bits, and this Alexa project is "voice coder", which is different than the paid and abandoned voicecode project. The difference between this and Talon, is Talon can be used for real programming and uses very low latency offline recognition, while voice coder looks like a guided tutorial and the user experience is probably a fairly high latency single command call/response based on my previous Alexa experience. I'm not sure Alexa can recognize a streaming series of commands without large recognition pauses between each command.
  - jamesfdickinson 8y ago
    Correct. Voice Coder is a guided tutorial for beginners, not a replacement for developing production code. It does allow for multi step answers, but is a high latency back and forth dialog with a smaller set of commands. The goal is to be extremely human friendly, not productive, to better understand the basics of coding.
  - melling 8y ago
    Talon seems impressive. https://youtu.be/ddFI63dgpaI https://youtu.be/ddFI63dgpaI How is its accuracy for daily use? How about random navigation within a file, for example. Does it have an active community?
unit91 8y ago
Headline should be "Voice Configurator -- Alexa skill to configure by voice."
FlatWhite 8y ago
Hi i'm not quite skilled in coding but is the barrier of entry high? Would someone like me with not a lot of knowledge be able to pick this up and start programming?
- savanaly 8y ago
  Well it's simply an alternative to entering code via the keyboard. If you can't code using a keyboard already this won't help in that regard.
- umanwizard 8y ago
  Nothing in life is impossible. But unless you have some sort of physical disability that prevents you from using a keyboard and monitor, you are much better off learning programming using standard tools that have tons of noob-friendly documentation, rather than something experimental like this.

Really cool! Perhaps you could also take a look at what we are building at https://snips.ai https://snips.ai (disclaimer: I'm a co-founder) to make a 100% on-device and private-by-design Voice AI which runs even on a RaspberryPi 3 it would be awesome to do this on a raspi, without requiring internet connection!

jamesfdickinson 8y ago
Great idea. I really like idea of not requiring an internet connection. Self contained projects are more reliable and portable. Also this project is aimed towards students so privacy is a real concern. The way I built it should be easy to use another voice ai input. The Alexa module of the code just passes the commands to the Core logic. It's all built in node.js. I built this for Alexa Skills Challenge: Tech for Good. https://devpost.com/software/code-by-voice https://devpost.com/software/code-by-voice . More details about how it was built can be found there.
- j88439h84 8y ago
  https://mycroft.ai/ https://mycroft.ai/ is a very popular assistant too. (No affiliation)
dovemint 8y ago
I'm curious to know how the device runs everything on device without excessive storage and also, can you create a skill to ask snips to fetch information on the internet without sacrificing privacy?
spacehome 8y ago
This looks neat! Is it possible to get started without creating an account on your website? I couldn't figure out how, but I figured there must be a way if it's "100% on-device".
- jamesfdickinson 8y ago
  You do not create an account on the website. Right now the only way is to open the Alexa skill and it will give you a 4 digit code for your game. After you open your game on the website you can continue adding rules to your game via Alexa. New game logic will update in the game in real time. I would like to make a version 100% on device. A few have suggest how.
  - spacehome 8y ago
    Thanks, but I was really asking my question about Snips.
- oulipo 8y ago
  Hi spacehome, you need to create an account in order to use the webconsole to build the assistant (create intents and queries), then once this is done you can use it 100% on-device!
  - doogan 8y ago
    You guys are are also incentivizing for app work done via the console / added to platform right? For a serious dev with experience in app deployment what could I look forward to? Most of my background is in data security so this is definitely up my alley.
MindSpaceGAL 8y ago
This would be so awesome! I'm wondering how Snips.ai functions with no internet connection? I'm an early stage programmer with not much RaspPi experience unfortunately. Is it just a closed circuit for home automation. How does it fetch data for tasks that need external servers to update?
TrinaryWorksToo 8y ago
I met you at maker fair! I was impressed by the demo and how well it could recognize my voice even when I was far away or facing hard surfaces.

sleepychu 8y ago

Seems like the skill needs a lot of semantic knowledge. This is less coding than a meta game.

jamesfdickinson 8y ago
The game template supplies the available events/action/value. The Game Creator guides you through the available options supplied by the template. It then tries to use a human speech like interface to receive the input, the game logic. Currently there is only one game template. Each new game template would need to supply it's own possible events/actions/values to suit it's needs. A game template looks something like this: https://gist.github.com/jamesfdickinson/65b9f5a05f4902468c464b09235572d0 https://gist.github.com/jamesfdickinson/65b9f5a05f4902468c46...
- sleepychu 8y ago
  Is this the template for the example? If so where does the logic for columns of boxes and stars come from?
  - jamesfdickinson 8y ago
    This game template was design for students with zero experience. So the create new objects command was omitted in the user defined rules. The game itself controls the box placement for this game. The goal of that game was to learn about events and actions. I plan to add more game templates with the create object command for more advanced users. You can set it's properties, game physic values and so on. The game itself will need to implement and execute that command. The game template just tells the Game Creator what that current game supports. So the magic happens in the game code. Hit view source on the game if want to see it. The project was built in a 2 weeks; server, Alexa integration, mini-game, and game creator logic. If the project gets any traction, I'd love to keep working on it.
leowoo91 8y ago
This is more like changing exposed parameters in Unity during gameplay for tweaking.

js8 8y ago

If you are interested in state-of-the-art voice coding, I recommend this video: https://www.youtube.com/watch?v=YRyYIIFKsdU https://www.youtube.com/watch?v=YRyYIIFKsdU

melling 8y ago
I have that and several other resources on a Github page: https://github.com/melling/ErgonomicNotes/blob/master/programming_by_voice.org https://github.com/melling/ErgonomicNotes/blob/master/progra...
lunixbochs 8y ago
Silvius isn't really state of the art. It's just newish. On their website it even says the project that inspired them has better performance. It references Aenea (which is a convoluted system of using Dragon on Linux in a VM) and calls out to Tavis Rudd's 5-year-old presentation and Dragonfly, none of which are state of the art. Silvius is unique in being the only project I know of to use Kaldi, which is very cool because it's open-source and doesn't require a GPU, but the recognition accuracy of Silvius isn't great and the voice model is fixed and has no feedback from the grammar.

nsomaru 8y ago

What’s the IP status of recordings sent to Amazon? Might that affect the IP of the code produced?

jamesfdickinson 8y ago
The project is designed to help students to understand the basic of coding logic using their voice. Think of it more like a coding lesson for beginners. Many at this level use drag and drop coding. Code.org has many drag and drop coding lessons. The difference here is it uses voice. The recordings are not part of the code produced, just plain text. This version cannot make production like code, just add game logic to an already created game template.

FundThrowaway 8y ago

I guess things have moved on from this Perl-Vista nightmare, I was personally looking forward to having all the coders around me screaming at their desktops in a fit of frustration and rage. https://www.youtube.com/watch?v=MzJ0CytAsec https://www.youtube.com/watch?v=MzJ0CytAsec

melling 8y ago
That’s 11 years old.
- FundThrowaway 8y ago
  It was a joke, relax.
  - melling 8y ago
    This isn’t Reddit

hanoz 8y ago

Me: "Alexa, open Voice Coder" Alexa: "Sorry, I don't know that"

jamesfdickinson 8y ago
go to https://www.amazon.com/dp/B07HFWQPKN https://www.amazon.com/dp/B07HFWQPKN and enable the skill.
- hanoz 8y ago
  Ah, that's why. It's not available in the UK.
  - jamesfdickinson 8y ago
    I put it in the catagory kids education. Alexa only supports the kids section in the US right now. I may change that so it can be accessed worldwide.

brtknr 8y ago

I think this approach will need a great more deal of maturity before I start programming in the way demonstrated in the video.

StavrosK 8y ago

I ask it for my code it and Alexa says, oddly and quickly, "0irk.". That code is not accepted.

jamesfdickinson 8y ago
I tested your code and it works for me. Make sure you type 0IRK and nothing else like spaces or periods. or click this link https://voicecoder.net/game.html?id=0IRK https://voicecoder.net/game.html?id=0IRK What browser and device are you using?
- StavrosK 8y ago
  I guess the problem was that it said "dot", so I included a period. It should probably not say that. It works now, thanks!
  - jamesfdickinson 8y ago
    Thank you for the feedback, I'll fix that.

rpjones7 8y ago

I have personally been playing with Snips. It has been an exciting project to work with. I do need some help with certain things, but reddit has been a good source of information.

jt3 8y ago

See https://talonvoice.com https://talonvoice.com for an extensible open source alternative to voice code. The creator/main developer is a friend of mine and is one of the smartest people I know. The project can be used to control IDA for reverse engineering, play video games, program, and more.

lunixbochs 8y ago
Thanks for the plug! Couple notes: Talon is free, but there are closed source bits, and this Alexa project is "voice coder", which is different than the paid and abandoned voicecode project. The difference between this and Talon, is Talon can be used for real programming and uses very low latency offline recognition, while voice coder looks like a guided tutorial and the user experience is probably a fairly high latency single command call/response based on my previous Alexa experience. I'm not sure Alexa can recognize a streaming series of commands without large recognition pauses between each command.
- jamesfdickinson 8y ago
  Correct. Voice Coder is a guided tutorial for beginners, not a replacement for developing production code. It does allow for multi step answers, but is a high latency back and forth dialog with a smaller set of commands. The goal is to be extremely human friendly, not productive, to better understand the basics of coding.
- melling 8y ago
  Talon seems impressive. https://youtu.be/ddFI63dgpaI https://youtu.be/ddFI63dgpaI How is its accuracy for daily use? How about random navigation within a file, for example. Does it have an active community?

unit91 8y ago

Headline should be "Voice Configurator -- Alexa skill to configure by voice."

FlatWhite 8y ago

Hi i'm not quite skilled in coding but is the barrier of entry high? Would someone like me with not a lot of knowledge be able to pick this up and start programming?

savanaly 8y ago
Well it's simply an alternative to entering code via the keyboard. If you can't code using a keyboard already this won't help in that regard.
umanwizard 8y ago
Nothing in life is impossible. But unless you have some sort of physical disability that prevents you from using a keyboard and monitor, you are much better off learning programming using standard tools that have tons of noob-friendly documentation, rather than something experimental like this.