7 ms·

Getting Alexa to Respond to Sign Language Using Your Webcam and TensorFlow.js

213 points by hardmaru 8y ago

hardmaru 8y ago
GitHub: https://github.com/shekit/alexa-sign-language-translator https://github.com/shekit/alexa-sign-language-translator Web Demo: https://shekit.github.io/alexa-sign-language-translator/ https://shekit.github.io/alexa-sign-language-translator/
tango24 8y ago
Whoa, this is really cool! My wife knows sign language and I showed her this and she was impressed, and had so many questions about how it worked =) I love the innovation — Thanks for sharing!
ai_ia 8y ago
Although off topic, I loved your paper with Schmidhuber, hardmaru. :)
jakamau 8y ago
This looks interesting. I'd be interested to see how the set-up works when it is trained with someone that signs natively. Most of my experience in the deaf community has been at trivia nights where the signing is incredibly fast (to me) and grammatically loose (just like everyone that's voicing at the bar). In the end I think it's an invaluable avenue to explore, the tech is important no matter what the end device looks like. Great work.
ausjke 8y ago
How many are keeping using alexa/echo these days? mine were turned off months ago and never felt the need to switch them on again, phone does pretty much everything I need, actually it does more than what I need. plus I don't feel comfortable that there is a silent listening device lurking 24x7 for all family members, so it is not just they're useless, but I intentionally do not want to have them on at all 99% of the time. the only amazon device I use is FireTV for netflix and Amazon prime video, all amazon ereaders, echo, alexa are replaced by my cellphone... to that extent, i feel amazon stock price is too high, too much hype for the alexa/echo/kindles at least in my opinion
- jonknee 8y ago
  Every day I am at home for weather, timers, lists.
  - ausjke 8y ago
    and phone does it all, plus being portable on-the-go
    - jonknee 8y ago
      Yes and we even did all these things before a phone and life was fine. The question wasn't if I use the Echo to do things that are impossible otherwise. I cook a lot and I find it a lot easier to just say "Alexa set a chicken timer for 15 minutes" or "Alexa add black pepper" than it is to get out my phone and do those things.
      - celestialjeu 8y ago
        agreed -- being able to verbally set timers is very valuable to me. Also I have Lifx bulbs in my living room and it is much easier to tell Alexa to turn them on/off/change brightness than it is to get out my phone. I only use the phone app for that when I want to set up some custom color thing (which is pretty rare tbh)
- snaky 8y ago
  We've yet to see any useful and no-nonsense functionality from that sound boxes I suppose. All the functions available today, like "weather" or "play a song" are definitely done faster by the phone - and by hands, with no voice command at all.
- outworlder 8y ago
  > I don't feel comfortable that there is a silent listening device lurking 24x7 for all family members They are listening for the wake word. If they were streaming this data anywhere, people would notice. Heck, I would notice as I keep a close eye on them. I find it amusing that you are not ok with Echo, but you are ok with a cellphone, which not only is listening all the time too ("Ok Google", "Hey Siri") but also has your location information, emails, photos, all sort of messages AND has its own backchannel to upload this information (the cellular data network), which is much more difficult to keep track of. My Echos are not useless, when you add some home automation they become much more useful. They are still pretty "dumb" though.
imglorp 8y ago
ASL student here. I just wanted to highlight how many challenges there are to translating, even for humans. Here's two examples. First is dialect. One would think the Internet/youtube/videochat would vaporize geography differences, and it's starting to happen, but that's not (yet) the case. Lots of signs have "synonyms" based on signer's preference, etymology, who they learned from and who they hang out with, etc. Another is grammar. What OP is doing in his video, for example is English grammar, where signs are directly substitited for words. But ASL has its own sign order, modifiers like facial expressions, and idioms for brevity. For example "have you ever been to San Francisco before?" might be "SF TOUCH YOU FINISH" with raised eyebrows at the end, as a modifier. Note also there are no conjugations or articles like some languages, but there are pronouns. In fact, there are local bindings where you make up a sign name for someone on the fly and then use it during a conversation. I think with large enough training sets, this will all be mitigated, like Google needed years of speech samples to get Translate working okay.
- mabbo 8y ago
  I took 2.5 years of ASL in University, and I've always been annoyed by this, but had a hard time describing what I mean to others. You have just summed up the problem perfectly in a few sentences, and I thank you for that. That said, I think the OP is still on the right track here: > I put it together so you can train it on your own set of word and sign/gesture combos. A Deaf person should be able to train the system on each command they want so that it works for them automatically in the dialect they want. > there are local bindings where you make up a sign name for someone on the fly and then use it during a conversation My very favourite thing about ASL, having learned programming beforehand, was that I could assign people to variables/registers in space. "John point at spot to my left said to Susan spot on right bla bla bla" and then later be able to just point at my left and everyone knows I mean John. And then if John moves to the right, I've just moved him into that register and can reassign the left one to someone or something else. And with a group of experienced Signers, everyone just comprehends this perfectly.
  - imglorp 8y ago
    Ah, pointing at sign space for someone... I like to think of that like a pronoun (there's probably an official term). There's also this thing where you introduce a character, then make up a "temporary name sign" on the fly. Don't know what that's called either :)
al_ramich 8y ago
Love the concept, great work. Came across these guys who are working on making travel more accessible for the deaf community. I would see some overlap in using Alexa to make more voice inclusive content accessible https://startupyard.com/meet-deaftravel-building-a-better-internet-for-deaf-people/ https://startupyard.com/meet-deaftravel-building-a-better-in...
pbhjpbhj 8y ago
Interesting I think if I were approaching BSL sign interpretation I'd start with recognising hand shapes, and then move on to other elements of sign notation ... in theory then you could feed the system a dictionary and recognise all signs. I think BSL uses a variant of Stokoe notation. Presumably the system of the OP could be modified to recognise dance/skating/snowboarding/martial arts moves. It seems that once you have an ability to model every aspect of body position from video (as there doing for enhanced love-action sequences in movies now) that this sort of thing becomes much easier.
abhi3188 8y ago
Hey, I'm the guy who made this! Happy to answer any questions that the blog didn't address
- hammock 8y ago
  Do you think you could get this to work for Google Assistant as well? Does ASL-to-text already exist outside of voice assistants, or is that part totally new?
cwt137 8y ago
This is similar to an AWS DeepLens community project called ASLens https://aws.amazon.com/deeplens/community-projects/ASLens/ https://aws.amazon.com/deeplens/community-projects/ASLens/
neha 8y ago
hi

213 points by hardmaru 8y ago

GitHub: https://github.com/shekit/alexa-sign-language-translator https://github.com/shekit/alexa-sign-language-translator Web Demo: https://shekit.github.io/alexa-sign-language-translator/ https://shekit.github.io/alexa-sign-language-translator/

tango24 8y ago

Whoa, this is really cool! My wife knows sign language and I showed her this and she was impressed, and had so many questions about how it worked =) I love the innovation — Thanks for sharing!

ai_ia 8y ago

Although off topic, I loved your paper with Schmidhuber, hardmaru. :)

jakamau 8y ago

This looks interesting. I'd be interested to see how the set-up works when it is trained with someone that signs natively. Most of my experience in the deaf community has been at trivia nights where the signing is incredibly fast (to me) and grammatically loose (just like everyone that's voicing at the bar). In the end I think it's an invaluable avenue to explore, the tech is important no matter what the end device looks like. Great work.

ausjke 8y ago

How many are keeping using alexa/echo these days? mine were turned off months ago and never felt the need to switch them on again, phone does pretty much everything I need, actually it does more than what I need. plus I don't feel comfortable that there is a silent listening device lurking 24x7 for all family members, so it is not just they're useless, but I intentionally do not want to have them on at all 99% of the time. the only amazon device I use is FireTV for netflix and Amazon prime video, all amazon ereaders, echo, alexa are replaced by my cellphone... to that extent, i feel amazon stock price is too high, too much hype for the alexa/echo/kindles at least in my opinion

jonknee 8y ago
Every day I am at home for weather, timers, lists.
- ausjke 8y ago
  and phone does it all, plus being portable on-the-go
  - jonknee 8y ago
    Yes and we even did all these things before a phone and life was fine. The question wasn't if I use the Echo to do things that are impossible otherwise. I cook a lot and I find it a lot easier to just say "Alexa set a chicken timer for 15 minutes" or "Alexa add black pepper" than it is to get out my phone and do those things.
    - celestialjeu 8y ago
      agreed -- being able to verbally set timers is very valuable to me. Also I have Lifx bulbs in my living room and it is much easier to tell Alexa to turn them on/off/change brightness than it is to get out my phone. I only use the phone app for that when I want to set up some custom color thing (which is pretty rare tbh)
snaky 8y ago
We've yet to see any useful and no-nonsense functionality from that sound boxes I suppose. All the functions available today, like "weather" or "play a song" are definitely done faster by the phone - and by hands, with no voice command at all.
outworlder 8y ago
> I don't feel comfortable that there is a silent listening device lurking 24x7 for all family members They are listening for the wake word. If they were streaming this data anywhere, people would notice. Heck, I would notice as I keep a close eye on them. I find it amusing that you are not ok with Echo, but you are ok with a cellphone, which not only is listening all the time too ("Ok Google", "Hey Siri") but also has your location information, emails, photos, all sort of messages AND has its own backchannel to upload this information (the cellular data network), which is much more difficult to keep track of. My Echos are not useless, when you add some home automation they become much more useful. They are still pretty "dumb" though.

imglorp 8y ago

ASL student here. I just wanted to highlight how many challenges there are to translating, even for humans. Here's two examples. First is dialect. One would think the Internet/youtube/videochat would vaporize geography differences, and it's starting to happen, but that's not (yet) the case. Lots of signs have "synonyms" based on signer's preference, etymology, who they learned from and who they hang out with, etc. Another is grammar. What OP is doing in his video, for example is English grammar, where signs are directly substitited for words. But ASL has its own sign order, modifiers like facial expressions, and idioms for brevity. For example "have you ever been to San Francisco before?" might be "SF TOUCH YOU FINISH" with raised eyebrows at the end, as a modifier. Note also there are no conjugations or articles like some languages, but there are pronouns. In fact, there are local bindings where you make up a sign name for someone on the fly and then use it during a conversation. I think with large enough training sets, this will all be mitigated, like Google needed years of speech samples to get Translate working okay.

mabbo 8y ago
I took 2.5 years of ASL in University, and I've always been annoyed by this, but had a hard time describing what I mean to others. You have just summed up the problem perfectly in a few sentences, and I thank you for that. That said, I think the OP is still on the right track here: > I put it together so you can train it on your own set of word and sign/gesture combos. A Deaf person should be able to train the system on each command they want so that it works for them automatically in the dialect they want. > there are local bindings where you make up a sign name for someone on the fly and then use it during a conversation My very favourite thing about ASL, having learned programming beforehand, was that I could assign people to variables/registers in space. "John point at spot to my left said to Susan spot on right bla bla bla" and then later be able to just point at my left and everyone knows I mean John. And then if John moves to the right, I've just moved him into that register and can reassign the left one to someone or something else. And with a group of experienced Signers, everyone just comprehends this perfectly.
- imglorp 8y ago
  Ah, pointing at sign space for someone... I like to think of that like a pronoun (there's probably an official term). There's also this thing where you introduce a character, then make up a "temporary name sign" on the fly. Don't know what that's called either :)

al_ramich 8y ago

Love the concept, great work. Came across these guys who are working on making travel more accessible for the deaf community. I would see some overlap in using Alexa to make more voice inclusive content accessible https://startupyard.com/meet-deaftravel-building-a-better-internet-for-deaf-people/ https://startupyard.com/meet-deaftravel-building-a-better-in...

pbhjpbhj 8y ago

Interesting I think if I were approaching BSL sign interpretation I'd start with recognising hand shapes, and then move on to other elements of sign notation ... in theory then you could feed the system a dictionary and recognise all signs. I think BSL uses a variant of Stokoe notation. Presumably the system of the OP could be modified to recognise dance/skating/snowboarding/martial arts moves. It seems that once you have an ability to model every aspect of body position from video (as there doing for enhanced love-action sequences in movies now) that this sort of thing becomes much easier.

abhi3188 8y ago

Hey, I'm the guy who made this! Happy to answer any questions that the blog didn't address

hammock 8y ago
Do you think you could get this to work for Google Assistant as well? Does ASL-to-text already exist outside of voice assistants, or is that part totally new?

cwt137 8y ago

This is similar to an AWS DeepLens community project called ASLens https://aws.amazon.com/deeplens/community-projects/ASLens/ https://aws.amazon.com/deeplens/community-projects/ASLens/

neha 8y ago